You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Anshul Gangwar <an...@accelerite.com> on 2016/06/24 04:30:21 UTC
Re: Xenserver 6.5 live migration with local volumes
Created the PR https://github.com/apache/cloudstack/pull/1596.
Regards,
Anshul
On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
Is there anybody that wants to take this on?
Thanks,
- Dave
> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com> wrote:
>
> I updated the issue referenced in this email (https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include CloudStack 4.8 as an affected version. If we could at least get someone assigned to it, that’d be a good start.
>
> Thanks,
>
> - Dave
>
> On May 24, 2016, at 9:42 AM, Ivan Derbenev <iv...@tech-corps.com>> wrote:
>
> Is there a way I can fix this on my installation, without recompiling jar files?
>
> Regards,
> IT engineer
> Farheap, Russia
> Ivan Derbenev
>
> -----Original Message-----
> From: Koushik Das [mailto:koushik.das@accelerite.com]
> Sent: Tuesday, May 24, 2016 2:06 PM
> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
> Subject: Re: Xenserver 6.5 live migration with local volumes
>
> This is a product bug and needs to be fixed. There is some serialization issue that needs fixing.
>
> In the below code volumeToFiler is null as Map is not a serializable type. It needs to be replaced with getVolumeToFilerAsList() and then the data needs to be read appropriately.
>
> final Map<VolumeTO, StorageFilerTO> volumeToFiler = command.getVolumeToFiler();
>
> -Koushik
>
> On 24/05/16, 3:23 PM, "Ivan Derbenev" <iv...@tech-corps.com>> wrote:
>
> Hello!
> After the migration onto XS 6.5 (and installing all the updates) and CS
> 4.8 I can't migrate VMs with Local Storage
>
> The issue is similar to this:
> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't have any solutions Any ideas?
>
> Errors:
>
> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process
> of VM state report. host: 136592
> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
> named cloud-main
> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
> cloud-main on host=10.7.130.103;
> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
> host 10.7.130.103 on device eth0
> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
> 2016-05-24 12:10:17,170 WARN
> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception java.lang.NullPointerException. Storage motion failed due to java.lang.NullPointerException java.lang.NullPointerException
> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq 136589-479633360314957887: Response Received:
> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
> 136589-479633360314957887: Processing: { Ans: , MgmtId:
> 128437433941265, via:
> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> 110,
> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
> java.lang.NullPointerException\n\tat
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
> andWrapper.java:86)\n\tat
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
> andWrapper.java:54)\n\tat
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
> apper.execute(CitrixRequestWrapper.java:122)\n\tat
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
> st(CitrixResourceBase.java:1677)\n\tat
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
> tAttache.java:315)\n\tat
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
> gedContextRunnable.java:49)\n\tat
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
> (DefaultManagedContext.java:56)\n\tat
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
> thContext(DefaultManagedContext.java:103)\n\tat
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
> hContext(DefaultManagedContext.java:53)\n\tat
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
> dContextRunnable.java:46)\n\tat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<http://ScheduledFutureTask.ru>
> n(ScheduledThreadPoolExecutor.java:292)\n\tat
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
> va:1145)\n\tat
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
> 136589-479633360314957887: No more commands found
> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
> MgmtId: 128437433941265, via:
> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> 110, { MigrateWithStorageAnswer } }
> 2016-05-24 12:10:17,171 ERROR
> [o.a.c.s.m.XenServerStorageMotionStrategy]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
> failed. Details: Exception: java.lang.NullPointerException
> Message: null
> Stack: java.lang.NullPointerException
> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-05-24 12:10:17,171 ERROR
> [o.a.c.s.m.XenServerStorageMotionStrategy]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) copy failed
> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
> java.lang.NullPointerException
> Message: null
> Stack: java.lang.NullPointerException
> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
> at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
> at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Unsupported data object (VOLUME,
> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2),
> no need to delete from object in store ref table
> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
> while migrating the vm VM[User|i-3-271-VM] to host
> Host[-136589-Routing]. Exception: java.lang.NullPointerException
> Message: null
> Stack: java.lang.NullPointerException
> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
> VM[User|i-3-271-VM]
> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from host
> cloudstack-computing01-sar01.tech-corps.com<http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar and pod
> zone1.sar
> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
> skipping send email
> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
> 128437433941265
> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
> MgmtId: 128437433941265, via:
> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
> MgmtId: 128437433941265, via:
> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
> 136589-479633360314957888: Executing request
> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) VM state transitted from :Migrating to Running with
> event: OperationFailedvm's original host id: 136592 new host id: 136592
> host id before state transition: 136589
> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying
> overprovisioning: 39900
> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
> applying overprovisioning: 11549697024
> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) release cpu from host: 136589, old used:
> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
> 39900; new used: 1995,reserved:0; movedfromreserved:
> false,moveToReserveredfalse
> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) release mem from host: 136589, old used:
> 5368709120,reserved: 0, total: 11549697024; new used:
> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Response Received:
> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
> 136589-479633360314957888: Processing: { Ans: , MgmtId:
> 128437433941265, via:
> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
> does not exist","wait":0}}] }
> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Unable to find listener.
> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for
> VM 271, job origin: 4874
> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
> instanceType: null, instanceId: null, cmd:
> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
> origin:4874
> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> Publish async job-4875 complete on message bus
> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> Wake up jobs related to job-4875
> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> Update db status for job-4875
> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> Wake up jobs joined with job-4875 and disjoin all subjobs created from
> job- 4875
> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> Remove job-4875 from job monitoring
> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
> umeCmd
> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
> async job-4874, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
> ],"errorcode":530,"errortext":"Failed to migrated vm
> VM[User|i-3-271-VM] along with its volumes. "}
> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
> async job-4874 complete on message bus
> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
> jobs related to job-4874
> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
> status for job-4874
> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
> jobs joined with job-4874 and disjoin all subjobs created from job-
> 4874
> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
> executing
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
> umeCmd for job-4874
> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
> job-4874 from job monitoring ^C
>
>
> Regards,
> IT engineer
> Farheap, Russia
> Ivan Derbenev
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: Xenserver 6.5 live migration with local volumes
Posted by Anshul Gangwar <an...@accelerite.com>.
Hi Remi,
For me it’s (migration across clusters with storage) working fine but am on internal branch. There could be some regressions due to recent refactors. As far as this patch goes this handles with cluster case.
PR https://github.com/apache/cloudstack/pull/925 is no more valid after fixing of https://issues.apache.org/jira/browse/CLOUDSTACK-9231. This patch handles the broken scenario.
Regards,
Anshul
On 24/06/16, 7:31 PM, "Remi Bergsma" <RB...@schubergphilis.com> wrote:
Hi Anshul,
Thanks! Did you ever try live migrating from CloudStack with storage between clusters? I can't get that to work. I'll collect some details in the next days but was wondering if it was supposed to work.
Regards, Remi
Sent from my iPhone
> On 24 Jun 2016, at 06:30, Anshul Gangwar <an...@accelerite.com> wrote:
>
> Created the PR https://github.com/apache/cloudstack/pull/1596.
>
> Regards,
> Anshul
>
> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
>
> Is there anybody that wants to take this on?
>
> Thanks,
>
> - Dave
>
>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com> wrote:
>>
>> I updated the issue referenced in this email (https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include CloudStack 4.8 as an affected version. If we could at least get someone assigned to it, that’d be a good start.
>>
>> Thanks,
>>
>> - Dave
>>
>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <iv...@tech-corps.com>> wrote:
>>
>> Is there a way I can fix this on my installation, without recompiling jar files?
>>
>> Regards,
>> IT engineer
>> Farheap, Russia
>> Ivan Derbenev
>>
>> -----Original Message-----
>> From: Koushik Das [mailto:koushik.das@accelerite.com]
>> Sent: Tuesday, May 24, 2016 2:06 PM
>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
>> Subject: Re: Xenserver 6.5 live migration with local volumes
>>
>> This is a product bug and needs to be fixed. There is some serialization issue that needs fixing.
>>
>> In the below code volumeToFiler is null as Map is not a serializable type. It needs to be replaced with getVolumeToFilerAsList() and then the data needs to be read appropriately.
>>
>> final Map<VolumeTO, StorageFilerTO> volumeToFiler = command.getVolumeToFiler();
>>
>> -Koushik
>>
>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <iv...@tech-corps.com>> wrote:
>>
>> Hello!
>> After the migration onto XS 6.5 (and installing all the updates) and CS
>> 4.8 I can't migrate VMs with Local Storage
>>
>> The issue is similar to this:
>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't have any solutions Any ideas?
>>
>> Errors:
>>
>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process
>> of VM state report. host: 136592
>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
>> named cloud-main
>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
>> cloud-main on host=10.7.130.103;
>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
>> host 10.7.130.103 on device eth0
>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
>> 2016-05-24 12:10:17,170 WARN
>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception java.lang.NullPointerException. Storage motion failed due to java.lang.NullPointerException java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq 136589-479633360314957887: Response Received:
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
>> 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 110,
>> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
>> java.lang.NullPointerException\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>> andWrapper.java:86)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>> andWrapper.java:54)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
>> st(CitrixResourceBase.java:1677)\n\tat
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
>> tAttache.java:315)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
>> gedContextRunnable.java:49)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
>> (DefaultManagedContext.java:56)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>> thContext(DefaultManagedContext.java:103)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>> hContext(DefaultManagedContext.java:53)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>> dContextRunnable.java:46)\n\tat
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<http://ScheduledFutureTask.ru>
>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>> va:1145)\n\tat
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>> 136589-479633360314957887: No more commands found
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 110, { MigrateWithStorageAnswer } }
>> 2016-05-24 12:10:17,171 ERROR
>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
>> failed. Details: Exception: java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 2016-05-24 12:10:17,171 ERROR
>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) copy failed
>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>> java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
>> at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
>> at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
>> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Unsupported data object (VOLUME,
>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2),
>> no need to delete from object in store ref table
>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
>> while migrating the vm VM[User|i-3-271-VM] to host
>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
>> VM[User|i-3-271-VM]
>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from host
>> cloudstack-computing01-sar01.tech-corps.com<http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar and pod
>> zone1.sar
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
>> skipping send email
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
>> 128437433941265
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
>> 136589-479633360314957888: Executing request
>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) VM state transitted from :Migrating to Running with
>> event: OperationFailedvm's original host id: 136592 new host id: 136592
>> host id before state transition: 136589
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying
>> overprovisioning: 39900
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
>> applying overprovisioning: 11549697024
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) release cpu from host: 136589, old used:
>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
>> 39900; new used: 1995,reserved:0; movedfromreserved:
>> false,moveToReserveredfalse
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) release mem from host: 136589, old used:
>> 5368709120,reserved: 0, total: 11549697024; new used:
>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Response Received:
>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
>> 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
>> does not exist","wait":0}}] }
>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Unable to find listener.
>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for
>> VM 271, job origin: 4874
>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
>> instanceType: null, instanceId: null, cmd:
>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
>> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
>> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
>> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
>> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
>> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
>> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
>> origin:4874
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
>> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
>> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
>> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
>> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
>> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
>> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
>> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
>> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
>> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
>> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
>> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
>> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
>> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
>> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
>> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
>> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
>> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
>> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
>> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
>> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
>> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
>> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
>> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
>> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
>> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
>> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
>> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
>> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
>> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
>> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
>> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
>> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
>> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
>> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
>> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
>> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
>> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
>> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
>> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
>> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
>> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
>> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
>> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
>> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Publish async job-4875 complete on message bus
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Wake up jobs related to job-4875
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Update db status for job-4875
>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
>> job- 4875
>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Remove job-4875 from job monitoring
>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
>> exception while executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>> umeCmd
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
>> ],"errorcode":530,"errortext":"Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes. "}
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
>> async job-4874 complete on message bus
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>> jobs related to job-4874
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
>> status for job-4874
>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>> jobs joined with job-4874 and disjoin all subjobs created from job-
>> 4874
>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
>> executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>> umeCmd for job-4874
>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
>> job-4874 from job monitoring ^C
>>
>>
>> Regards,
>> IT engineer
>> Farheap, Russia
>> Ivan Derbenev
>>
>>
>>
>>
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: Xenserver 6.5 live migration with local volumes
Posted by Remi Bergsma <RB...@schubergphilis.com>.
FYI testing this patch as it seems related to the issue I observe: https://github.com/apache/cloudstack/pull/925
Regards, Remi
Sent from my iPhone
> On 26 Jun 2016, at 17:41, Remi Bergsma <RB...@schubergphilis.com> wrote:
>
> This is about xenserver 6.5.
>
> Regards, Remi
>
> Sent from my iPhone
>
>> On 26 Jun 2016, at 14:14, Will Stevens <wi...@gmail.com> wrote:
>>
>> Which hypervisor are you using?
>>> On Jun 26, 2016 2:00 AM, "Remi Bergsma" <RB...@schubergphilis.com> wrote:
>>>
>>> Hi Makrand,
>>>
>>> Thanks for the info. What I want is migrate a VM from one cluster (with
>>> cluster scope shared storage) to another one. The UI / API allows that (it
>>> says: storage migration required) but it doesn't work. The goal is to
>>> consolidate two half empty clusters into one and then reprovison the empty
>>> one.
>>>
>>> I was wondering whether others know this feature even works in which case
>>> my issue might be a local config issue or so.
>>>
>>> I'll try to reproduce in a test setup so that I can hook in the debugger
>>> and see what's going on.
>>>
>>> If there are tips please let me know :-)
>>>
>>> Thanks, Remi
>>>
>>> Sent from my iPhone
>>>
>>>> On 25 Jun 2016, at 17:41, Makrand <ma...@gmail.com> wrote:
>>>>
>>>> Hi Remi,
>>>>
>>>> If you're talking about common storage for all clusters, then its Zone
>>> wide
>>>> primary storage. I think it works pretty well, only if the new created VM
>>>> have disks already created on same new storage.
>>>>
>>>> With my previous employer, we rolled out zonewide primary (with cluster
>>>> level primary setup already in place) and it worked pretty good once we
>>>> vmotioned VMs from *cluster primary* to *Zone wide primary* and updated
>>> DB
>>>> (storage & volumes tables). (We eventually ditched Cluster primary).
>>>>
>>>> --
>>>> Best,
>>>> Makrand
>>>>
>>>>
>>>> On Fri, Jun 24, 2016 at 7:31 PM, Remi Bergsma <
>>> RBergsma@schubergphilis.com>
>>>> wrote:
>>>>
>>>>> Hi Anshul,
>>>>>
>>>>> Thanks! Did you ever try live migrating from CloudStack with storage
>>>>> between clusters? I can't get that to work. I'll collect some details in
>>>>> the next days but was wondering if it was supposed to work.
>>>>>
>>>>> Regards, Remi
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>>> On 24 Jun 2016, at 06:30, Anshul Gangwar <
>>> anshul.gangwar@accelerite.com>
>>>>>> wrote:
>>>>>>
>>>>>> Created the PR https://github.com/apache/cloudstack/pull/1596.
>>>>>>
>>>>>> Regards,
>>>>>> Anshul
>>>>>>
>>>>>> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
>>>>>>
>>>>>> Is there anybody that wants to take this on?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> - Dave
>>>>>>
>>>>>>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com>
>>>>> wrote:
>>>>>>>
>>>>>>> I updated the issue referenced in this email (
>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include
>>>>> CloudStack 4.8 as an affected version. If we could at least get someone
>>>>> assigned to it, that’d be a good start.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> - Dave
>>>>>>>
>>>>>>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <
>>>>> ivan.derbenev@tech-corps.com<ma...@tech-corps.com>>
>>> wrote:
>>>>>>>
>>>>>>> Is there a way I can fix this on my installation, without recompiling
>>>>> jar files?
>>>>>>>
>>>>>>> Regards,
>>>>>>> IT engineer
>>>>>>> Farheap, Russia
>>>>>>> Ivan Derbenev
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Koushik Das [mailto:koushik.das@accelerite.com]
>>>>>>> Sent: Tuesday, May 24, 2016 2:06 PM
>>>>>>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
>>>>>>> Subject: Re: Xenserver 6.5 live migration with local volumes
>>>>>>>
>>>>>>> This is a product bug and needs to be fixed. There is some
>>>>> serialization issue that needs fixing.
>>>>>>>
>>>>>>> In the below code volumeToFiler is null as Map is not a serializable
>>>>> type. It needs to be replaced with getVolumeToFilerAsList() and then the
>>>>> data needs to be read appropriately.
>>>>>>>
>>>>>>> final Map<VolumeTO, StorageFilerTO> volumeToFiler =
>>>>> command.getVolumeToFiler();
>>>>>>>
>>>>>>> -Koushik
>>>>>>>
>>>>>>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbenev@tech-corps.com
>>>>> <ma...@tech-corps.com>> wrote:
>>>>>>>
>>>>>>> Hello!
>>>>>>> After the migration onto XS 6.5 (and installing all the updates) and
>>> CS
>>>>>>> 4.8 I can't migrate VMs with Local Storage
>>>>>>>
>>>>>>> The issue is similar to this:
>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't
>>>>> have any solutions Any ideas?
>>>>>>>
>>>>>>> Errors:
>>>>>>>
>>>>>>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>>>>>>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with
>>> process
>>>>>>> of VM state report. host: 136592
>>>>>>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
>>>>>>> named cloud-main
>>>>>>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
>>>>>>> cloud-main on host=10.7.130.103;
>>>>>>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
>>>>>>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
>>>>>>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
>>>>>>> host 10.7.130.103 on device eth0
>>>>>>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
>>>>>>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
>>>>>>> 2016-05-24 12:10:17,170 WARN
>>>>>>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper]
>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception
>>>>> java.lang.NullPointerException. Storage motion failed due to
>>>>> java.lang.NullPointerException java.lang.NullPointerException
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>>> at
>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>>> 136589-479633360314957887: Response Received:
>>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>>>>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
>>>>>>> 128437433941265, via:
>>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>>> 110,
>>> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
>>>>>>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
>>>>>>> java.lang.NullPointerException\n\tat
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>>>>> andWrapper.java:86)\n\tat
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>>>>> andWrapper.java:54)\n\tat
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
>>>>>>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
>>>>>>> st(CitrixResourceBase.java:1677)\n\tat
>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
>>>>>>> tAttache.java:315)\n\tat
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
>>>>>>> gedContextRunnable.java:49)\n\tat
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
>>>>>>> (DefaultManagedContext.java:56)\n\tat
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>>>>>>> thContext(DefaultManagedContext.java:103)\n\tat
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>>>>>>> hContext(DefaultManagedContext.java:53)\n\tat
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>>>>>>> dContextRunnable.java:46)\n\tat
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
>>>>>>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<
>>>>> http://ScheduledFutureTask.ru>
>>>>>>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>>>>>>> va:1145)\n\tat
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>>>>>>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>>>>> 136589-479633360314957887: No more commands found
>>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
>>>>>>> MgmtId: 128437433941265, via:
>>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>>> 110, { MigrateWithStorageAnswer } }
>>>>>>> 2016-05-24 12:10:17,171 ERROR
>>>>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
>>>>>>> failed. Details: Exception: java.lang.NullPointerException
>>>>>>> Message: null
>>>>>>> Stack: java.lang.NullPointerException
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>>> at
>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 2016-05-24 12:10:17,171 ERROR
>>>>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) copy failed
>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
>>>>>>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>>>>>>> java.lang.NullPointerException
>>>>>>> Message: null
>>>>>>> Stack: java.lang.NullPointerException
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>>> at
>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> at
>>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
>>>>>>> at
>>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
>>>>>>> at
>>> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
>>>>>>> at
>>> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
>>>>>>> at
>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>>> at
>>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Unsupported data object (VOLUME,
>>>>>>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2
>>> ),
>>>>>>> no need to delete from object in store ref table
>>>>>>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
>>>>>>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
>>>>>>> while migrating the vm VM[User|i-3-271-VM] to host
>>>>>>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
>>>>>>> Message: null
>>>>>>> Stack: java.lang.NullPointerException
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>>> at
>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
>>>>>>> VM[User|i-3-271-VM]
>>>>>>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
>>>>>>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from
>>> host
>>>>>>> cloudstack-computing01-sar01.tech-corps.com<
>>>>> http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar
>>> and
>>>>> pod
>>>>>>> zone1.sar
>>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
>>>>>>> skipping send email
>>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
>>>>>>> 128437433941265
>>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
>>>>>>> MgmtId: 128437433941265, via:
>>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>>> 100011,
>>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
>>>>>>> MgmtId: 128437433941265, via:
>>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>>> 100011,
>>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>>>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
>>>>>>> 136589-479633360314957888: Executing request
>>>>>>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) VM state transitted from :Migrating to Running with
>>>>>>> event: OperationFailedvm's original host id: 136592 new host id:
>>> 136592
>>>>>>> host id before state transition: 136589
>>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after
>>> applying
>>>>>>> overprovisioning: 39900
>>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
>>>>>>> applying overprovisioning: 11549697024
>>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) release cpu from host: 136589, old used:
>>>>>>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
>>>>>>> 39900; new used: 1995,reserved:0; movedfromreserved:
>>>>>>> false,moveToReserveredfalse
>>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) release mem from host: 136589, old used:
>>>>>>> 5368709120,reserved: 0, total: 11549697024; new used:
>>>>>>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
>>>>>>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>>> 136589-479633360314957888: Response Received:
>>>>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
>>>>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>>>>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
>>>>>>> 128437433941265, via:
>>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
>>>>>>> does not exist","wait":0}}] }
>>>>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache]
>>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>>> 136589-479633360314957888: Unable to find listener.
>>>>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy]
>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>> (logid:8748bffe) Invocation exception, caused by:
>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy]
>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>> (logid:8748bffe) Rethrow exception
>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage
>>> for
>>>>>>> VM 271, job origin: 4874
>>>>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
>>>>>>> instanceType: null, instanceId: null, cmd:
>>>>>>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
>>> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
>>> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
>>> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
>>> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
>>> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
>>> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
>>>>>>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
>>>>>>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
>>>>>>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
>>>>>>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
>>>>>>> origin:4874
>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>>> at
>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>>> at
>>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
>>> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
>>> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
>>> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
>>> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
>>> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
>>> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
>>> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
>>> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
>>> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
>>> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
>>> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
>>> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
>>> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
>>> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
>>> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
>>> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
>>> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
>>> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
>>> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
>>> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
>>> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
>>> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
>>> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
>>> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
>>> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
>>> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
>>> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>>> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
>>> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
>>> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
>>> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
>>> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
>>> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
>>> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
>>> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
>>> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
>>> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
>>> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
>>> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
>>> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
>>> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
>>> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
>>> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
>>> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
>>> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
>>>>>>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
>>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Publish async job-4875 complete on message bus
>>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Wake up jobs related to job-4875
>>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Update db status for job-4875
>>>>>>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
>>>>>>> job- 4875
>>>>>>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
>>>>>>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Remove job-4875 from job monitoring
>>>>>>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
>>>>>>> exception while executing
>>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>>>>> umeCmd
>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>>> at
>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>>> at
>>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
>>>>>>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
>>>>>>> ],"errorcode":530,"errortext":"Failed to migrated vm
>>>>>>> VM[User|i-3-271-VM] along with its volumes. "}
>>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
>>>>>>> async job-4874 complete on message bus
>>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>>>>> jobs related to job-4874
>>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
>>>>>>> status for job-4874
>>>>>>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>>>>> jobs joined with job-4874 and disjoin all subjobs created from job-
>>>>>>> 4874
>>>>>>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
>>>>>>> executing
>>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>>>>> umeCmd for job-4874
>>>>>>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
>>>>>>> job-4874 from job monitoring ^C
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> IT engineer
>>>>>>> Farheap, Russia
>>>>>>> Ivan Derbenev
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> DISCLAIMER
>>>>>>> ==========
>>>>>>> This e-mail may contain privileged and confidential information which
>>>>> is the property of Accelerite, a Persistent Systems business. It is
>>>>> intended only for the use of the individual or entity to which it is
>>>>> addressed. If you are not the intended recipient, you are not
>>> authorized to
>>>>> read, retain, copy, print, distribute or use this message. If you have
>>>>> received this communication in error, please notify the sender and
>>> delete
>>>>> all copies of this message. Accelerite, a Persistent Systems business
>>> does
>>>>> not accept any liability for virus infected mails.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> DISCLAIMER
>>>>>> ==========
>>>>>> This e-mail may contain privileged and confidential information which
>>> is
>>>>> the property of Accelerite, a Persistent Systems business. It is
>>> intended
>>>>> only for the use of the individual or entity to which it is addressed.
>>> If
>>>>> you are not the intended recipient, you are not authorized to read,
>>> retain,
>>>>> copy, print, distribute or use this message. If you have received this
>>>>> communication in error, please notify the sender and delete all copies
>>> of
>>>>> this message. Accelerite, a Persistent Systems business does not accept
>>> any
>>>>> liability for virus infected mails.
>
Re: Xenserver 6.5 live migration with local volumes
Posted by Remi Bergsma <RB...@schubergphilis.com>.
FYI testing this patch as it seems related to the issue I observe: https://github.com/apache/cloudstack/pull/925
Regards, Remi
Sent from my iPhone
> On 26 Jun 2016, at 17:41, Remi Bergsma <RB...@schubergphilis.com> wrote:
>
> This is about xenserver 6.5.
>
> Regards, Remi
>
> Sent from my iPhone
>
>> On 26 Jun 2016, at 14:14, Will Stevens <wi...@gmail.com> wrote:
>>
>> Which hypervisor are you using?
>>> On Jun 26, 2016 2:00 AM, "Remi Bergsma" <RB...@schubergphilis.com> wrote:
>>>
>>> Hi Makrand,
>>>
>>> Thanks for the info. What I want is migrate a VM from one cluster (with
>>> cluster scope shared storage) to another one. The UI / API allows that (it
>>> says: storage migration required) but it doesn't work. The goal is to
>>> consolidate two half empty clusters into one and then reprovison the empty
>>> one.
>>>
>>> I was wondering whether others know this feature even works in which case
>>> my issue might be a local config issue or so.
>>>
>>> I'll try to reproduce in a test setup so that I can hook in the debugger
>>> and see what's going on.
>>>
>>> If there are tips please let me know :-)
>>>
>>> Thanks, Remi
>>>
>>> Sent from my iPhone
>>>
>>>> On 25 Jun 2016, at 17:41, Makrand <ma...@gmail.com> wrote:
>>>>
>>>> Hi Remi,
>>>>
>>>> If you're talking about common storage for all clusters, then its Zone
>>> wide
>>>> primary storage. I think it works pretty well, only if the new created VM
>>>> have disks already created on same new storage.
>>>>
>>>> With my previous employer, we rolled out zonewide primary (with cluster
>>>> level primary setup already in place) and it worked pretty good once we
>>>> vmotioned VMs from *cluster primary* to *Zone wide primary* and updated
>>> DB
>>>> (storage & volumes tables). (We eventually ditched Cluster primary).
>>>>
>>>> --
>>>> Best,
>>>> Makrand
>>>>
>>>>
>>>> On Fri, Jun 24, 2016 at 7:31 PM, Remi Bergsma <
>>> RBergsma@schubergphilis.com>
>>>> wrote:
>>>>
>>>>> Hi Anshul,
>>>>>
>>>>> Thanks! Did you ever try live migrating from CloudStack with storage
>>>>> between clusters? I can't get that to work. I'll collect some details in
>>>>> the next days but was wondering if it was supposed to work.
>>>>>
>>>>> Regards, Remi
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>>> On 24 Jun 2016, at 06:30, Anshul Gangwar <
>>> anshul.gangwar@accelerite.com>
>>>>>> wrote:
>>>>>>
>>>>>> Created the PR https://github.com/apache/cloudstack/pull/1596.
>>>>>>
>>>>>> Regards,
>>>>>> Anshul
>>>>>>
>>>>>> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
>>>>>>
>>>>>> Is there anybody that wants to take this on?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> - Dave
>>>>>>
>>>>>>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com>
>>>>> wrote:
>>>>>>>
>>>>>>> I updated the issue referenced in this email (
>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include
>>>>> CloudStack 4.8 as an affected version. If we could at least get someone
>>>>> assigned to it, that’d be a good start.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> - Dave
>>>>>>>
>>>>>>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <
>>>>> ivan.derbenev@tech-corps.com<ma...@tech-corps.com>>
>>> wrote:
>>>>>>>
>>>>>>> Is there a way I can fix this on my installation, without recompiling
>>>>> jar files?
>>>>>>>
>>>>>>> Regards,
>>>>>>> IT engineer
>>>>>>> Farheap, Russia
>>>>>>> Ivan Derbenev
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Koushik Das [mailto:koushik.das@accelerite.com]
>>>>>>> Sent: Tuesday, May 24, 2016 2:06 PM
>>>>>>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
>>>>>>> Subject: Re: Xenserver 6.5 live migration with local volumes
>>>>>>>
>>>>>>> This is a product bug and needs to be fixed. There is some
>>>>> serialization issue that needs fixing.
>>>>>>>
>>>>>>> In the below code volumeToFiler is null as Map is not a serializable
>>>>> type. It needs to be replaced with getVolumeToFilerAsList() and then the
>>>>> data needs to be read appropriately.
>>>>>>>
>>>>>>> final Map<VolumeTO, StorageFilerTO> volumeToFiler =
>>>>> command.getVolumeToFiler();
>>>>>>>
>>>>>>> -Koushik
>>>>>>>
>>>>>>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbenev@tech-corps.com
>>>>> <ma...@tech-corps.com>> wrote:
>>>>>>>
>>>>>>> Hello!
>>>>>>> After the migration onto XS 6.5 (and installing all the updates) and
>>> CS
>>>>>>> 4.8 I can't migrate VMs with Local Storage
>>>>>>>
>>>>>>> The issue is similar to this:
>>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't
>>>>> have any solutions Any ideas?
>>>>>>>
>>>>>>> Errors:
>>>>>>>
>>>>>>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>>>>>>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with
>>> process
>>>>>>> of VM state report. host: 136592
>>>>>>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
>>>>>>> named cloud-main
>>>>>>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
>>>>>>> cloud-main on host=10.7.130.103;
>>>>>>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
>>>>>>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
>>>>>>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
>>>>>>> host 10.7.130.103 on device eth0
>>>>>>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
>>>>>>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
>>>>>>> 2016-05-24 12:10:17,170 WARN
>>>>>>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper]
>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception
>>>>> java.lang.NullPointerException. Storage motion failed due to
>>>>> java.lang.NullPointerException java.lang.NullPointerException
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>>> at
>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>>> 136589-479633360314957887: Response Received:
>>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>>>>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
>>>>>>> 128437433941265, via:
>>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>>> 110,
>>> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
>>>>>>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
>>>>>>> java.lang.NullPointerException\n\tat
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>>>>> andWrapper.java:86)\n\tat
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>>>>> andWrapper.java:54)\n\tat
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
>>>>>>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
>>>>>>> st(CitrixResourceBase.java:1677)\n\tat
>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
>>>>>>> tAttache.java:315)\n\tat
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
>>>>>>> gedContextRunnable.java:49)\n\tat
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
>>>>>>> (DefaultManagedContext.java:56)\n\tat
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>>>>>>> thContext(DefaultManagedContext.java:103)\n\tat
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>>>>>>> hContext(DefaultManagedContext.java:53)\n\tat
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>>>>>>> dContextRunnable.java:46)\n\tat
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
>>>>>>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<
>>>>> http://ScheduledFutureTask.ru>
>>>>>>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>>>>>>> va:1145)\n\tat
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>>>>>>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
>>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>>>>> 136589-479633360314957887: No more commands found
>>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
>>>>>>> MgmtId: 128437433941265, via:
>>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>>> 110, { MigrateWithStorageAnswer } }
>>>>>>> 2016-05-24 12:10:17,171 ERROR
>>>>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
>>>>>>> failed. Details: Exception: java.lang.NullPointerException
>>>>>>> Message: null
>>>>>>> Stack: java.lang.NullPointerException
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>>> at
>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 2016-05-24 12:10:17,171 ERROR
>>>>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) copy failed
>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
>>>>>>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>>>>>>> java.lang.NullPointerException
>>>>>>> Message: null
>>>>>>> Stack: java.lang.NullPointerException
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>>> at
>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> at
>>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
>>>>>>> at
>>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
>>>>>>> at
>>> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
>>>>>>> at
>>> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
>>>>>>> at
>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>>> at
>>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Unsupported data object (VOLUME,
>>>>>>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2
>>> ),
>>>>>>> no need to delete from object in store ref table
>>>>>>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
>>>>>>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
>>>>>>> while migrating the vm VM[User|i-3-271-VM] to host
>>>>>>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
>>>>>>> Message: null
>>>>>>> Stack: java.lang.NullPointerException
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>>> at
>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>>> at
>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
>>>>>>> VM[User|i-3-271-VM]
>>>>>>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
>>>>>>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from
>>> host
>>>>>>> cloudstack-computing01-sar01.tech-corps.com<
>>>>> http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar
>>> and
>>>>> pod
>>>>>>> zone1.sar
>>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
>>>>>>> skipping send email
>>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
>>>>>>> 128437433941265
>>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
>>>>>>> MgmtId: 128437433941265, via:
>>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>>> 100011,
>>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
>>>>>>> MgmtId: 128437433941265, via:
>>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>>> 100011,
>>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>>>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
>>>>>>> 136589-479633360314957888: Executing request
>>>>>>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) VM state transitted from :Migrating to Running with
>>>>>>> event: OperationFailedvm's original host id: 136592 new host id:
>>> 136592
>>>>>>> host id before state transition: 136589
>>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after
>>> applying
>>>>>>> overprovisioning: 39900
>>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
>>>>>>> applying overprovisioning: 11549697024
>>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) release cpu from host: 136589, old used:
>>>>>>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
>>>>>>> 39900; new used: 1995,reserved:0; movedfromreserved:
>>>>>>> false,moveToReserveredfalse
>>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>>> (logid:8748bffe) release mem from host: 136589, old used:
>>>>>>> 5368709120,reserved: 0, total: 11549697024; new used:
>>>>>>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
>>>>>>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>>> 136589-479633360314957888: Response Received:
>>>>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
>>>>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>>>>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
>>>>>>> 128437433941265, via:
>>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
>>>>>>> does not exist","wait":0}}] }
>>>>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache]
>>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>>> 136589-479633360314957888: Unable to find listener.
>>>>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy]
>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>> (logid:8748bffe) Invocation exception, caused by:
>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy]
>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>> (logid:8748bffe) Rethrow exception
>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage
>>> for
>>>>>>> VM 271, job origin: 4874
>>>>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
>>>>>>> instanceType: null, instanceId: null, cmd:
>>>>>>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
>>> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
>>> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
>>> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
>>> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
>>> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
>>> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
>>>>>>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
>>>>>>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
>>>>>>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
>>>>>>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
>>>>>>> origin:4874
>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>>> at
>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>>> at
>>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
>>> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
>>> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
>>> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
>>> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
>>> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
>>> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
>>> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
>>> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
>>> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
>>> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
>>> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
>>> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
>>> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
>>> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
>>> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
>>> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
>>> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
>>> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
>>> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
>>> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
>>> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
>>> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
>>> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
>>> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
>>> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
>>> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
>>> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>>> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
>>> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
>>> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
>>> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
>>> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
>>> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
>>> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
>>> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
>>> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
>>> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
>>> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
>>> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
>>> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
>>> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
>>> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
>>> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
>>> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
>>> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
>>>>>>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
>>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Publish async job-4875 complete on message bus
>>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Wake up jobs related to job-4875
>>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Update db status for job-4875
>>>>>>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
>>>>>>> job- 4875
>>>>>>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
>>>>>>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>>> Remove job-4875 from job monitoring
>>>>>>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
>>>>>>> exception while executing
>>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>>>>> umeCmd
>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>>> at
>>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at
>>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>>> at
>>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>>> at
>>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>>> at
>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>>> at
>>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>>> at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
>>>>>>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
>>>>>>> ],"errorcode":530,"errortext":"Failed to migrated vm
>>>>>>> VM[User|i-3-271-VM] along with its volumes. "}
>>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
>>>>>>> async job-4874 complete on message bus
>>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>>>>> jobs related to job-4874
>>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
>>>>>>> status for job-4874
>>>>>>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>>>>> jobs joined with job-4874 and disjoin all subjobs created from job-
>>>>>>> 4874
>>>>>>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
>>>>>>> executing
>>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>>>>> umeCmd for job-4874
>>>>>>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
>>>>>>> job-4874 from job monitoring ^C
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> IT engineer
>>>>>>> Farheap, Russia
>>>>>>> Ivan Derbenev
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> DISCLAIMER
>>>>>>> ==========
>>>>>>> This e-mail may contain privileged and confidential information which
>>>>> is the property of Accelerite, a Persistent Systems business. It is
>>>>> intended only for the use of the individual or entity to which it is
>>>>> addressed. If you are not the intended recipient, you are not
>>> authorized to
>>>>> read, retain, copy, print, distribute or use this message. If you have
>>>>> received this communication in error, please notify the sender and
>>> delete
>>>>> all copies of this message. Accelerite, a Persistent Systems business
>>> does
>>>>> not accept any liability for virus infected mails.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> DISCLAIMER
>>>>>> ==========
>>>>>> This e-mail may contain privileged and confidential information which
>>> is
>>>>> the property of Accelerite, a Persistent Systems business. It is
>>> intended
>>>>> only for the use of the individual or entity to which it is addressed.
>>> If
>>>>> you are not the intended recipient, you are not authorized to read,
>>> retain,
>>>>> copy, print, distribute or use this message. If you have received this
>>>>> communication in error, please notify the sender and delete all copies
>>> of
>>>>> this message. Accelerite, a Persistent Systems business does not accept
>>> any
>>>>> liability for virus infected mails.
>
Re: Xenserver 6.5 live migration with local volumes
Posted by Remi Bergsma <RB...@schubergphilis.com>.
This is about xenserver 6.5.
Regards, Remi
Sent from my iPhone
> On 26 Jun 2016, at 14:14, Will Stevens <wi...@gmail.com> wrote:
>
> Which hypervisor are you using?
>> On Jun 26, 2016 2:00 AM, "Remi Bergsma" <RB...@schubergphilis.com> wrote:
>>
>> Hi Makrand,
>>
>> Thanks for the info. What I want is migrate a VM from one cluster (with
>> cluster scope shared storage) to another one. The UI / API allows that (it
>> says: storage migration required) but it doesn't work. The goal is to
>> consolidate two half empty clusters into one and then reprovison the empty
>> one.
>>
>> I was wondering whether others know this feature even works in which case
>> my issue might be a local config issue or so.
>>
>> I'll try to reproduce in a test setup so that I can hook in the debugger
>> and see what's going on.
>>
>> If there are tips please let me know :-)
>>
>> Thanks, Remi
>>
>> Sent from my iPhone
>>
>>> On 25 Jun 2016, at 17:41, Makrand <ma...@gmail.com> wrote:
>>>
>>> Hi Remi,
>>>
>>> If you're talking about common storage for all clusters, then its Zone
>> wide
>>> primary storage. I think it works pretty well, only if the new created VM
>>> have disks already created on same new storage.
>>>
>>> With my previous employer, we rolled out zonewide primary (with cluster
>>> level primary setup already in place) and it worked pretty good once we
>>> vmotioned VMs from *cluster primary* to *Zone wide primary* and updated
>> DB
>>> (storage & volumes tables). (We eventually ditched Cluster primary).
>>>
>>> --
>>> Best,
>>> Makrand
>>>
>>>
>>> On Fri, Jun 24, 2016 at 7:31 PM, Remi Bergsma <
>> RBergsma@schubergphilis.com>
>>> wrote:
>>>
>>>> Hi Anshul,
>>>>
>>>> Thanks! Did you ever try live migrating from CloudStack with storage
>>>> between clusters? I can't get that to work. I'll collect some details in
>>>> the next days but was wondering if it was supposed to work.
>>>>
>>>> Regards, Remi
>>>>
>>>> Sent from my iPhone
>>>>
>>>>>> On 24 Jun 2016, at 06:30, Anshul Gangwar <
>> anshul.gangwar@accelerite.com>
>>>>> wrote:
>>>>>
>>>>> Created the PR https://github.com/apache/cloudstack/pull/1596.
>>>>>
>>>>> Regards,
>>>>> Anshul
>>>>>
>>>>> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
>>>>>
>>>>> Is there anybody that wants to take this on?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> - Dave
>>>>>
>>>>>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com>
>>>> wrote:
>>>>>>
>>>>>> I updated the issue referenced in this email (
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include
>>>> CloudStack 4.8 as an affected version. If we could at least get someone
>>>> assigned to it, that’d be a good start.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> - Dave
>>>>>>
>>>>>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <
>>>> ivan.derbenev@tech-corps.com<ma...@tech-corps.com>>
>> wrote:
>>>>>>
>>>>>> Is there a way I can fix this on my installation, without recompiling
>>>> jar files?
>>>>>>
>>>>>> Regards,
>>>>>> IT engineer
>>>>>> Farheap, Russia
>>>>>> Ivan Derbenev
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Koushik Das [mailto:koushik.das@accelerite.com]
>>>>>> Sent: Tuesday, May 24, 2016 2:06 PM
>>>>>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
>>>>>> Subject: Re: Xenserver 6.5 live migration with local volumes
>>>>>>
>>>>>> This is a product bug and needs to be fixed. There is some
>>>> serialization issue that needs fixing.
>>>>>>
>>>>>> In the below code volumeToFiler is null as Map is not a serializable
>>>> type. It needs to be replaced with getVolumeToFilerAsList() and then the
>>>> data needs to be read appropriately.
>>>>>>
>>>>>> final Map<VolumeTO, StorageFilerTO> volumeToFiler =
>>>> command.getVolumeToFiler();
>>>>>>
>>>>>> -Koushik
>>>>>>
>>>>>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbenev@tech-corps.com
>>>> <ma...@tech-corps.com>> wrote:
>>>>>>
>>>>>> Hello!
>>>>>> After the migration onto XS 6.5 (and installing all the updates) and
>> CS
>>>>>> 4.8 I can't migrate VMs with Local Storage
>>>>>>
>>>>>> The issue is similar to this:
>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't
>>>> have any solutions Any ideas?
>>>>>>
>>>>>> Errors:
>>>>>>
>>>>>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>>>>>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with
>> process
>>>>>> of VM state report. host: 136592
>>>>>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
>>>>>> named cloud-main
>>>>>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
>>>>>> cloud-main on host=10.7.130.103;
>>>>>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
>>>>>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
>>>>>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
>>>>>> host 10.7.130.103 on device eth0
>>>>>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
>>>>>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
>>>>>> 2016-05-24 12:10:17,170 WARN
>>>>>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception
>>>> java.lang.NullPointerException. Storage motion failed due to
>>>> java.lang.NullPointerException java.lang.NullPointerException
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>> 136589-479633360314957887: Response Received:
>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>>>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
>>>>>> 128437433941265, via:
>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>> 110,
>> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
>>>>>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
>>>>>> java.lang.NullPointerException\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>>>> andWrapper.java:86)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>>>> andWrapper.java:54)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
>>>>>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
>>>>>> st(CitrixResourceBase.java:1677)\n\tat
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
>>>>>> tAttache.java:315)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
>>>>>> gedContextRunnable.java:49)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
>>>>>> (DefaultManagedContext.java:56)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>>>>>> thContext(DefaultManagedContext.java:103)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>>>>>> hContext(DefaultManagedContext.java:53)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>>>>>> dContextRunnable.java:46)\n\tat
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
>>>>>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<
>>>> http://ScheduledFutureTask.ru>
>>>>>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>>>>>> va:1145)\n\tat
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>>>>>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>>>> 136589-479633360314957887: No more commands found
>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
>>>>>> MgmtId: 128437433941265, via:
>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>> 110, { MigrateWithStorageAnswer } }
>>>>>> 2016-05-24 12:10:17,171 ERROR
>>>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
>>>>>> failed. Details: Exception: java.lang.NullPointerException
>>>>>> Message: null
>>>>>> Stack: java.lang.NullPointerException
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> 2016-05-24 12:10:17,171 ERROR
>>>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) copy failed
>>>>>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
>>>>>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>>>>>> java.lang.NullPointerException
>>>>>> Message: null
>>>>>> Stack: java.lang.NullPointerException
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> at
>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
>>>>>> at
>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
>>>>>> at
>> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
>>>>>> at
>> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
>>>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>> at
>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Unsupported data object (VOLUME,
>>>>>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2
>> ),
>>>>>> no need to delete from object in store ref table
>>>>>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
>>>>>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
>>>>>> while migrating the vm VM[User|i-3-271-VM] to host
>>>>>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
>>>>>> Message: null
>>>>>> Stack: java.lang.NullPointerException
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
>>>>>> VM[User|i-3-271-VM]
>>>>>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
>>>>>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from
>> host
>>>>>> cloudstack-computing01-sar01.tech-corps.com<
>>>> http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar
>> and
>>>> pod
>>>>>> zone1.sar
>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
>>>>>> skipping send email
>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
>>>>>> 128437433941265
>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
>>>>>> MgmtId: 128437433941265, via:
>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
>>>>>> MgmtId: 128437433941265, via:
>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
>>>>>> 136589-479633360314957888: Executing request
>>>>>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) VM state transitted from :Migrating to Running with
>>>>>> event: OperationFailedvm's original host id: 136592 new host id:
>> 136592
>>>>>> host id before state transition: 136589
>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after
>> applying
>>>>>> overprovisioning: 39900
>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
>>>>>> applying overprovisioning: 11549697024
>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) release cpu from host: 136589, old used:
>>>>>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
>>>>>> 39900; new used: 1995,reserved:0; movedfromreserved:
>>>>>> false,moveToReserveredfalse
>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) release mem from host: 136589, old used:
>>>>>> 5368709120,reserved: 0, total: 11549697024; new used:
>>>>>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
>>>>>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>> 136589-479633360314957888: Response Received:
>>>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
>>>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>>>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
>>>>>> 128437433941265, via:
>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
>>>>>> does not exist","wait":0}}] }
>>>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache]
>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>> 136589-479633360314957888: Unable to find listener.
>>>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Invocation exception, caused by:
>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Rethrow exception
>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage
>> for
>>>>>> VM 271, job origin: 4874
>>>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
>>>>>> instanceType: null, instanceId: null, cmd:
>>>>>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
>> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
>> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
>> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
>> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
>> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
>> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
>>>>>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
>>>>>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
>>>>>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
>>>>>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
>>>>>> origin:4874
>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>> at
>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
>> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
>> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
>> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
>> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
>> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
>> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
>> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
>> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
>> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
>> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
>> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
>> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
>> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
>> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
>> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
>> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
>> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
>> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
>> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
>> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
>> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
>> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
>> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
>> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
>> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
>> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
>> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
>> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
>> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
>> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
>> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
>> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
>> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
>> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
>> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
>> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
>> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
>> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
>> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
>> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
>> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
>> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
>> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
>> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
>>>>>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Publish async job-4875 complete on message bus
>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Wake up jobs related to job-4875
>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Update db status for job-4875
>>>>>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
>>>>>> job- 4875
>>>>>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
>>>>>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Remove job-4875 from job monitoring
>>>>>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
>>>>>> exception while executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>>>> umeCmd
>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>> at
>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
>>>>>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
>>>>>> ],"errorcode":530,"errortext":"Failed to migrated vm
>>>>>> VM[User|i-3-271-VM] along with its volumes. "}
>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
>>>>>> async job-4874 complete on message bus
>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>>>> jobs related to job-4874
>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
>>>>>> status for job-4874
>>>>>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>>>> jobs joined with job-4874 and disjoin all subjobs created from job-
>>>>>> 4874
>>>>>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
>>>>>> executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>>>> umeCmd for job-4874
>>>>>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
>>>>>> job-4874 from job monitoring ^C
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> IT engineer
>>>>>> Farheap, Russia
>>>>>> Ivan Derbenev
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> DISCLAIMER
>>>>>> ==========
>>>>>> This e-mail may contain privileged and confidential information which
>>>> is the property of Accelerite, a Persistent Systems business. It is
>>>> intended only for the use of the individual or entity to which it is
>>>> addressed. If you are not the intended recipient, you are not
>> authorized to
>>>> read, retain, copy, print, distribute or use this message. If you have
>>>> received this communication in error, please notify the sender and
>> delete
>>>> all copies of this message. Accelerite, a Persistent Systems business
>> does
>>>> not accept any liability for virus infected mails.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> DISCLAIMER
>>>>> ==========
>>>>> This e-mail may contain privileged and confidential information which
>> is
>>>> the property of Accelerite, a Persistent Systems business. It is
>> intended
>>>> only for the use of the individual or entity to which it is addressed.
>> If
>>>> you are not the intended recipient, you are not authorized to read,
>> retain,
>>>> copy, print, distribute or use this message. If you have received this
>>>> communication in error, please notify the sender and delete all copies
>> of
>>>> this message. Accelerite, a Persistent Systems business does not accept
>> any
>>>> liability for virus infected mails.
>>
>>
Re: Xenserver 6.5 live migration with local volumes
Posted by Remi Bergsma <RB...@schubergphilis.com>.
This is about xenserver 6.5.
Regards, Remi
Sent from my iPhone
> On 26 Jun 2016, at 14:14, Will Stevens <wi...@gmail.com> wrote:
>
> Which hypervisor are you using?
>> On Jun 26, 2016 2:00 AM, "Remi Bergsma" <RB...@schubergphilis.com> wrote:
>>
>> Hi Makrand,
>>
>> Thanks for the info. What I want is migrate a VM from one cluster (with
>> cluster scope shared storage) to another one. The UI / API allows that (it
>> says: storage migration required) but it doesn't work. The goal is to
>> consolidate two half empty clusters into one and then reprovison the empty
>> one.
>>
>> I was wondering whether others know this feature even works in which case
>> my issue might be a local config issue or so.
>>
>> I'll try to reproduce in a test setup so that I can hook in the debugger
>> and see what's going on.
>>
>> If there are tips please let me know :-)
>>
>> Thanks, Remi
>>
>> Sent from my iPhone
>>
>>> On 25 Jun 2016, at 17:41, Makrand <ma...@gmail.com> wrote:
>>>
>>> Hi Remi,
>>>
>>> If you're talking about common storage for all clusters, then its Zone
>> wide
>>> primary storage. I think it works pretty well, only if the new created VM
>>> have disks already created on same new storage.
>>>
>>> With my previous employer, we rolled out zonewide primary (with cluster
>>> level primary setup already in place) and it worked pretty good once we
>>> vmotioned VMs from *cluster primary* to *Zone wide primary* and updated
>> DB
>>> (storage & volumes tables). (We eventually ditched Cluster primary).
>>>
>>> --
>>> Best,
>>> Makrand
>>>
>>>
>>> On Fri, Jun 24, 2016 at 7:31 PM, Remi Bergsma <
>> RBergsma@schubergphilis.com>
>>> wrote:
>>>
>>>> Hi Anshul,
>>>>
>>>> Thanks! Did you ever try live migrating from CloudStack with storage
>>>> between clusters? I can't get that to work. I'll collect some details in
>>>> the next days but was wondering if it was supposed to work.
>>>>
>>>> Regards, Remi
>>>>
>>>> Sent from my iPhone
>>>>
>>>>>> On 24 Jun 2016, at 06:30, Anshul Gangwar <
>> anshul.gangwar@accelerite.com>
>>>>> wrote:
>>>>>
>>>>> Created the PR https://github.com/apache/cloudstack/pull/1596.
>>>>>
>>>>> Regards,
>>>>> Anshul
>>>>>
>>>>> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
>>>>>
>>>>> Is there anybody that wants to take this on?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> - Dave
>>>>>
>>>>>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com>
>>>> wrote:
>>>>>>
>>>>>> I updated the issue referenced in this email (
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include
>>>> CloudStack 4.8 as an affected version. If we could at least get someone
>>>> assigned to it, that’d be a good start.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> - Dave
>>>>>>
>>>>>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <
>>>> ivan.derbenev@tech-corps.com<ma...@tech-corps.com>>
>> wrote:
>>>>>>
>>>>>> Is there a way I can fix this on my installation, without recompiling
>>>> jar files?
>>>>>>
>>>>>> Regards,
>>>>>> IT engineer
>>>>>> Farheap, Russia
>>>>>> Ivan Derbenev
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Koushik Das [mailto:koushik.das@accelerite.com]
>>>>>> Sent: Tuesday, May 24, 2016 2:06 PM
>>>>>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
>>>>>> Subject: Re: Xenserver 6.5 live migration with local volumes
>>>>>>
>>>>>> This is a product bug and needs to be fixed. There is some
>>>> serialization issue that needs fixing.
>>>>>>
>>>>>> In the below code volumeToFiler is null as Map is not a serializable
>>>> type. It needs to be replaced with getVolumeToFilerAsList() and then the
>>>> data needs to be read appropriately.
>>>>>>
>>>>>> final Map<VolumeTO, StorageFilerTO> volumeToFiler =
>>>> command.getVolumeToFiler();
>>>>>>
>>>>>> -Koushik
>>>>>>
>>>>>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbenev@tech-corps.com
>>>> <ma...@tech-corps.com>> wrote:
>>>>>>
>>>>>> Hello!
>>>>>> After the migration onto XS 6.5 (and installing all the updates) and
>> CS
>>>>>> 4.8 I can't migrate VMs with Local Storage
>>>>>>
>>>>>> The issue is similar to this:
>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't
>>>> have any solutions Any ideas?
>>>>>>
>>>>>> Errors:
>>>>>>
>>>>>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>>>>>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with
>> process
>>>>>> of VM state report. host: 136592
>>>>>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
>>>>>> named cloud-main
>>>>>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
>>>>>> cloud-main on host=10.7.130.103;
>>>>>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
>>>>>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
>>>>>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
>>>>>> host 10.7.130.103 on device eth0
>>>>>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
>>>>>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
>>>>>> 2016-05-24 12:10:17,170 WARN
>>>>>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception
>>>> java.lang.NullPointerException. Storage motion failed due to
>>>> java.lang.NullPointerException java.lang.NullPointerException
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>> 136589-479633360314957887: Response Received:
>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>>>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
>>>>>> 128437433941265, via:
>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>> 110,
>> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
>>>>>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
>>>>>> java.lang.NullPointerException\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>>>> andWrapper.java:86)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>>>> andWrapper.java:54)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
>>>>>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
>>>>>> st(CitrixResourceBase.java:1677)\n\tat
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
>>>>>> tAttache.java:315)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
>>>>>> gedContextRunnable.java:49)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
>>>>>> (DefaultManagedContext.java:56)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>>>>>> thContext(DefaultManagedContext.java:103)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>>>>>> hContext(DefaultManagedContext.java:53)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>>>>>> dContextRunnable.java:46)\n\tat
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
>>>>>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<
>>>> http://ScheduledFutureTask.ru>
>>>>>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>>>>>> va:1145)\n\tat
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>>>>>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
>>>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>>>> 136589-479633360314957887: No more commands found
>>>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
>>>>>> MgmtId: 128437433941265, via:
>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>> 110, { MigrateWithStorageAnswer } }
>>>>>> 2016-05-24 12:10:17,171 ERROR
>>>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
>>>>>> failed. Details: Exception: java.lang.NullPointerException
>>>>>> Message: null
>>>>>> Stack: java.lang.NullPointerException
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> 2016-05-24 12:10:17,171 ERROR
>>>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) copy failed
>>>>>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
>>>>>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>>>>>> java.lang.NullPointerException
>>>>>> Message: null
>>>>>> Stack: java.lang.NullPointerException
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> at
>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
>>>>>> at
>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
>>>>>> at
>> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
>>>>>> at
>> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
>>>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>> at
>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Unsupported data object (VOLUME,
>>>>>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2
>> ),
>>>>>> no need to delete from object in store ref table
>>>>>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
>>>>>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
>>>>>> while migrating the vm VM[User|i-3-271-VM] to host
>>>>>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
>>>>>> Message: null
>>>>>> Stack: java.lang.NullPointerException
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
>>>>>> VM[User|i-3-271-VM]
>>>>>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
>>>>>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from
>> host
>>>>>> cloudstack-computing01-sar01.tech-corps.com<
>>>> http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar
>> and
>>>> pod
>>>>>> zone1.sar
>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
>>>>>> skipping send email
>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
>>>>>> 128437433941265
>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
>>>>>> MgmtId: 128437433941265, via:
>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
>>>>>> MgmtId: 128437433941265, via:
>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
>>>>>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
>>>>>> 136589-479633360314957888: Executing request
>>>>>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) VM state transitted from :Migrating to Running with
>>>>>> event: OperationFailedvm's original host id: 136592 new host id:
>> 136592
>>>>>> host id before state transition: 136589
>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after
>> applying
>>>>>> overprovisioning: 39900
>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
>>>>>> applying overprovisioning: 11549697024
>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) release cpu from host: 136589, old used:
>>>>>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
>>>>>> 39900; new used: 1995,reserved:0; movedfromreserved:
>>>>>> false,moveToReserveredfalse
>>>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>>>> (logid:8748bffe) release mem from host: 136589, old used:
>>>>>> 5368709120,reserved: 0, total: 11549697024; new used:
>>>>>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
>>>>>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>> 136589-479633360314957888: Response Received:
>>>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
>>>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>>>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
>>>>>> 128437433941265, via:
>>>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>>>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>>>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
>>>>>> does not exist","wait":0}}] }
>>>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache]
>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>> 136589-479633360314957888: Unable to find listener.
>>>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Invocation exception, caused by:
>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Rethrow exception
>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage
>> for
>>>>>> VM 271, job origin: 4874
>>>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
>>>>>> instanceType: null, instanceId: null, cmd:
>>>>>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
>> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
>> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
>> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
>> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
>> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
>> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
>>>>>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
>>>>>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
>>>>>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
>>>>>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
>>>>>> origin:4874
>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>> at
>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
>> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
>> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
>> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
>> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
>> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
>> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
>> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
>> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
>> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
>> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
>> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
>> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
>> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
>> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
>> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
>> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
>> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
>> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
>> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
>> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
>> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
>> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
>> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
>> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
>> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
>> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
>> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
>> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
>> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
>> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
>> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
>> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
>> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
>> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
>> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
>> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
>> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
>> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
>> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
>> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
>> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
>> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
>> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
>> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
>>>>>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Publish async job-4875 complete on message bus
>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Wake up jobs related to job-4875
>>>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Update db status for job-4875
>>>>>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
>>>>>> job- 4875
>>>>>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
>>>>>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>>>> Remove job-4875 from job monitoring
>>>>>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
>>>>>> exception while executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>>>> umeCmd
>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>>>> VM[User|i-3-271-VM] along with its volumes.
>>>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>>>> at
>>>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>>>> at
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
>>>>>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
>>>>>> ],"errorcode":530,"errortext":"Failed to migrated vm
>>>>>> VM[User|i-3-271-VM] along with its volumes. "}
>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
>>>>>> async job-4874 complete on message bus
>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>>>> jobs related to job-4874
>>>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
>>>>>> status for job-4874
>>>>>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>>>> jobs joined with job-4874 and disjoin all subjobs created from job-
>>>>>> 4874
>>>>>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
>>>>>> executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>>>> umeCmd for job-4874
>>>>>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
>>>>>> job-4874 from job monitoring ^C
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> IT engineer
>>>>>> Farheap, Russia
>>>>>> Ivan Derbenev
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> DISCLAIMER
>>>>>> ==========
>>>>>> This e-mail may contain privileged and confidential information which
>>>> is the property of Accelerite, a Persistent Systems business. It is
>>>> intended only for the use of the individual or entity to which it is
>>>> addressed. If you are not the intended recipient, you are not
>> authorized to
>>>> read, retain, copy, print, distribute or use this message. If you have
>>>> received this communication in error, please notify the sender and
>> delete
>>>> all copies of this message. Accelerite, a Persistent Systems business
>> does
>>>> not accept any liability for virus infected mails.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> DISCLAIMER
>>>>> ==========
>>>>> This e-mail may contain privileged and confidential information which
>> is
>>>> the property of Accelerite, a Persistent Systems business. It is
>> intended
>>>> only for the use of the individual or entity to which it is addressed.
>> If
>>>> you are not the intended recipient, you are not authorized to read,
>> retain,
>>>> copy, print, distribute or use this message. If you have received this
>>>> communication in error, please notify the sender and delete all copies
>> of
>>>> this message. Accelerite, a Persistent Systems business does not accept
>> any
>>>> liability for virus infected mails.
>>
>>
Re: Xenserver 6.5 live migration with local volumes
Posted by Will Stevens <wi...@gmail.com>.
Which hypervisor are you using?
On Jun 26, 2016 2:00 AM, "Remi Bergsma" <RB...@schubergphilis.com> wrote:
> Hi Makrand,
>
> Thanks for the info. What I want is migrate a VM from one cluster (with
> cluster scope shared storage) to another one. The UI / API allows that (it
> says: storage migration required) but it doesn't work. The goal is to
> consolidate two half empty clusters into one and then reprovison the empty
> one.
>
> I was wondering whether others know this feature even works in which case
> my issue might be a local config issue or so.
>
> I'll try to reproduce in a test setup so that I can hook in the debugger
> and see what's going on.
>
> If there are tips please let me know :-)
>
> Thanks, Remi
>
> Sent from my iPhone
>
> > On 25 Jun 2016, at 17:41, Makrand <ma...@gmail.com> wrote:
> >
> > Hi Remi,
> >
> > If you're talking about common storage for all clusters, then its Zone
> wide
> > primary storage. I think it works pretty well, only if the new created VM
> > have disks already created on same new storage.
> >
> > With my previous employer, we rolled out zonewide primary (with cluster
> > level primary setup already in place) and it worked pretty good once we
> > vmotioned VMs from *cluster primary* to *Zone wide primary* and updated
> DB
> > (storage & volumes tables). (We eventually ditched Cluster primary).
> >
> > --
> > Best,
> > Makrand
> >
> >
> > On Fri, Jun 24, 2016 at 7:31 PM, Remi Bergsma <
> RBergsma@schubergphilis.com>
> > wrote:
> >
> >> Hi Anshul,
> >>
> >> Thanks! Did you ever try live migrating from CloudStack with storage
> >> between clusters? I can't get that to work. I'll collect some details in
> >> the next days but was wondering if it was supposed to work.
> >>
> >> Regards, Remi
> >>
> >> Sent from my iPhone
> >>
> >>>> On 24 Jun 2016, at 06:30, Anshul Gangwar <
> anshul.gangwar@accelerite.com>
> >>> wrote:
> >>>
> >>> Created the PR https://github.com/apache/cloudstack/pull/1596.
> >>>
> >>> Regards,
> >>> Anshul
> >>>
> >>> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
> >>>
> >>> Is there anybody that wants to take this on?
> >>>
> >>> Thanks,
> >>>
> >>> - Dave
> >>>
> >>>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com>
> >> wrote:
> >>>>
> >>>> I updated the issue referenced in this email (
> >> https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include
> >> CloudStack 4.8 as an affected version. If we could at least get someone
> >> assigned to it, that’d be a good start.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> - Dave
> >>>>
> >>>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <
> >> ivan.derbenev@tech-corps.com<ma...@tech-corps.com>>
> wrote:
> >>>>
> >>>> Is there a way I can fix this on my installation, without recompiling
> >> jar files?
> >>>>
> >>>> Regards,
> >>>> IT engineer
> >>>> Farheap, Russia
> >>>> Ivan Derbenev
> >>>>
> >>>> -----Original Message-----
> >>>> From: Koushik Das [mailto:koushik.das@accelerite.com]
> >>>> Sent: Tuesday, May 24, 2016 2:06 PM
> >>>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
> >>>> Subject: Re: Xenserver 6.5 live migration with local volumes
> >>>>
> >>>> This is a product bug and needs to be fixed. There is some
> >> serialization issue that needs fixing.
> >>>>
> >>>> In the below code volumeToFiler is null as Map is not a serializable
> >> type. It needs to be replaced with getVolumeToFilerAsList() and then the
> >> data needs to be read appropriately.
> >>>>
> >>>> final Map<VolumeTO, StorageFilerTO> volumeToFiler =
> >> command.getVolumeToFiler();
> >>>>
> >>>> -Koushik
> >>>>
> >>>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbenev@tech-corps.com
> >> <ma...@tech-corps.com>> wrote:
> >>>>
> >>>> Hello!
> >>>> After the migration onto XS 6.5 (and installing all the updates) and
> CS
> >>>> 4.8 I can't migrate VMs with Local Storage
> >>>>
> >>>> The issue is similar to this:
> >>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't
> >> have any solutions Any ideas?
> >>>>
> >>>> Errors:
> >>>>
> >>>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> >>>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with
> process
> >>>> of VM state report. host: 136592
> >>>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
> >>>> named cloud-main
> >>>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
> >>>> cloud-main on host=10.7.130.103;
> >>>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
> >>>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
> >>>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
> >>>> host 10.7.130.103 on device eth0
> >>>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
> >>>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
> >>>> 2016-05-24 12:10:17,170 WARN
> >>>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper]
> >> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception
> >> java.lang.NullPointerException. Storage motion failed due to
> >> java.lang.NullPointerException java.lang.NullPointerException
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >>>> at
> >>
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache]
> >> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
> >> 136589-479633360314957887: Response Received:
> >>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
> >>>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
> >>>> 128437433941265, via:
> >>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
> >> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >>>> 110,
> >>>>
> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
> >>>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
> >>>> java.lang.NullPointerException\n\tat
> >>>>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
> >>>>
> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
> >>>> andWrapper.java:86)\n\tat
> >>>>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
> >>>>
> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
> >>>> andWrapper.java:54)\n\tat
> >>>>
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
> >>>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
> >>>>
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
> >>>> st(CitrixResourceBase.java:1677)\n\tat
> >>>>
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
> >>>> tAttache.java:315)\n\tat
> >>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
> >>>> gedContextRunnable.java:49)\n\tat
> >>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
> >>>> (DefaultManagedContext.java:56)\n\tat
> >>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
> >>>> thContext(DefaultManagedContext.java:103)\n\tat
> >>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
> >>>> hContext(DefaultManagedContext.java:53)\n\tat
> >>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
> >>>> dContextRunnable.java:46)\n\tat
> >>>>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
> >>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
> >>>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
> >>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<
> >> http://ScheduledFutureTask.ru>
> >>>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
> >>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
> >>>> va:1145)\n\tat
> >>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> >>>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
> >>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
> >>>> 136589-479633360314957887: No more commands found
> >>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
> >>>> MgmtId: 128437433941265, via:
> >>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
> >> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >>>> 110, { MigrateWithStorageAnswer } }
> >>>> 2016-05-24 12:10:17,171 ERROR
> >>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
> >>>> failed. Details: Exception: java.lang.NullPointerException
> >>>> Message: null
> >>>> Stack: java.lang.NullPointerException
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >>>> at
> >>
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> 2016-05-24 12:10:17,171 ERROR
> >>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) copy failed
> >>>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
> >>>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
> >>>> java.lang.NullPointerException
> >>>> Message: null
> >>>> Stack: java.lang.NullPointerException
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >>>> at
> >>
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> at
> >>
> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
> >>>> at
> >>
> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
> >>>> at
> >>
> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
> >>>> at
> >>
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
> >>>> at
> >>
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> at java.lang.reflect.Method.invoke(Method.java:606)
> >>>> at
> >>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> >>>> at
> >> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Unsupported data object (VOLUME,
> >>>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2
> ),
> >>>> no need to delete from object in store ref table
> >>>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
> >>>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
> >>>> while migrating the vm VM[User|i-3-271-VM] to host
> >>>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
> >>>> Message: null
> >>>> Stack: java.lang.NullPointerException
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >>>> at
> >>
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
> >>>> VM[User|i-3-271-VM]
> >>>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
> >>>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from
> host
> >>>> cloudstack-computing01-sar01.tech-corps.com<
> >> http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar
> and
> >> pod
> >>>> zone1.sar
> >>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
> >>>> skipping send email
> >>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
> >>>> 128437433941265
> >>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
> >>>> MgmtId: 128437433941265, via:
> >>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
> >> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >>>> 100011,
> >>>>
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
> >>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
> >>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
> >>>> MgmtId: 128437433941265, via:
> >>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
> >> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >>>> 100011,
> >>>>
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
> >>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
> >>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
> >>>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
> >>>> 136589-479633360314957888: Executing request
> >>>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) VM state transitted from :Migrating to Running with
> >>>> event: OperationFailedvm's original host id: 136592 new host id:
> 136592
> >>>> host id before state transition: 136589
> >>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after
> applying
> >>>> overprovisioning: 39900
> >>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
> >>>> applying overprovisioning: 11549697024
> >>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) release cpu from host: 136589, old used:
> >>>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
> >>>> 39900; new used: 1995,reserved:0; movedfromreserved:
> >>>> false,moveToReserveredfalse
> >>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) release mem from host: 136589, old used:
> >>>> 5368709120,reserved: 0, total: 11549697024; new used:
> >>>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
> >>>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache]
> >> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
> >> 136589-479633360314957888: Response Received:
> >>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
> >>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
> >>>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
> >>>> 128437433941265, via:
> >>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
> >> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >>>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
> >>>> does not exist","wait":0}}] }
> >>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache]
> >> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
> >> 136589-479633360314957888: Unable to find listener.
> >>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Invocation exception, caused by:
> >> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> >> VM[User|i-3-271-VM] along with its volumes.
> >>>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Rethrow exception
> >> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> >> VM[User|i-3-271-VM] along with its volumes.
> >>>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage
> for
> >>>> VM 271, job origin: 4874
> >>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
> >>>> instanceType: null, instanceId: null, cmd:
> >>>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
> >>>>
> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
> >>>>
> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
> >>>>
> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
> >>>>
> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
> >>>>
> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
> >>>>
> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
> >>>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
> >>>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
> >>>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
> >>>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
> >>>> origin:4874
> >>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> >> VM[User|i-3-271-VM] along with its volumes.
> >>>> at
> >>
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> at java.lang.reflect.Method.invoke(Method.java:606)
> >>>> at
> >>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> >>>> at
> >> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
> >>>>
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
> >>>>
> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
> >>>>
> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
> >>>>
> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
> >>>>
> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
> >>>>
> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
> >>>>
> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
> >>>>
> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
> >>>>
> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
> >>>>
> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
> >>>>
> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
> >>>>
> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
> >>>>
> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
> >>>>
> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
> >>>>
> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
> >>>>
> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
> >>>>
> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
> >>>>
> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
> >>>>
> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
> >>>>
> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
> >>>>
> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
> >>>>
> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
> >>>>
> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
> >>>>
> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
> >>>>
> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
> >>>>
> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
> >>>>
> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
> >>>>
> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
> >>>>
> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
> >>>>
> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
> >>>>
> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
> >>>>
> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
> >>>>
> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
> >>>>
> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
> >>>>
> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
> >>>>
> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
> >>>>
> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
> >>>>
> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
> >>>>
> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
> >>>>
> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
> >>>>
> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
> >>>>
> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
> >>>>
> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
> >>>>
> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
> >>>>
> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
> >>>>
> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
> >>>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
> >>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Publish async job-4875 complete on message bus
> >>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Wake up jobs related to job-4875
> >>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Update db status for job-4875
> >>>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
> >>>> job- 4875
> >>>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
> >>>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Remove job-4875 from job monitoring
> >>>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
> >>>> exception while executing
> >>>>
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
> >>>> umeCmd
> >>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> >> VM[User|i-3-271-VM] along with its volumes.
> >>>> at
> >>
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> at java.lang.reflect.Method.invoke(Method.java:606)
> >>>> at
> >>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> >>>> at
> >> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
> >>>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
> >>>>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
> >>>> ],"errorcode":530,"errortext":"Failed to migrated vm
> >>>> VM[User|i-3-271-VM] along with its volumes. "}
> >>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
> >>>> async job-4874 complete on message bus
> >>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
> >>>> jobs related to job-4874
> >>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
> >>>> status for job-4874
> >>>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
> >>>> jobs joined with job-4874 and disjoin all subjobs created from job-
> >>>> 4874
> >>>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
> >>>> executing
> >>>>
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
> >>>> umeCmd for job-4874
> >>>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
> >>>> job-4874 from job monitoring ^C
> >>>>
> >>>>
> >>>> Regards,
> >>>> IT engineer
> >>>> Farheap, Russia
> >>>> Ivan Derbenev
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> DISCLAIMER
> >>>> ==========
> >>>> This e-mail may contain privileged and confidential information which
> >> is the property of Accelerite, a Persistent Systems business. It is
> >> intended only for the use of the individual or entity to which it is
> >> addressed. If you are not the intended recipient, you are not
> authorized to
> >> read, retain, copy, print, distribute or use this message. If you have
> >> received this communication in error, please notify the sender and
> delete
> >> all copies of this message. Accelerite, a Persistent Systems business
> does
> >> not accept any liability for virus infected mails.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> DISCLAIMER
> >>> ==========
> >>> This e-mail may contain privileged and confidential information which
> is
> >> the property of Accelerite, a Persistent Systems business. It is
> intended
> >> only for the use of the individual or entity to which it is addressed.
> If
> >> you are not the intended recipient, you are not authorized to read,
> retain,
> >> copy, print, distribute or use this message. If you have received this
> >> communication in error, please notify the sender and delete all copies
> of
> >> this message. Accelerite, a Persistent Systems business does not accept
> any
> >> liability for virus infected mails.
> >>
> >>
>
>
Re: Xenserver 6.5 live migration with local volumes
Posted by Will Stevens <wi...@gmail.com>.
Which hypervisor are you using?
On Jun 26, 2016 2:00 AM, "Remi Bergsma" <RB...@schubergphilis.com> wrote:
> Hi Makrand,
>
> Thanks for the info. What I want is migrate a VM from one cluster (with
> cluster scope shared storage) to another one. The UI / API allows that (it
> says: storage migration required) but it doesn't work. The goal is to
> consolidate two half empty clusters into one and then reprovison the empty
> one.
>
> I was wondering whether others know this feature even works in which case
> my issue might be a local config issue or so.
>
> I'll try to reproduce in a test setup so that I can hook in the debugger
> and see what's going on.
>
> If there are tips please let me know :-)
>
> Thanks, Remi
>
> Sent from my iPhone
>
> > On 25 Jun 2016, at 17:41, Makrand <ma...@gmail.com> wrote:
> >
> > Hi Remi,
> >
> > If you're talking about common storage for all clusters, then its Zone
> wide
> > primary storage. I think it works pretty well, only if the new created VM
> > have disks already created on same new storage.
> >
> > With my previous employer, we rolled out zonewide primary (with cluster
> > level primary setup already in place) and it worked pretty good once we
> > vmotioned VMs from *cluster primary* to *Zone wide primary* and updated
> DB
> > (storage & volumes tables). (We eventually ditched Cluster primary).
> >
> > --
> > Best,
> > Makrand
> >
> >
> > On Fri, Jun 24, 2016 at 7:31 PM, Remi Bergsma <
> RBergsma@schubergphilis.com>
> > wrote:
> >
> >> Hi Anshul,
> >>
> >> Thanks! Did you ever try live migrating from CloudStack with storage
> >> between clusters? I can't get that to work. I'll collect some details in
> >> the next days but was wondering if it was supposed to work.
> >>
> >> Regards, Remi
> >>
> >> Sent from my iPhone
> >>
> >>>> On 24 Jun 2016, at 06:30, Anshul Gangwar <
> anshul.gangwar@accelerite.com>
> >>> wrote:
> >>>
> >>> Created the PR https://github.com/apache/cloudstack/pull/1596.
> >>>
> >>> Regards,
> >>> Anshul
> >>>
> >>> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
> >>>
> >>> Is there anybody that wants to take this on?
> >>>
> >>> Thanks,
> >>>
> >>> - Dave
> >>>
> >>>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com>
> >> wrote:
> >>>>
> >>>> I updated the issue referenced in this email (
> >> https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include
> >> CloudStack 4.8 as an affected version. If we could at least get someone
> >> assigned to it, that’d be a good start.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> - Dave
> >>>>
> >>>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <
> >> ivan.derbenev@tech-corps.com<ma...@tech-corps.com>>
> wrote:
> >>>>
> >>>> Is there a way I can fix this on my installation, without recompiling
> >> jar files?
> >>>>
> >>>> Regards,
> >>>> IT engineer
> >>>> Farheap, Russia
> >>>> Ivan Derbenev
> >>>>
> >>>> -----Original Message-----
> >>>> From: Koushik Das [mailto:koushik.das@accelerite.com]
> >>>> Sent: Tuesday, May 24, 2016 2:06 PM
> >>>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
> >>>> Subject: Re: Xenserver 6.5 live migration with local volumes
> >>>>
> >>>> This is a product bug and needs to be fixed. There is some
> >> serialization issue that needs fixing.
> >>>>
> >>>> In the below code volumeToFiler is null as Map is not a serializable
> >> type. It needs to be replaced with getVolumeToFilerAsList() and then the
> >> data needs to be read appropriately.
> >>>>
> >>>> final Map<VolumeTO, StorageFilerTO> volumeToFiler =
> >> command.getVolumeToFiler();
> >>>>
> >>>> -Koushik
> >>>>
> >>>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbenev@tech-corps.com
> >> <ma...@tech-corps.com>> wrote:
> >>>>
> >>>> Hello!
> >>>> After the migration onto XS 6.5 (and installing all the updates) and
> CS
> >>>> 4.8 I can't migrate VMs with Local Storage
> >>>>
> >>>> The issue is similar to this:
> >>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't
> >> have any solutions Any ideas?
> >>>>
> >>>> Errors:
> >>>>
> >>>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> >>>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with
> process
> >>>> of VM state report. host: 136592
> >>>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
> >>>> named cloud-main
> >>>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
> >>>> cloud-main on host=10.7.130.103;
> >>>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
> >>>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
> >>>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
> >>>> host 10.7.130.103 on device eth0
> >>>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
> >>>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
> >>>> 2016-05-24 12:10:17,170 WARN
> >>>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper]
> >> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception
> >> java.lang.NullPointerException. Storage motion failed due to
> >> java.lang.NullPointerException java.lang.NullPointerException
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >>>> at
> >>
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache]
> >> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
> >> 136589-479633360314957887: Response Received:
> >>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
> >>>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
> >>>> 128437433941265, via:
> >>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
> >> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >>>> 110,
> >>>>
> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
> >>>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
> >>>> java.lang.NullPointerException\n\tat
> >>>>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
> >>>>
> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
> >>>> andWrapper.java:86)\n\tat
> >>>>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
> >>>>
> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
> >>>> andWrapper.java:54)\n\tat
> >>>>
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
> >>>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
> >>>>
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
> >>>> st(CitrixResourceBase.java:1677)\n\tat
> >>>>
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
> >>>> tAttache.java:315)\n\tat
> >>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
> >>>> gedContextRunnable.java:49)\n\tat
> >>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
> >>>> (DefaultManagedContext.java:56)\n\tat
> >>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
> >>>> thContext(DefaultManagedContext.java:103)\n\tat
> >>>>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
> >>>> hContext(DefaultManagedContext.java:53)\n\tat
> >>>>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
> >>>> dContextRunnable.java:46)\n\tat
> >>>>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
> >>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
> >>>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
> >>>>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<
> >> http://ScheduledFutureTask.ru>
> >>>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
> >>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
> >>>> va:1145)\n\tat
> >>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> >>>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
> >>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
> >>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
> >>>> 136589-479633360314957887: No more commands found
> >>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
> >>>> MgmtId: 128437433941265, via:
> >>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
> >> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >>>> 110, { MigrateWithStorageAnswer } }
> >>>> 2016-05-24 12:10:17,171 ERROR
> >>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
> >>>> failed. Details: Exception: java.lang.NullPointerException
> >>>> Message: null
> >>>> Stack: java.lang.NullPointerException
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >>>> at
> >>
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> 2016-05-24 12:10:17,171 ERROR
> >>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) copy failed
> >>>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
> >>>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
> >>>> java.lang.NullPointerException
> >>>> Message: null
> >>>> Stack: java.lang.NullPointerException
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >>>> at
> >>
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> at
> >>
> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
> >>>> at
> >>
> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
> >>>> at
> >>
> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
> >>>> at
> >>
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
> >>>> at
> >>
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> at java.lang.reflect.Method.invoke(Method.java:606)
> >>>> at
> >>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> >>>> at
> >> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Unsupported data object (VOLUME,
> >>>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2
> ),
> >>>> no need to delete from object in store ref table
> >>>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
> >>>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
> >>>> while migrating the vm VM[User|i-3-271-VM] to host
> >>>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
> >>>> Message: null
> >>>> Stack: java.lang.NullPointerException
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >>>> at
> >>
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >>>> at
> >>
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >>>> at
> >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>>
> >>>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
> >>>> VM[User|i-3-271-VM]
> >>>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
> >>>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from
> host
> >>>> cloudstack-computing01-sar01.tech-corps.com<
> >> http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar
> and
> >> pod
> >>>> zone1.sar
> >>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
> >>>> skipping send email
> >>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
> >>>> 128437433941265
> >>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
> >>>> MgmtId: 128437433941265, via:
> >>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
> >> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >>>> 100011,
> >>>>
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
> >>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
> >>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
> >>>> MgmtId: 128437433941265, via:
> >>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
> >> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >>>> 100011,
> >>>>
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
> >>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
> >>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
> >>>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
> >>>> 136589-479633360314957888: Executing request
> >>>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) VM state transitted from :Migrating to Running with
> >>>> event: OperationFailedvm's original host id: 136592 new host id:
> 136592
> >>>> host id before state transition: 136589
> >>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after
> applying
> >>>> overprovisioning: 39900
> >>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
> >>>> applying overprovisioning: 11549697024
> >>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) release cpu from host: 136589, old used:
> >>>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
> >>>> 39900; new used: 1995,reserved:0; movedfromreserved:
> >>>> false,moveToReserveredfalse
> >>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >>>> (logid:8748bffe) release mem from host: 136589, old used:
> >>>> 5368709120,reserved: 0, total: 11549697024; new used:
> >>>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
> >>>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache]
> >> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
> >> 136589-479633360314957888: Response Received:
> >>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
> >>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
> >>>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
> >>>> 128437433941265, via:
> >>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
> >> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >>>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
> >>>> does not exist","wait":0}}] }
> >>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache]
> >> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
> >> 136589-479633360314957888: Unable to find listener.
> >>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Invocation exception, caused by:
> >> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> >> VM[User|i-3-271-VM] along with its volumes.
> >>>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Rethrow exception
> >> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> >> VM[User|i-3-271-VM] along with its volumes.
> >>>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage
> for
> >>>> VM 271, job origin: 4874
> >>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
> >>>> instanceType: null, instanceId: null, cmd:
> >>>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
> >>>>
> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
> >>>>
> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
> >>>>
> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
> >>>>
> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
> >>>>
> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
> >>>>
> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
> >>>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
> >>>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
> >>>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
> >>>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
> >>>> origin:4874
> >>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> >> VM[User|i-3-271-VM] along with its volumes.
> >>>> at
> >>
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> at java.lang.reflect.Method.invoke(Method.java:606)
> >>>> at
> >>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> >>>> at
> >> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
> >>>>
> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
> >>>>
> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
> >>>>
> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
> >>>>
> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
> >>>>
> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
> >>>>
> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
> >>>>
> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
> >>>>
> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
> >>>>
> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
> >>>>
> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
> >>>>
> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
> >>>>
> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
> >>>>
> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
> >>>>
> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
> >>>>
> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
> >>>>
> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
> >>>>
> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
> >>>>
> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
> >>>>
> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
> >>>>
> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
> >>>>
> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
> >>>>
> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
> >>>>
> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
> >>>>
> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
> >>>>
> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
> >>>>
> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
> >>>>
> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
> >>>>
> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
> >>>>
> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
> >>>>
> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
> >>>>
> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
> >>>>
> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
> >>>>
> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
> >>>>
> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
> >>>>
> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
> >>>>
> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
> >>>>
> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
> >>>>
> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
> >>>>
> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
> >>>>
> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
> >>>>
> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
> >>>>
> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
> >>>>
> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
> >>>>
> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
> >>>>
> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
> >>>>
> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
> >>>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
> >>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Publish async job-4875 complete on message bus
> >>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Wake up jobs related to job-4875
> >>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Update db status for job-4875
> >>>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
> >>>> job- 4875
> >>>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
> >>>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >>>> Remove job-4875 from job monitoring
> >>>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
> >>>> exception while executing
> >>>>
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
> >>>> umeCmd
> >>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> >> VM[User|i-3-271-VM] along with its volumes.
> >>>> at
> >>
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >>>> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >>>> at java.lang.reflect.Method.invoke(Method.java:606)
> >>>> at
> >>
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >>>> at
> >>
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> >>>> at
> >> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >>>> at
> >>
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >>>> at
> >>
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> >>>> at
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >>>> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>>> at java.lang.Thread.run(Thread.java:745)
> >>>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
> >>>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
> >>>>
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
> >>>> ],"errorcode":530,"errortext":"Failed to migrated vm
> >>>> VM[User|i-3-271-VM] along with its volumes. "}
> >>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
> >>>> async job-4874 complete on message bus
> >>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
> >>>> jobs related to job-4874
> >>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
> >>>> status for job-4874
> >>>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
> >>>> jobs joined with job-4874 and disjoin all subjobs created from job-
> >>>> 4874
> >>>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
> >>>> executing
> >>>>
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
> >>>> umeCmd for job-4874
> >>>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
> >>>> job-4874 from job monitoring ^C
> >>>>
> >>>>
> >>>> Regards,
> >>>> IT engineer
> >>>> Farheap, Russia
> >>>> Ivan Derbenev
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> DISCLAIMER
> >>>> ==========
> >>>> This e-mail may contain privileged and confidential information which
> >> is the property of Accelerite, a Persistent Systems business. It is
> >> intended only for the use of the individual or entity to which it is
> >> addressed. If you are not the intended recipient, you are not
> authorized to
> >> read, retain, copy, print, distribute or use this message. If you have
> >> received this communication in error, please notify the sender and
> delete
> >> all copies of this message. Accelerite, a Persistent Systems business
> does
> >> not accept any liability for virus infected mails.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> DISCLAIMER
> >>> ==========
> >>> This e-mail may contain privileged and confidential information which
> is
> >> the property of Accelerite, a Persistent Systems business. It is
> intended
> >> only for the use of the individual or entity to which it is addressed.
> If
> >> you are not the intended recipient, you are not authorized to read,
> retain,
> >> copy, print, distribute or use this message. If you have received this
> >> communication in error, please notify the sender and delete all copies
> of
> >> this message. Accelerite, a Persistent Systems business does not accept
> any
> >> liability for virus infected mails.
> >>
> >>
>
>
Re: Xenserver 6.5 live migration with local volumes
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Hi Makrand,
Thanks for the info. What I want is migrate a VM from one cluster (with cluster scope shared storage) to another one. The UI / API allows that (it says: storage migration required) but it doesn't work. The goal is to consolidate two half empty clusters into one and then reprovison the empty one.
I was wondering whether others know this feature even works in which case my issue might be a local config issue or so.
I'll try to reproduce in a test setup so that I can hook in the debugger and see what's going on.
If there are tips please let me know :-)
Thanks, Remi
Sent from my iPhone
> On 25 Jun 2016, at 17:41, Makrand <ma...@gmail.com> wrote:
>
> Hi Remi,
>
> If you're talking about common storage for all clusters, then its Zone wide
> primary storage. I think it works pretty well, only if the new created VM
> have disks already created on same new storage.
>
> With my previous employer, we rolled out zonewide primary (with cluster
> level primary setup already in place) and it worked pretty good once we
> vmotioned VMs from *cluster primary* to *Zone wide primary* and updated DB
> (storage & volumes tables). (We eventually ditched Cluster primary).
>
> --
> Best,
> Makrand
>
>
> On Fri, Jun 24, 2016 at 7:31 PM, Remi Bergsma <RB...@schubergphilis.com>
> wrote:
>
>> Hi Anshul,
>>
>> Thanks! Did you ever try live migrating from CloudStack with storage
>> between clusters? I can't get that to work. I'll collect some details in
>> the next days but was wondering if it was supposed to work.
>>
>> Regards, Remi
>>
>> Sent from my iPhone
>>
>>>> On 24 Jun 2016, at 06:30, Anshul Gangwar <an...@accelerite.com>
>>> wrote:
>>>
>>> Created the PR https://github.com/apache/cloudstack/pull/1596.
>>>
>>> Regards,
>>> Anshul
>>>
>>> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
>>>
>>> Is there anybody that wants to take this on?
>>>
>>> Thanks,
>>>
>>> - Dave
>>>
>>>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com>
>> wrote:
>>>>
>>>> I updated the issue referenced in this email (
>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include
>> CloudStack 4.8 as an affected version. If we could at least get someone
>> assigned to it, that’d be a good start.
>>>>
>>>> Thanks,
>>>>
>>>> - Dave
>>>>
>>>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <
>> ivan.derbenev@tech-corps.com<ma...@tech-corps.com>> wrote:
>>>>
>>>> Is there a way I can fix this on my installation, without recompiling
>> jar files?
>>>>
>>>> Regards,
>>>> IT engineer
>>>> Farheap, Russia
>>>> Ivan Derbenev
>>>>
>>>> -----Original Message-----
>>>> From: Koushik Das [mailto:koushik.das@accelerite.com]
>>>> Sent: Tuesday, May 24, 2016 2:06 PM
>>>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
>>>> Subject: Re: Xenserver 6.5 live migration with local volumes
>>>>
>>>> This is a product bug and needs to be fixed. There is some
>> serialization issue that needs fixing.
>>>>
>>>> In the below code volumeToFiler is null as Map is not a serializable
>> type. It needs to be replaced with getVolumeToFilerAsList() and then the
>> data needs to be read appropriately.
>>>>
>>>> final Map<VolumeTO, StorageFilerTO> volumeToFiler =
>> command.getVolumeToFiler();
>>>>
>>>> -Koushik
>>>>
>>>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbenev@tech-corps.com
>> <ma...@tech-corps.com>> wrote:
>>>>
>>>> Hello!
>>>> After the migration onto XS 6.5 (and installing all the updates) and CS
>>>> 4.8 I can't migrate VMs with Local Storage
>>>>
>>>> The issue is similar to this:
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't
>> have any solutions Any ideas?
>>>>
>>>> Errors:
>>>>
>>>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>>>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process
>>>> of VM state report. host: 136592
>>>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
>>>> named cloud-main
>>>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
>>>> cloud-main on host=10.7.130.103;
>>>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
>>>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
>>>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
>>>> host 10.7.130.103 on device eth0
>>>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
>>>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
>>>> 2016-05-24 12:10:17,170 WARN
>>>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception
>> java.lang.NullPointerException. Storage motion failed due to
>> java.lang.NullPointerException java.lang.NullPointerException
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>> 136589-479633360314957887: Response Received:
>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
>>>> 128437433941265, via:
>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>> 110,
>>>> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
>>>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
>>>> java.lang.NullPointerException\n\tat
>>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>>>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>> andWrapper.java:86)\n\tat
>>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>>>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>> andWrapper.java:54)\n\tat
>>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
>>>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
>>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
>>>> st(CitrixResourceBase.java:1677)\n\tat
>>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
>>>> tAttache.java:315)\n\tat
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
>>>> gedContextRunnable.java:49)\n\tat
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
>>>> (DefaultManagedContext.java:56)\n\tat
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>>>> thContext(DefaultManagedContext.java:103)\n\tat
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>>>> hContext(DefaultManagedContext.java:53)\n\tat
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>>>> dContextRunnable.java:46)\n\tat
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
>>>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<
>> http://ScheduledFutureTask.ru>
>>>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>>>> va:1145)\n\tat
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>>>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>> 136589-479633360314957887: No more commands found
>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
>>>> MgmtId: 128437433941265, via:
>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>> 110, { MigrateWithStorageAnswer } }
>>>> 2016-05-24 12:10:17,171 ERROR
>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
>>>> failed. Details: Exception: java.lang.NullPointerException
>>>> Message: null
>>>> Stack: java.lang.NullPointerException
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> 2016-05-24 12:10:17,171 ERROR
>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) copy failed
>>>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
>>>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>>>> java.lang.NullPointerException
>>>> Message: null
>>>> Stack: java.lang.NullPointerException
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> at
>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
>>>> at
>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
>>>> at
>> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
>>>> at
>> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>> at
>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Unsupported data object (VOLUME,
>>>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2),
>>>> no need to delete from object in store ref table
>>>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
>>>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
>>>> while migrating the vm VM[User|i-3-271-VM] to host
>>>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
>>>> Message: null
>>>> Stack: java.lang.NullPointerException
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
>>>> VM[User|i-3-271-VM]
>>>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
>>>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from host
>>>> cloudstack-computing01-sar01.tech-corps.com<
>> http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar and
>> pod
>>>> zone1.sar
>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
>>>> skipping send email
>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
>>>> 128437433941265
>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
>>>> MgmtId: 128437433941265, via:
>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>> 100011,
>>>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
>>>> MgmtId: 128437433941265, via:
>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>> 100011,
>>>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
>>>> 136589-479633360314957888: Executing request
>>>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) VM state transitted from :Migrating to Running with
>>>> event: OperationFailedvm's original host id: 136592 new host id: 136592
>>>> host id before state transition: 136589
>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying
>>>> overprovisioning: 39900
>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
>>>> applying overprovisioning: 11549697024
>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) release cpu from host: 136589, old used:
>>>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
>>>> 39900; new used: 1995,reserved:0; movedfromreserved:
>>>> false,moveToReserveredfalse
>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) release mem from host: 136589, old used:
>>>> 5368709120,reserved: 0, total: 11549697024; new used:
>>>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
>>>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache]
>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>> 136589-479633360314957888: Response Received:
>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
>>>> 128437433941265, via:
>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
>>>> does not exist","wait":0}}] }
>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache]
>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>> 136589-479633360314957888: Unable to find listener.
>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Invocation exception, caused by:
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes.
>>>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Rethrow exception
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes.
>>>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for
>>>> VM 271, job origin: 4874
>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
>>>> instanceType: null, instanceId: null, cmd:
>>>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
>>>> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
>>>> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
>>>> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
>>>> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
>>>> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
>>>> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
>>>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
>>>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
>>>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
>>>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
>>>> origin:4874
>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes.
>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>> at
>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
>>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
>>>> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
>>>> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
>>>> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
>>>> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
>>>> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
>>>> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
>>>> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
>>>> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
>>>> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
>>>> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
>>>> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
>>>> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
>>>> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
>>>> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
>>>> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
>>>> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
>>>> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
>>>> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
>>>> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
>>>> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
>>>> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
>>>> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
>>>> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
>>>> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
>>>> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
>>>> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
>>>> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>>>> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
>>>> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
>>>> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
>>>> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
>>>> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
>>>> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
>>>> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
>>>> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
>>>> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
>>>> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
>>>> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
>>>> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
>>>> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
>>>> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
>>>> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
>>>> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
>>>> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
>>>> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
>>>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Publish async job-4875 complete on message bus
>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Wake up jobs related to job-4875
>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Update db status for job-4875
>>>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
>>>> job- 4875
>>>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
>>>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Remove job-4875 from job monitoring
>>>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
>>>> exception while executing
>>>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>> umeCmd
>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes.
>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>> at
>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
>>>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
>>>> ],"errorcode":530,"errortext":"Failed to migrated vm
>>>> VM[User|i-3-271-VM] along with its volumes. "}
>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
>>>> async job-4874 complete on message bus
>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>> jobs related to job-4874
>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
>>>> status for job-4874
>>>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>> jobs joined with job-4874 and disjoin all subjobs created from job-
>>>> 4874
>>>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
>>>> executing
>>>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>> umeCmd for job-4874
>>>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
>>>> job-4874 from job monitoring ^C
>>>>
>>>>
>>>> Regards,
>>>> IT engineer
>>>> Farheap, Russia
>>>> Ivan Derbenev
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> DISCLAIMER
>>>> ==========
>>>> This e-mail may contain privileged and confidential information which
>> is the property of Accelerite, a Persistent Systems business. It is
>> intended only for the use of the individual or entity to which it is
>> addressed. If you are not the intended recipient, you are not authorized to
>> read, retain, copy, print, distribute or use this message. If you have
>> received this communication in error, please notify the sender and delete
>> all copies of this message. Accelerite, a Persistent Systems business does
>> not accept any liability for virus infected mails.
>>>
>>>
>>>
>>>
>>>
>>>
>>> DISCLAIMER
>>> ==========
>>> This e-mail may contain privileged and confidential information which is
>> the property of Accelerite, a Persistent Systems business. It is intended
>> only for the use of the individual or entity to which it is addressed. If
>> you are not the intended recipient, you are not authorized to read, retain,
>> copy, print, distribute or use this message. If you have received this
>> communication in error, please notify the sender and delete all copies of
>> this message. Accelerite, a Persistent Systems business does not accept any
>> liability for virus infected mails.
>>
>>
Re: Xenserver 6.5 live migration with local volumes
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Hi Makrand,
Thanks for the info. What I want is migrate a VM from one cluster (with cluster scope shared storage) to another one. The UI / API allows that (it says: storage migration required) but it doesn't work. The goal is to consolidate two half empty clusters into one and then reprovison the empty one.
I was wondering whether others know this feature even works in which case my issue might be a local config issue or so.
I'll try to reproduce in a test setup so that I can hook in the debugger and see what's going on.
If there are tips please let me know :-)
Thanks, Remi
Sent from my iPhone
> On 25 Jun 2016, at 17:41, Makrand <ma...@gmail.com> wrote:
>
> Hi Remi,
>
> If you're talking about common storage for all clusters, then its Zone wide
> primary storage. I think it works pretty well, only if the new created VM
> have disks already created on same new storage.
>
> With my previous employer, we rolled out zonewide primary (with cluster
> level primary setup already in place) and it worked pretty good once we
> vmotioned VMs from *cluster primary* to *Zone wide primary* and updated DB
> (storage & volumes tables). (We eventually ditched Cluster primary).
>
> --
> Best,
> Makrand
>
>
> On Fri, Jun 24, 2016 at 7:31 PM, Remi Bergsma <RB...@schubergphilis.com>
> wrote:
>
>> Hi Anshul,
>>
>> Thanks! Did you ever try live migrating from CloudStack with storage
>> between clusters? I can't get that to work. I'll collect some details in
>> the next days but was wondering if it was supposed to work.
>>
>> Regards, Remi
>>
>> Sent from my iPhone
>>
>>>> On 24 Jun 2016, at 06:30, Anshul Gangwar <an...@accelerite.com>
>>> wrote:
>>>
>>> Created the PR https://github.com/apache/cloudstack/pull/1596.
>>>
>>> Regards,
>>> Anshul
>>>
>>> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
>>>
>>> Is there anybody that wants to take this on?
>>>
>>> Thanks,
>>>
>>> - Dave
>>>
>>>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com>
>> wrote:
>>>>
>>>> I updated the issue referenced in this email (
>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include
>> CloudStack 4.8 as an affected version. If we could at least get someone
>> assigned to it, that’d be a good start.
>>>>
>>>> Thanks,
>>>>
>>>> - Dave
>>>>
>>>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <
>> ivan.derbenev@tech-corps.com<ma...@tech-corps.com>> wrote:
>>>>
>>>> Is there a way I can fix this on my installation, without recompiling
>> jar files?
>>>>
>>>> Regards,
>>>> IT engineer
>>>> Farheap, Russia
>>>> Ivan Derbenev
>>>>
>>>> -----Original Message-----
>>>> From: Koushik Das [mailto:koushik.das@accelerite.com]
>>>> Sent: Tuesday, May 24, 2016 2:06 PM
>>>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
>>>> Subject: Re: Xenserver 6.5 live migration with local volumes
>>>>
>>>> This is a product bug and needs to be fixed. There is some
>> serialization issue that needs fixing.
>>>>
>>>> In the below code volumeToFiler is null as Map is not a serializable
>> type. It needs to be replaced with getVolumeToFilerAsList() and then the
>> data needs to be read appropriately.
>>>>
>>>> final Map<VolumeTO, StorageFilerTO> volumeToFiler =
>> command.getVolumeToFiler();
>>>>
>>>> -Koushik
>>>>
>>>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbenev@tech-corps.com
>> <ma...@tech-corps.com>> wrote:
>>>>
>>>> Hello!
>>>> After the migration onto XS 6.5 (and installing all the updates) and CS
>>>> 4.8 I can't migrate VMs with Local Storage
>>>>
>>>> The issue is similar to this:
>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't
>> have any solutions Any ideas?
>>>>
>>>> Errors:
>>>>
>>>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>>>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process
>>>> of VM state report. host: 136592
>>>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
>>>> named cloud-main
>>>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
>>>> cloud-main on host=10.7.130.103;
>>>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
>>>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
>>>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
>>>> host 10.7.130.103 on device eth0
>>>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
>>>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
>>>> 2016-05-24 12:10:17,170 WARN
>>>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception
>> java.lang.NullPointerException. Storage motion failed due to
>> java.lang.NullPointerException java.lang.NullPointerException
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>> 136589-479633360314957887: Response Received:
>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
>>>> 128437433941265, via:
>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>> 110,
>>>> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
>>>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
>>>> java.lang.NullPointerException\n\tat
>>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>>>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>> andWrapper.java:86)\n\tat
>>>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>>>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>>>> andWrapper.java:54)\n\tat
>>>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
>>>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
>>>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
>>>> st(CitrixResourceBase.java:1677)\n\tat
>>>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
>>>> tAttache.java:315)\n\tat
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
>>>> gedContextRunnable.java:49)\n\tat
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
>>>> (DefaultManagedContext.java:56)\n\tat
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>>>> thContext(DefaultManagedContext.java:103)\n\tat
>>>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>>>> hContext(DefaultManagedContext.java:53)\n\tat
>>>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>>>> dContextRunnable.java:46)\n\tat
>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
>>>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<
>> http://ScheduledFutureTask.ru>
>>>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>>>> va:1145)\n\tat
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>>>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
>>>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>>>> 136589-479633360314957887: No more commands found
>>>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
>>>> MgmtId: 128437433941265, via:
>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>> 110, { MigrateWithStorageAnswer } }
>>>> 2016-05-24 12:10:17,171 ERROR
>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
>>>> failed. Details: Exception: java.lang.NullPointerException
>>>> Message: null
>>>> Stack: java.lang.NullPointerException
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> 2016-05-24 12:10:17,171 ERROR
>>>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) copy failed
>>>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
>>>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>>>> java.lang.NullPointerException
>>>> Message: null
>>>> Stack: java.lang.NullPointerException
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> at
>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
>>>> at
>> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
>>>> at
>> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
>>>> at
>> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>> at
>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Unsupported data object (VOLUME,
>>>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2),
>>>> no need to delete from object in store ref table
>>>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
>>>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
>>>> while migrating the vm VM[User|i-3-271-VM] to host
>>>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
>>>> Message: null
>>>> Stack: java.lang.NullPointerException
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>>>> at
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>>>> at
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>>>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
>>>> VM[User|i-3-271-VM]
>>>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
>>>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from host
>>>> cloudstack-computing01-sar01.tech-corps.com<
>> http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar and
>> pod
>>>> zone1.sar
>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
>>>> skipping send email
>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
>>>> 128437433941265
>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
>>>> MgmtId: 128437433941265, via:
>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>> 100011,
>>>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
>>>> MgmtId: 128437433941265, via:
>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>> 100011,
>>>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>>>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>>>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
>>>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
>>>> 136589-479633360314957888: Executing request
>>>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) VM state transitted from :Migrating to Running with
>>>> event: OperationFailedvm's original host id: 136592 new host id: 136592
>>>> host id before state transition: 136589
>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying
>>>> overprovisioning: 39900
>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
>>>> applying overprovisioning: 11549697024
>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) release cpu from host: 136589, old used:
>>>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
>>>> 39900; new used: 1995,reserved:0; movedfromreserved:
>>>> false,moveToReserveredfalse
>>>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>>>> (logid:8748bffe) release mem from host: 136589, old used:
>>>> 5368709120,reserved: 0, total: 11549697024; new used:
>>>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
>>>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache]
>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>> 136589-479633360314957888: Response Received:
>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
>>>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>>>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
>>>> 128437433941265, via:
>>>> 136589(cloudstack-computing03-sar01.tech-corps.com<
>> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>>>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
>>>> does not exist","wait":0}}] }
>>>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache]
>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>> 136589-479633360314957888: Unable to find listener.
>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Invocation exception, caused by:
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes.
>>>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Rethrow exception
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes.
>>>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for
>>>> VM 271, job origin: 4874
>>>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
>>>> instanceType: null, instanceId: null, cmd:
>>>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
>>>> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
>>>> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
>>>> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
>>>> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
>>>> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
>>>> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
>>>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
>>>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
>>>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
>>>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
>>>> origin:4874
>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes.
>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>> at
>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
>>>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
>>>> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
>>>> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
>>>> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
>>>> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
>>>> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
>>>> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
>>>> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
>>>> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
>>>> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
>>>> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
>>>> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
>>>> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
>>>> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
>>>> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
>>>> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
>>>> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
>>>> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
>>>> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
>>>> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
>>>> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
>>>> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
>>>> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
>>>> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
>>>> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
>>>> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
>>>> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
>>>> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>>>> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
>>>> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
>>>> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
>>>> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
>>>> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
>>>> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
>>>> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
>>>> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
>>>> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
>>>> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
>>>> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
>>>> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
>>>> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
>>>> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
>>>> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
>>>> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
>>>> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
>>>> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
>>>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Publish async job-4875 complete on message bus
>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Wake up jobs related to job-4875
>>>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Update db status for job-4875
>>>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
>>>> job- 4875
>>>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
>>>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>>>> Remove job-4875 from job monitoring
>>>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
>>>> exception while executing
>>>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>> umeCmd
>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes.
>>>> at
>> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>>>> at
>> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>>>> at
>> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>>>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
>>>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
>>>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
>>>> ],"errorcode":530,"errortext":"Failed to migrated vm
>>>> VM[User|i-3-271-VM] along with its volumes. "}
>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
>>>> async job-4874 complete on message bus
>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>> jobs related to job-4874
>>>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
>>>> status for job-4874
>>>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>>>> jobs joined with job-4874 and disjoin all subjobs created from job-
>>>> 4874
>>>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
>>>> executing
>>>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>>>> umeCmd for job-4874
>>>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>>>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
>>>> job-4874 from job monitoring ^C
>>>>
>>>>
>>>> Regards,
>>>> IT engineer
>>>> Farheap, Russia
>>>> Ivan Derbenev
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> DISCLAIMER
>>>> ==========
>>>> This e-mail may contain privileged and confidential information which
>> is the property of Accelerite, a Persistent Systems business. It is
>> intended only for the use of the individual or entity to which it is
>> addressed. If you are not the intended recipient, you are not authorized to
>> read, retain, copy, print, distribute or use this message. If you have
>> received this communication in error, please notify the sender and delete
>> all copies of this message. Accelerite, a Persistent Systems business does
>> not accept any liability for virus infected mails.
>>>
>>>
>>>
>>>
>>>
>>>
>>> DISCLAIMER
>>> ==========
>>> This e-mail may contain privileged and confidential information which is
>> the property of Accelerite, a Persistent Systems business. It is intended
>> only for the use of the individual or entity to which it is addressed. If
>> you are not the intended recipient, you are not authorized to read, retain,
>> copy, print, distribute or use this message. If you have received this
>> communication in error, please notify the sender and delete all copies of
>> this message. Accelerite, a Persistent Systems business does not accept any
>> liability for virus infected mails.
>>
>>
Re: Xenserver 6.5 live migration with local volumes
Posted by Makrand <ma...@gmail.com>.
Hi Remi,
If you're talking about common storage for all clusters, then its Zone wide
primary storage. I think it works pretty well, only if the new created VM
have disks already created on same new storage.
With my previous employer, we rolled out zonewide primary (with cluster
level primary setup already in place) and it worked pretty good once we
vmotioned VMs from *cluster primary* to *Zone wide primary* and updated DB
(storage & volumes tables). (We eventually ditched Cluster primary).
--
Best,
Makrand
On Fri, Jun 24, 2016 at 7:31 PM, Remi Bergsma <RB...@schubergphilis.com>
wrote:
> Hi Anshul,
>
> Thanks! Did you ever try live migrating from CloudStack with storage
> between clusters? I can't get that to work. I'll collect some details in
> the next days but was wondering if it was supposed to work.
>
> Regards, Remi
>
> Sent from my iPhone
>
> > On 24 Jun 2016, at 06:30, Anshul Gangwar <an...@accelerite.com>
> wrote:
> >
> > Created the PR https://github.com/apache/cloudstack/pull/1596.
> >
> > Regards,
> > Anshul
> >
> > On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
> >
> > Is there anybody that wants to take this on?
> >
> > Thanks,
> >
> > - Dave
> >
> >> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com>
> wrote:
> >>
> >> I updated the issue referenced in this email (
> https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include
> CloudStack 4.8 as an affected version. If we could at least get someone
> assigned to it, that’d be a good start.
> >>
> >> Thanks,
> >>
> >> - Dave
> >>
> >> On May 24, 2016, at 9:42 AM, Ivan Derbenev <
> ivan.derbenev@tech-corps.com<ma...@tech-corps.com>> wrote:
> >>
> >> Is there a way I can fix this on my installation, without recompiling
> jar files?
> >>
> >> Regards,
> >> IT engineer
> >> Farheap, Russia
> >> Ivan Derbenev
> >>
> >> -----Original Message-----
> >> From: Koushik Das [mailto:koushik.das@accelerite.com]
> >> Sent: Tuesday, May 24, 2016 2:06 PM
> >> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
> >> Subject: Re: Xenserver 6.5 live migration with local volumes
> >>
> >> This is a product bug and needs to be fixed. There is some
> serialization issue that needs fixing.
> >>
> >> In the below code volumeToFiler is null as Map is not a serializable
> type. It needs to be replaced with getVolumeToFilerAsList() and then the
> data needs to be read appropriately.
> >>
> >> final Map<VolumeTO, StorageFilerTO> volumeToFiler =
> command.getVolumeToFiler();
> >>
> >> -Koushik
> >>
> >> On 24/05/16, 3:23 PM, "Ivan Derbenev" <ivan.derbenev@tech-corps.com
> <ma...@tech-corps.com>> wrote:
> >>
> >> Hello!
> >> After the migration onto XS 6.5 (and installing all the updates) and CS
> >> 4.8 I can't migrate VMs with Local Storage
> >>
> >> The issue is similar to this:
> >> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't
> have any solutions Any ideas?
> >>
> >> Errors:
> >>
> >> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
> >> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process
> >> of VM state report. host: 136592
> >> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
> >> named cloud-main
> >> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
> >> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
> >> cloud-main on host=10.7.130.103;
> >> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
> >> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
> >> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
> >> host 10.7.130.103 on device eth0
> >> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
> >> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
> >> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
> >> 2016-05-24 12:10:17,170 WARN
> >> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper]
> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception
> java.lang.NullPointerException. Storage motion failed due to
> java.lang.NullPointerException java.lang.NullPointerException
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:745)
> >> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
> 136589-479633360314957887: Response Received:
> >> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
> >> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
> >> 136589-479633360314957887: Processing: { Ans: , MgmtId:
> >> 128437433941265, via:
> >> 136589(cloudstack-computing03-sar01.tech-corps.com<
> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >> 110,
> >> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
> >> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
> >> java.lang.NullPointerException\n\tat
> >> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
> >> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
> >> andWrapper.java:86)\n\tat
> >> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
> >> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
> >> andWrapper.java:54)\n\tat
> >> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
> >> apper.execute(CitrixRequestWrapper.java:122)\n\tat
> >> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
> >> st(CitrixResourceBase.java:1677)\n\tat
> >> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
> >> tAttache.java:315)\n\tat
> >> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
> >> gedContextRunnable.java:49)\n\tat
> >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
> >> (DefaultManagedContext.java:56)\n\tat
> >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
> >> thContext(DefaultManagedContext.java:103)\n\tat
> >> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
> >> hContext(DefaultManagedContext.java:53)\n\tat
> >> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
> >> dContextRunnable.java:46)\n\tat
> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
> >> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<
> http://ScheduledFutureTask.ru>
> >> n(ScheduledThreadPoolExecutor.java:292)\n\tat
> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
> >> va:1145)\n\tat
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
> >> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
> >> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
> >> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
> >> 136589-479633360314957887: No more commands found
> >> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
> >> MgmtId: 128437433941265, via:
> >> 136589(cloudstack-computing03-sar01.tech-corps.com<
> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >> 110, { MigrateWithStorageAnswer } }
> >> 2016-05-24 12:10:17,171 ERROR
> >> [o.a.c.s.m.XenServerStorageMotionStrategy]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
> >> failed. Details: Exception: java.lang.NullPointerException
> >> Message: null
> >> Stack: java.lang.NullPointerException
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> 2016-05-24 12:10:17,171 ERROR
> >> [o.a.c.s.m.XenServerStorageMotionStrategy]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) copy failed
> >> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
> >> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
> >> java.lang.NullPointerException
> >> Message: null
> >> Stack: java.lang.NullPointerException
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> at
> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
> >> at
> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
> >> at
> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
> >> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
> >> at
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
> >> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> >> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> >> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> >> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:745)
> >> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Unsupported data object (VOLUME,
> >> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2),
> >> no need to delete from object in store ref table
> >> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
> >> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
> >> while migrating the vm VM[User|i-3-271-VM] to host
> >> Host[-136589-Routing]. Exception: java.lang.NullPointerException
> >> Message: null
> >> Stack: java.lang.NullPointerException
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> >> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> >> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> >> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> >> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> >> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:745)
> >>
> >> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
> >> VM[User|i-3-271-VM]
> >> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
> >> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from host
> >> cloudstack-computing01-sar01.tech-corps.com<
> http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar and
> pod
> >> zone1.sar
> >> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
> >> skipping send email
> >> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
> >> 128437433941265
> >> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
> >> MgmtId: 128437433941265, via:
> >> 136589(cloudstack-computing03-sar01.tech-corps.com<
> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >> 100011,
> >> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
> >> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
> >> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
> >> MgmtId: 128437433941265, via:
> >> 136589(cloudstack-computing03-sar01.tech-corps.com<
> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >> 100011,
> >> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
> >> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
> >> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
> >> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
> >> 136589-479633360314957888: Executing request
> >> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) VM state transitted from :Migrating to Running with
> >> event: OperationFailedvm's original host id: 136592 new host id: 136592
> >> host id before state transition: 136589
> >> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying
> >> overprovisioning: 39900
> >> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
> >> applying overprovisioning: 11549697024
> >> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) release cpu from host: 136589, old used:
> >> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
> >> 39900; new used: 1995,reserved:0; movedfromreserved:
> >> false,moveToReserveredfalse
> >> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> >> (logid:8748bffe) release mem from host: 136589, old used:
> >> 5368709120,reserved: 0, total: 11549697024; new used:
> >> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
> >> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
> 136589-479633360314957888: Response Received:
> >> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
> >> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
> >> 136589-479633360314957888: Processing: { Ans: , MgmtId:
> >> 128437433941265, via:
> >> 136589(cloudstack-computing03-sar01.tech-corps.com<
> http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
> >> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
> >> does not exist","wait":0}}] }
> >> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache]
> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
> 136589-479633360314957888: Unable to find listener.
> >> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Invocation exception, caused by:
> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> VM[User|i-3-271-VM] along with its volumes.
> >> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
> (logid:8748bffe) Rethrow exception
> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> VM[User|i-3-271-VM] along with its volumes.
> >> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for
> >> VM 271, job origin: 4874
> >> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
> >> instanceType: null, instanceId: null, cmd:
> >> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
> >> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
> >> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
> >> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
> >> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
> >> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
> >> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
> >> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
> >> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
> >> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
> >> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
> >> origin:4874
> >> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> VM[User|i-3-271-VM] along with its volumes.
> >> at
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
> >> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> >> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> >> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> >> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:745)
> >> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
> >> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
> >> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
> >> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
> >> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
> >> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
> >> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
> >> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
> >> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
> >> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
> >> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
> >> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
> >> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
> >> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
> >> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
> >> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
> >> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
> >> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
> >> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
> >> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
> >> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
> >> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
> >> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
> >> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
> >> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
> >> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
> >> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
> >> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
> >> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
> >> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
> >> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
> >> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
> >> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
> >> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
> >> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
> >> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
> >> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
> >> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
> >> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
> >> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
> >> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
> >> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
> >> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
> >> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
> >> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
> >> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
> >> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
> >> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
> >> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >> Publish async job-4875 complete on message bus
> >> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >> Wake up jobs related to job-4875
> >> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >> Update db status for job-4875
> >> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >> Wake up jobs joined with job-4875 and disjoin all subjobs created from
> >> job- 4875
> >> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
> >> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
> >> Remove job-4875 from job monitoring
> >> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
> >> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
> >> exception while executing
> >> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
> >> umeCmd
> >> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> VM[User|i-3-271-VM] along with its volumes.
> >> at
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
> >> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> >> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> >> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> >> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> >> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> >> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> >> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> >> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> >> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:745)
> >> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
> >> async job-4874, jobStatus: FAILED, resultCode: 530, result:
> >> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
> >> ],"errorcode":530,"errortext":"Failed to migrated vm
> >> VM[User|i-3-271-VM] along with its volumes. "}
> >> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
> >> async job-4874 complete on message bus
> >> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
> >> jobs related to job-4874
> >> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
> >> status for job-4874
> >> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
> >> jobs joined with job-4874 and disjoin all subjobs created from job-
> >> 4874
> >> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
> >> executing
> >> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
> >> umeCmd for job-4874
> >> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> >> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
> >> job-4874 from job monitoring ^C
> >>
> >>
> >> Regards,
> >> IT engineer
> >> Farheap, Russia
> >> Ivan Derbenev
> >>
> >>
> >>
> >>
> >>
> >> DISCLAIMER
> >> ==========
> >> This e-mail may contain privileged and confidential information which
> is the property of Accelerite, a Persistent Systems business. It is
> intended only for the use of the individual or entity to which it is
> addressed. If you are not the intended recipient, you are not authorized to
> read, retain, copy, print, distribute or use this message. If you have
> received this communication in error, please notify the sender and delete
> all copies of this message. Accelerite, a Persistent Systems business does
> not accept any liability for virus infected mails.
> >
> >
> >
> >
> >
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>
>
Re: Xenserver 6.5 live migration with local volumes
Posted by Anshul Gangwar <an...@accelerite.com>.
Hi Remi,
For me it’s (migration across clusters with storage) working fine but am on internal branch. There could be some regressions due to recent refactors. As far as this patch goes this handles with cluster case.
PR https://github.com/apache/cloudstack/pull/925 is no more valid after fixing of https://issues.apache.org/jira/browse/CLOUDSTACK-9231. This patch handles the broken scenario.
Regards,
Anshul
On 24/06/16, 7:31 PM, "Remi Bergsma" <RB...@schubergphilis.com> wrote:
Hi Anshul,
Thanks! Did you ever try live migrating from CloudStack with storage between clusters? I can't get that to work. I'll collect some details in the next days but was wondering if it was supposed to work.
Regards, Remi
Sent from my iPhone
> On 24 Jun 2016, at 06:30, Anshul Gangwar <an...@accelerite.com> wrote:
>
> Created the PR https://github.com/apache/cloudstack/pull/1596.
>
> Regards,
> Anshul
>
> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
>
> Is there anybody that wants to take this on?
>
> Thanks,
>
> - Dave
>
>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com> wrote:
>>
>> I updated the issue referenced in this email (https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include CloudStack 4.8 as an affected version. If we could at least get someone assigned to it, that’d be a good start.
>>
>> Thanks,
>>
>> - Dave
>>
>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <iv...@tech-corps.com>> wrote:
>>
>> Is there a way I can fix this on my installation, without recompiling jar files?
>>
>> Regards,
>> IT engineer
>> Farheap, Russia
>> Ivan Derbenev
>>
>> -----Original Message-----
>> From: Koushik Das [mailto:koushik.das@accelerite.com]
>> Sent: Tuesday, May 24, 2016 2:06 PM
>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
>> Subject: Re: Xenserver 6.5 live migration with local volumes
>>
>> This is a product bug and needs to be fixed. There is some serialization issue that needs fixing.
>>
>> In the below code volumeToFiler is null as Map is not a serializable type. It needs to be replaced with getVolumeToFilerAsList() and then the data needs to be read appropriately.
>>
>> final Map<VolumeTO, StorageFilerTO> volumeToFiler = command.getVolumeToFiler();
>>
>> -Koushik
>>
>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <iv...@tech-corps.com>> wrote:
>>
>> Hello!
>> After the migration onto XS 6.5 (and installing all the updates) and CS
>> 4.8 I can't migrate VMs with Local Storage
>>
>> The issue is similar to this:
>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't have any solutions Any ideas?
>>
>> Errors:
>>
>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process
>> of VM state report. host: 136592
>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
>> named cloud-main
>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
>> cloud-main on host=10.7.130.103;
>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
>> host 10.7.130.103 on device eth0
>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
>> 2016-05-24 12:10:17,170 WARN
>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception java.lang.NullPointerException. Storage motion failed due to java.lang.NullPointerException java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq 136589-479633360314957887: Response Received:
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
>> 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 110,
>> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
>> java.lang.NullPointerException\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>> andWrapper.java:86)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>> andWrapper.java:54)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
>> st(CitrixResourceBase.java:1677)\n\tat
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
>> tAttache.java:315)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
>> gedContextRunnable.java:49)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
>> (DefaultManagedContext.java:56)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>> thContext(DefaultManagedContext.java:103)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>> hContext(DefaultManagedContext.java:53)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>> dContextRunnable.java:46)\n\tat
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<http://ScheduledFutureTask.ru>
>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>> va:1145)\n\tat
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>> 136589-479633360314957887: No more commands found
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 110, { MigrateWithStorageAnswer } }
>> 2016-05-24 12:10:17,171 ERROR
>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
>> failed. Details: Exception: java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 2016-05-24 12:10:17,171 ERROR
>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) copy failed
>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>> java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
>> at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
>> at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
>> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Unsupported data object (VOLUME,
>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2),
>> no need to delete from object in store ref table
>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
>> while migrating the vm VM[User|i-3-271-VM] to host
>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
>> VM[User|i-3-271-VM]
>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from host
>> cloudstack-computing01-sar01.tech-corps.com<http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar and pod
>> zone1.sar
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
>> skipping send email
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
>> 128437433941265
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
>> 136589-479633360314957888: Executing request
>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) VM state transitted from :Migrating to Running with
>> event: OperationFailedvm's original host id: 136592 new host id: 136592
>> host id before state transition: 136589
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying
>> overprovisioning: 39900
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
>> applying overprovisioning: 11549697024
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) release cpu from host: 136589, old used:
>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
>> 39900; new used: 1995,reserved:0; movedfromreserved:
>> false,moveToReserveredfalse
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) release mem from host: 136589, old used:
>> 5368709120,reserved: 0, total: 11549697024; new used:
>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Response Received:
>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
>> 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
>> does not exist","wait":0}}] }
>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Unable to find listener.
>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for
>> VM 271, job origin: 4874
>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
>> instanceType: null, instanceId: null, cmd:
>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
>> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
>> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
>> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
>> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
>> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
>> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
>> origin:4874
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
>> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
>> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
>> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
>> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
>> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
>> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
>> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
>> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
>> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
>> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
>> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
>> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
>> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
>> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
>> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
>> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
>> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
>> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
>> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
>> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
>> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
>> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
>> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
>> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
>> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
>> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
>> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
>> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
>> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
>> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
>> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
>> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
>> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
>> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
>> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
>> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
>> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
>> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
>> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
>> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
>> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
>> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
>> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
>> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Publish async job-4875 complete on message bus
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Wake up jobs related to job-4875
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Update db status for job-4875
>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
>> job- 4875
>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Remove job-4875 from job monitoring
>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
>> exception while executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>> umeCmd
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
>> ],"errorcode":530,"errortext":"Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes. "}
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
>> async job-4874 complete on message bus
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>> jobs related to job-4874
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
>> status for job-4874
>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>> jobs joined with job-4874 and disjoin all subjobs created from job-
>> 4874
>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
>> executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>> umeCmd for job-4874
>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
>> job-4874 from job monitoring ^C
>>
>>
>> Regards,
>> IT engineer
>> Farheap, Russia
>> Ivan Derbenev
>>
>>
>>
>>
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: Xenserver 6.5 live migration with local volumes
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Hi Anshul,
Thanks! Did you ever try live migrating from CloudStack with storage between clusters? I can't get that to work. I'll collect some details in the next days but was wondering if it was supposed to work.
Regards, Remi
Sent from my iPhone
> On 24 Jun 2016, at 06:30, Anshul Gangwar <an...@accelerite.com> wrote:
>
> Created the PR https://github.com/apache/cloudstack/pull/1596.
>
> Regards,
> Anshul
>
> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
>
> Is there anybody that wants to take this on?
>
> Thanks,
>
> - Dave
>
>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com> wrote:
>>
>> I updated the issue referenced in this email (https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include CloudStack 4.8 as an affected version. If we could at least get someone assigned to it, that’d be a good start.
>>
>> Thanks,
>>
>> - Dave
>>
>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <iv...@tech-corps.com>> wrote:
>>
>> Is there a way I can fix this on my installation, without recompiling jar files?
>>
>> Regards,
>> IT engineer
>> Farheap, Russia
>> Ivan Derbenev
>>
>> -----Original Message-----
>> From: Koushik Das [mailto:koushik.das@accelerite.com]
>> Sent: Tuesday, May 24, 2016 2:06 PM
>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
>> Subject: Re: Xenserver 6.5 live migration with local volumes
>>
>> This is a product bug and needs to be fixed. There is some serialization issue that needs fixing.
>>
>> In the below code volumeToFiler is null as Map is not a serializable type. It needs to be replaced with getVolumeToFilerAsList() and then the data needs to be read appropriately.
>>
>> final Map<VolumeTO, StorageFilerTO> volumeToFiler = command.getVolumeToFiler();
>>
>> -Koushik
>>
>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <iv...@tech-corps.com>> wrote:
>>
>> Hello!
>> After the migration onto XS 6.5 (and installing all the updates) and CS
>> 4.8 I can't migrate VMs with Local Storage
>>
>> The issue is similar to this:
>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't have any solutions Any ideas?
>>
>> Errors:
>>
>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process
>> of VM state report. host: 136592
>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
>> named cloud-main
>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
>> cloud-main on host=10.7.130.103;
>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
>> host 10.7.130.103 on device eth0
>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
>> 2016-05-24 12:10:17,170 WARN
>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception java.lang.NullPointerException. Storage motion failed due to java.lang.NullPointerException java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq 136589-479633360314957887: Response Received:
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
>> 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 110,
>> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
>> java.lang.NullPointerException\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>> andWrapper.java:86)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>> andWrapper.java:54)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
>> st(CitrixResourceBase.java:1677)\n\tat
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
>> tAttache.java:315)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
>> gedContextRunnable.java:49)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
>> (DefaultManagedContext.java:56)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>> thContext(DefaultManagedContext.java:103)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>> hContext(DefaultManagedContext.java:53)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>> dContextRunnable.java:46)\n\tat
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<http://ScheduledFutureTask.ru>
>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>> va:1145)\n\tat
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>> 136589-479633360314957887: No more commands found
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 110, { MigrateWithStorageAnswer } }
>> 2016-05-24 12:10:17,171 ERROR
>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
>> failed. Details: Exception: java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 2016-05-24 12:10:17,171 ERROR
>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) copy failed
>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>> java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
>> at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
>> at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
>> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Unsupported data object (VOLUME,
>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2),
>> no need to delete from object in store ref table
>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
>> while migrating the vm VM[User|i-3-271-VM] to host
>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
>> VM[User|i-3-271-VM]
>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from host
>> cloudstack-computing01-sar01.tech-corps.com<http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar and pod
>> zone1.sar
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
>> skipping send email
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
>> 128437433941265
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
>> 136589-479633360314957888: Executing request
>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) VM state transitted from :Migrating to Running with
>> event: OperationFailedvm's original host id: 136592 new host id: 136592
>> host id before state transition: 136589
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying
>> overprovisioning: 39900
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
>> applying overprovisioning: 11549697024
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) release cpu from host: 136589, old used:
>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
>> 39900; new used: 1995,reserved:0; movedfromreserved:
>> false,moveToReserveredfalse
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) release mem from host: 136589, old used:
>> 5368709120,reserved: 0, total: 11549697024; new used:
>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Response Received:
>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
>> 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
>> does not exist","wait":0}}] }
>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Unable to find listener.
>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for
>> VM 271, job origin: 4874
>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
>> instanceType: null, instanceId: null, cmd:
>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
>> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
>> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
>> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
>> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
>> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
>> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
>> origin:4874
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
>> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
>> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
>> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
>> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
>> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
>> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
>> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
>> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
>> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
>> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
>> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
>> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
>> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
>> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
>> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
>> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
>> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
>> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
>> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
>> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
>> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
>> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
>> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
>> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
>> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
>> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
>> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
>> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
>> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
>> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
>> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
>> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
>> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
>> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
>> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
>> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
>> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
>> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
>> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
>> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
>> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
>> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
>> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
>> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Publish async job-4875 complete on message bus
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Wake up jobs related to job-4875
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Update db status for job-4875
>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
>> job- 4875
>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Remove job-4875 from job monitoring
>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
>> exception while executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>> umeCmd
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
>> ],"errorcode":530,"errortext":"Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes. "}
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
>> async job-4874 complete on message bus
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>> jobs related to job-4874
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
>> status for job-4874
>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>> jobs joined with job-4874 and disjoin all subjobs created from job-
>> 4874
>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
>> executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>> umeCmd for job-4874
>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
>> job-4874 from job monitoring ^C
>>
>>
>> Regards,
>> IT engineer
>> Farheap, Russia
>> Ivan Derbenev
>>
>>
>>
>>
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
Re: Xenserver 6.5 live migration with local volumes
Posted by Remi Bergsma <RB...@schubergphilis.com>.
Hi Anshul,
Thanks! Did you ever try live migrating from CloudStack with storage between clusters? I can't get that to work. I'll collect some details in the next days but was wondering if it was supposed to work.
Regards, Remi
Sent from my iPhone
> On 24 Jun 2016, at 06:30, Anshul Gangwar <an...@accelerite.com> wrote:
>
> Created the PR https://github.com/apache/cloudstack/pull/1596.
>
> Regards,
> Anshul
>
> On 24/06/16, 2:33 AM, "Dave Garbus" <da...@spiceworks.com> wrote:
>
> Is there anybody that wants to take this on?
>
> Thanks,
>
> - Dave
>
>> On May 24, 2016, at 10:57 AM, Dave Garbus <da...@spiceworks.com> wrote:
>>
>> I updated the issue referenced in this email (https://issues.apache.org/jira/browse/CLOUDSTACK-9353) to include CloudStack 4.8 as an affected version. If we could at least get someone assigned to it, that’d be a good start.
>>
>> Thanks,
>>
>> - Dave
>>
>> On May 24, 2016, at 9:42 AM, Ivan Derbenev <iv...@tech-corps.com>> wrote:
>>
>> Is there a way I can fix this on my installation, without recompiling jar files?
>>
>> Regards,
>> IT engineer
>> Farheap, Russia
>> Ivan Derbenev
>>
>> -----Original Message-----
>> From: Koushik Das [mailto:koushik.das@accelerite.com]
>> Sent: Tuesday, May 24, 2016 2:06 PM
>> To: users@cloudstack.apache.org<ma...@cloudstack.apache.org>
>> Subject: Re: Xenserver 6.5 live migration with local volumes
>>
>> This is a product bug and needs to be fixed. There is some serialization issue that needs fixing.
>>
>> In the below code volumeToFiler is null as Map is not a serializable type. It needs to be replaced with getVolumeToFilerAsList() and then the data needs to be read appropriately.
>>
>> final Map<VolumeTO, StorageFilerTO> volumeToFiler = command.getVolumeToFiler();
>>
>> -Koushik
>>
>> On 24/05/16, 3:23 PM, "Ivan Derbenev" <iv...@tech-corps.com>> wrote:
>>
>> Hello!
>> After the migration onto XS 6.5 (and installing all the updates) and CS
>> 4.8 I can't migrate VMs with Local Storage
>>
>> The issue is similar to this:
>> https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't have any solutions Any ideas?
>>
>> Errors:
>>
>> 2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>> (DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process
>> of VM state report. host: 136592
>> 2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network
>> named cloud-main
>> 2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
>> cloud-main on host=10.7.130.103;
>> Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
>> pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
>> 2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on
>> host 10.7.130.103 on device eth0
>> 2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for
>> 1007. The uuid is ec053fee-0426-9665-150a-77399562196b
>> 2016-05-24 12:10:17,170 WARN
>> [c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception java.lang.NullPointerException. Storage motion failed due to java.lang.NullPointerException java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq 136589-479633360314957887: Response Received:
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>> 136589-479633360314957887: Processing: { Ans: , MgmtId:
>> 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 110,
>> [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"detai
>> ls":"Exception: java.lang.NullPointerException\nMessage: null\nStack:
>> java.lang.NullPointerException\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>> andWrapper.java:86)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610Migr
>> ateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageComm
>> andWrapper.java:54)\n\tat
>> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWr
>> apper.execute(CitrixRequestWrapper.java:122)\n\tat
>> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeReque
>> st(CitrixResourceBase.java:1677)\n\tat
>> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgen
>> tAttache.java:315)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Mana
>> gedContextRunnable.java:49)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call
>> (DefaultManagedContext.java:56)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWi
>> thContext(DefaultManagedContext.java:103)\n\tat
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWit
>> hContext(DefaultManagedContext.java:53)\n\tat
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(Manage
>> dContextRunnable.java:46)\n\tat
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> \n\tat java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ac
>> cess$201(ScheduledThreadPoolExecutor.java:178)\n\tat
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.ru<http://ScheduledFutureTask.ru>
>> n(ScheduledThreadPoolExecutor.java:292)\n\tat
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja
>> va:1145)\n\tat
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>> ava:615)\n\tat java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
>> (DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq
>> 136589-479633360314957887: No more commands found
>> 2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 110, { MigrateWithStorageAnswer } }
>> 2016-05-24 12:10:17,171 ERROR
>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM]
>> failed. Details: Exception: java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 2016-05-24 12:10:17,171 ERROR
>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) copy failed
>> com.cloud.utils.exception.CloudRuntimeException: Error while migrating
>> the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>> java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
>> at org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
>> at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
>> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Unsupported data object (VOLUME,
>> org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2),
>> no need to delete from object in store ref table
>> 2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with
>> its volumes. com.cloud.utils.exception.CloudRuntimeException: Error
>> while migrating the vm VM[User|i-3-271-VM] to host
>> Host[-136589-Routing]. Exception: java.lang.NullPointerException
>> Message: null
>> Stack: java.lang.NullPointerException
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
>> at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
>> at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
>> at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
>> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> 2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Migration was unsuccessful. Cleaning up:
>> VM[User|i-3-271-VM]
>> 2016-05-24 12:10:17,184 WARN [o.a.c.alerts]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
>> clusterId:: null // message:: Unable to migrate vm i-3-271-VM from host
>> cloudstack-computing01-sar01.tech-corps.com<http://cloudstack-computing01-sar01.tech-corps.com> in zone zone1.sar and pod
>> zone1.sar
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Have already sent: 1 emails for alert type '17' --
>> skipping send email
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Routed from
>> 128437433941265
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd ,
>> MgmtId: 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 100011,
>> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence
>> ":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}] }
>> 2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
>> (DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq
>> 136589-479633360314957888: Executing request
>> 2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) VM state transitted from :Migrating to Running with
>> event: OperationFailedvm's original host id: 136592 new host id: 136592
>> host id before state transition: 136589
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying
>> overprovisioning: 39900
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after
>> applying overprovisioning: 11549697024
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) release cpu from host: 136589, old used:
>> 3990,reserved: 0, actual total: 15960, total with overprovisioning:
>> 39900; new used: 1995,reserved:0; movedfromreserved:
>> false,moveToReserveredfalse
>> 2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>> (logid:8748bffe) release mem from host: 136589, old used:
>> 5368709120,reserved: 0, total: 11549697024; new used:
>> 4294967296,reserved:0; movedfromreserved: false,moveToReserveredfalse
>> 2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Response Received:
>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request]
>> (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq
>> 136589-479633360314957888: Processing: { Ans: , MgmtId:
>> 128437433941265, via:
>> 136589(cloudstack-computing03-sar01.tech-corps.com<http://cloudstack-computing03-sar01.tech-corps.com>), Ver: v1, Flags:
>> 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM
>> does not exist","wait":0}}] }
>> 2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888: Unable to find listener.
>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> 2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> 2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Done with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for
>> VM 271, job origin: 4874
>> 2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Unable to complete AsyncJobVO {id:4875, userId: 17, accountId: 3,
>> instanceType: null, instanceId: null, cmd:
>> com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
>> rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXM
>> CAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdX
>> RpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZ
>> EoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hw
>> AAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAA
>> AAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYW
>> RGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg, cmdVersion: 0,
>> status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null,
>> initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
>> lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job
>> origin:4874
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Complete async job-4875, jobStatus: FAILED, resultCode: 0, result:
>> rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGl
>> vbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcH
>> Rpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATa
>> mF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Ro
>> cm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2t
>> UcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeG
>> NlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0Z
>> WQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAe
>> W0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF
>> 2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZG
>> VjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4c
>> AAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9s
>> dW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9
>> sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW
>> 1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlnc
>> mF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRz
>> dW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2N
>> lc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZX
>> NxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsd
>> AAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAY
>> amF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt
>> 0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZG
>> xlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAf
>> gAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1X
>> b3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGU
>> uY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNX
>> QAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0A
>> D5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAA
>> AADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYX
>> VsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhb
>> GxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5p
>> bXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgA
>> MAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2
>> hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxA
>> H4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0
>> aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmp
>> hdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdA
>> APRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlb
>> nQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5X
>> b3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN
>> 1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1
>> RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZ
>> Uxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMk
>> VW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29
>> sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleH
>> AAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Publish async job-4875 complete on message bus
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Wake up jobs related to job-4875
>> 2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Update db status for job-4875
>> 2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Wake up jobs joined with job-4875 and disjoin all subjobs created from
>> job- 4875
>> 2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Done executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
>> 2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe)
>> Remove job-4875 from job monitoring
>> 2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
>> exception while executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>> umeCmd
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm VM[User|i-3-271-VM] along with its volumes.
>> at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
>> at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
>> at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
>> at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete
>> async job-4874, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[
>> ],"errorcode":530,"errortext":"Failed to migrated vm
>> VM[User|i-3-271-VM] along with its volumes. "}
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish
>> async job-4874 complete on message bus
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>> jobs related to job-4874
>> 2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db
>> status for job-4874
>> 2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up
>> jobs joined with job-4874 and disjoin all subjobs created from job-
>> 4874
>> 2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done
>> executing
>> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVol
>> umeCmd for job-4874
>> 2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove
>> job-4874 from job monitoring ^C
>>
>>
>> Regards,
>> IT engineer
>> Farheap, Russia
>> Ivan Derbenev
>>
>>
>>
>>
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.
>
>
>
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the property of Accelerite, a Persistent Systems business. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Accelerite, a Persistent Systems business does not accept any liability for virus infected mails.