You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Sailaja Mada (JIRA)" <ji...@apache.org> on 2013/07/29 10:05:48 UTC
[jira] [Updated] (CLOUDSTACK-3891) [VMWAREDVS] Failed to Live
Migrate instances across DVSwitch clusters
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sailaja Mada updated CLOUDSTACK-3891:
-------------------------------------
Description:
Setup: VMWARE 2 clusters (5.0 Update2) , DVS enabled at the zone level , Each cluster is using different dvSwitch
Steps:
1. Configure Adv Zone with dvs enabled at the zone level
2. Configure Cluster1 with dvswitch1 (Traffic lables configured at the zone level with 2 physical networks ( Network1 - Mgmt with standard switch lable , Network2 - Guest and public with dvswitch lable )
3. Add second cluster with dvswitch2
4. Deploy instances on cluster2
5. Tried to live migrate instance from cluster2 to cluster1.
Observation:
Failed to Live Migrate instances across DVSwitch clusters
2013-07-29 13:08:53,026 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-54:null) Seq 5-669909134: Executing request
2013-07-29 13:08:53,036 INFO [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Executing resource MigrateWithStorageCommand: {"vm":{"id":8,"name":"i-3-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS 5.3 (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8e56770c1bb49556","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"a604653b-3838-47ec-b17c-a57abf3bf4f8","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"11692616-9357-40e9-9163-cda1723a5290","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3ecae8a1-c59a-33ee-b22b-6bd5758f10e8","id":1,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/dvsps1","port":2049}},"name":"ROOT-8","size":0,"path":"ROOT-8-13","volumeId":13,"vmName":"i-3-8-VM","accountId":3,"format":"OVA","id":13,"hypervisorType":"None"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"af03f814-e66f-4367-9506-369aa00120ad","ip":"10.1.1.20","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:71:f8:00:03","dns1":"10.103.128.15","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://791","isolationUri":"vlan://791","isSecurityGroupEnabled":false,"name":"dvSwitch18,,vmwaredvs"}]},"volumeToFiler":{"Vol[13|ROOT|ROOT-8-13|0]":{"id":1,"uuid":"3ecae8a1-c59a-33ee-b22b-6bd5758f10e8","host":"10.102.192.100","path":"/cpg_vol/sailaja/dvsps1","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7279@10.102.192.248","wait":0}
2013-07-29 13:08:53,073 DEBUG [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Preparing spec for volume : ROOT-8
2013-07-29 13:08:53,111 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Look for disk device info from volume : [3ecae8a1c59a33eeb22b6bd5758f10e8] ROOT-8-13.vmdk
2013-07-29 13:08:53,112 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Test against disk device, controller key: 200, unit number: 1
2013-07-29 13:08:53,113 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Test against disk backing : [3ecae8a1c59a33eeb22b6bd5758f10e8] ROOT-8-13.vmdk
2013-07-29 13:08:53,114 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Disk backing : [3ecae8a1c59a33eeb22b6bd5758f10e8] ROOT-8-13.vmdk matches ==> ide0:1
2013-07-29 13:08:53,114 INFO [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Prepare network on vmwaredvs P[dvSwitch23:untagged] with name prefix: cloud.guest
2013-07-29 13:08:53,131 INFO [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Found distributed vSwitch com.vmware.vim25.ManagedObjectReference@9391607
2013-07-29 13:08:53,140 INFO [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Found Distributed Virtual Port group cloud.guest.791.200.1-dvSwitch23
2013-07-29 13:08:53,161 INFO [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Updating Distributed Virtual Port group cloud.guest.791.200.1-dvSwitch23
2013-07-29 13:08:53,240 DEBUG [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Added custom field : cloud.gc.dvp
2013-07-29 13:08:53,539 DEBUG [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Successfully migrated storage of VM i-3-8-VM to target datastore(s)
2013-07-29 13:08:53,849 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 1 routers to update status.
2013-07-29 13:08:53,851 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-07-29 13:08:53,998 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 1 routers to update status.
2013-07-29 13:08:54,000 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-07-29 13:08:54,973 WARN [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) MigrationCommand failed due to Exception: java.lang.RuntimeException
Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-29 13:08:54,976 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-54:null) Seq 5-669909134: Response Received:
2013-07-29 13:08:54,977 DEBUG [agent.transport.Request] (DirectAgent-54:null) Seq 5-669909134: Processing: { Ans: , MgmtId: 187767034175903, via: 5, Ver: v1, Flags: 110, [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"details":"Exception: java.lang.RuntimeException\nMessage: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.\nStack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.\n\tat com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)\n\tat com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)\n\tat com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)\n\tat com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:166)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
2013-07-29 13:08:54,977 DEBUG [agent.transport.Request] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 5-669909134: Received: { Ans: , MgmtId: 187767034175903, via: 5, Ver: v1, Flags: 110, { MigrateWithStorageAnswer } }
2013-07-29 13:08:54,978 ERROR [storage.motion.VmwareStorageMotionStrategy] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Migration with storage of vm VM[User|instance2] failed. Details: Exception: java.lang.RuntimeException
Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
Stack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-29 13:08:54,980 ERROR [storage.motion.VmwareStorageMotionStrategy] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) copy failed
com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm VM[User|instance2] to host Host[-1-Routing]. Exception: java.lang.RuntimeException
Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
Stack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
at org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.migrateVmWithVolumesAcrossCluster(VmwareStorageMotionStrategy.java:140)
at org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.copyAsync(VmwareStorageMotionStrategy.java:99)
at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:67)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:940)
at com.cloud.storage.VolumeManagerImpl.migrateVolumes(VolumeManagerImpl.java:2274)
at com.cloud.vm.VirtualMachineManagerImpl.migrateWithStorage(VirtualMachineManagerImpl.java:1782)
at com.cloud.vm.UserVmManagerImpl.migrateVirtualMachineWithVolume(UserVmManagerImpl.java:4250)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd.execute(MigrateVirtualMachineWithVolumeCmd.java:137)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-29 13:08:54,985 DEBUG [agent.manager.AgentAttache] (DirectAgent-54:null) Seq 5-669909134: No more commands found
2013-07-29 13:08:54,994 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@447a99c)
2013-07-29 13:08:54,994 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Failed to migrated vm VM[User|instance2] along with its volumes. com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm VM[User|instance2] to host Host[-1-Routing]. Exception: java.lang.RuntimeException
Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
Stack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
is not accessible.
at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-29 13:08:54,995 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Migration was unsuccessful. Cleaning up: VM[User|instance2]
2013-07-29 13:08:54,996 WARN [apache.cloudstack.alerts] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) alertType:: 17 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to migrate vm i-3-8-VM from host 10.102.192.23 in zone DVSzone1 and pod DVSzone1
2013-07-29 13:08:55,014 DEBUG [agent.manager.AgentAttache] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 1-317260064: Routed from 187767034175903
2013-07-29 13:08:55,014 DEBUG [agent.transport.Request] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 1-317260064: Sending { Cmd , MgmtId: 187767034175903, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-3-8-VM","wait":0}}] }
2013-07-29 13:08:55,015 DEBUG [agent.transport.Request] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 1-317260064: Executing: { Cmd , MgmtId: 187767034175903, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-3-8-VM","wait":0}}] }
2013-07-29 13:08:55,015 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-335:null) Seq 1-317260064: Executing request
2013-07-29 13:08:55,015 INFO [vmware.resource.VmwareResource] (DirectAgent-335:10.102.192.18) Executing resource StopCommand: {"isProxy":false,"executeInSequence":false,"vmName":"i-3-8-VM","wait":0}
was:
Setup: VMWARE 2 clusters (5.0 Update2) , DVS enabled at the zone level , Each cluster is using different dvSwitch
Steps:
1. Configure Adv Zone with dvs enabled at the zone level
2. Configure Cluster1 with dvswitch1 (Traffic lables configured at the zone level with 2 physical networks ( Network1 - Mgmt with standard switch lable , Network2 - Guest and public with dvswitch lable )
3. Add second cluster with dvswitch2
4. Deploy instances on cluster2
5. Tried to live migrate instance from cluster2 to cluster1.
Observation:
Failed to Live Migrate instances acroos DVSwitch clusters
2013-07-29 13:08:53,026 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-54:null) Seq 5-669909134: Executing request
2013-07-29 13:08:53,036 INFO [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Executing resource MigrateWithStorageCommand: {"vm":{"id":8,"name":"i-3-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS 5.3 (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8e56770c1bb49556","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"a604653b-3838-47ec-b17c-a57abf3bf4f8","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"11692616-9357-40e9-9163-cda1723a5290","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3ecae8a1-c59a-33ee-b22b-6bd5758f10e8","id":1,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/dvsps1","port":2049}},"name":"ROOT-8","size":0,"path":"ROOT-8-13","volumeId":13,"vmName":"i-3-8-VM","accountId":3,"format":"OVA","id":13,"hypervisorType":"None"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"af03f814-e66f-4367-9506-369aa00120ad","ip":"10.1.1.20","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:71:f8:00:03","dns1":"10.103.128.15","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://791","isolationUri":"vlan://791","isSecurityGroupEnabled":false,"name":"dvSwitch18,,vmwaredvs"}]},"volumeToFiler":{"Vol[13|ROOT|ROOT-8-13|0]":{"id":1,"uuid":"3ecae8a1-c59a-33ee-b22b-6bd5758f10e8","host":"10.102.192.100","path":"/cpg_vol/sailaja/dvsps1","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7279@10.102.192.248","wait":0}
2013-07-29 13:08:53,073 DEBUG [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Preparing spec for volume : ROOT-8
2013-07-29 13:08:53,111 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Look for disk device info from volume : [3ecae8a1c59a33eeb22b6bd5758f10e8] ROOT-8-13.vmdk
2013-07-29 13:08:53,112 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Test against disk device, controller key: 200, unit number: 1
2013-07-29 13:08:53,113 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Test against disk backing : [3ecae8a1c59a33eeb22b6bd5758f10e8] ROOT-8-13.vmdk
2013-07-29 13:08:53,114 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Disk backing : [3ecae8a1c59a33eeb22b6bd5758f10e8] ROOT-8-13.vmdk matches ==> ide0:1
2013-07-29 13:08:53,114 INFO [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Prepare network on vmwaredvs P[dvSwitch23:untagged] with name prefix: cloud.guest
2013-07-29 13:08:53,131 INFO [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Found distributed vSwitch com.vmware.vim25.ManagedObjectReference@9391607
2013-07-29 13:08:53,140 INFO [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Found Distributed Virtual Port group cloud.guest.791.200.1-dvSwitch23
2013-07-29 13:08:53,161 INFO [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Updating Distributed Virtual Port group cloud.guest.791.200.1-dvSwitch23
2013-07-29 13:08:53,240 DEBUG [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Added custom field : cloud.gc.dvp
2013-07-29 13:08:53,539 DEBUG [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Successfully migrated storage of VM i-3-8-VM to target datastore(s)
2013-07-29 13:08:53,849 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 1 routers to update status.
2013-07-29 13:08:53,851 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-07-29 13:08:53,998 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 1 routers to update status.
2013-07-29 13:08:54,000 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2013-07-29 13:08:54,973 WARN [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) MigrationCommand failed due to Exception: java.lang.RuntimeException
Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-29 13:08:54,976 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-54:null) Seq 5-669909134: Response Received:
2013-07-29 13:08:54,977 DEBUG [agent.transport.Request] (DirectAgent-54:null) Seq 5-669909134: Processing: { Ans: , MgmtId: 187767034175903, via: 5, Ver: v1, Flags: 110, [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"details":"Exception: java.lang.RuntimeException\nMessage: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.\nStack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.\n\tat com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)\n\tat com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)\n\tat com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)\n\tat com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:166)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
2013-07-29 13:08:54,977 DEBUG [agent.transport.Request] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 5-669909134: Received: { Ans: , MgmtId: 187767034175903, via: 5, Ver: v1, Flags: 110, { MigrateWithStorageAnswer } }
2013-07-29 13:08:54,978 ERROR [storage.motion.VmwareStorageMotionStrategy] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Migration with storage of vm VM[User|instance2] failed. Details: Exception: java.lang.RuntimeException
Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
Stack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-29 13:08:54,980 ERROR [storage.motion.VmwareStorageMotionStrategy] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) copy failed
com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm VM[User|instance2] to host Host[-1-Routing]. Exception: java.lang.RuntimeException
Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
Stack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
at org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.migrateVmWithVolumesAcrossCluster(VmwareStorageMotionStrategy.java:140)
at org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.copyAsync(VmwareStorageMotionStrategy.java:99)
at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:67)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:940)
at com.cloud.storage.VolumeManagerImpl.migrateVolumes(VolumeManagerImpl.java:2274)
at com.cloud.vm.VirtualMachineManagerImpl.migrateWithStorage(VirtualMachineManagerImpl.java:1782)
at com.cloud.vm.UserVmManagerImpl.migrateVirtualMachineWithVolume(UserVmManagerImpl.java:4250)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd.execute(MigrateVirtualMachineWithVolumeCmd.java:137)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-29 13:08:54,985 DEBUG [agent.manager.AgentAttache] (DirectAgent-54:null) Seq 5-669909134: No more commands found
2013-07-29 13:08:54,994 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@447a99c)
2013-07-29 13:08:54,994 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Failed to migrated vm VM[User|instance2] along with its volumes. com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm VM[User|instance2] to host Host[-1-Routing]. Exception: java.lang.RuntimeException
Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
Stack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
is not accessible.
at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-29 13:08:54,995 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Migration was unsuccessful. Cleaning up: VM[User|instance2]
2013-07-29 13:08:54,996 WARN [apache.cloudstack.alerts] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) alertType:: 17 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to migrate vm i-3-8-VM from host 10.102.192.23 in zone DVSzone1 and pod DVSzone1
2013-07-29 13:08:55,014 DEBUG [agent.manager.AgentAttache] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 1-317260064: Routed from 187767034175903
2013-07-29 13:08:55,014 DEBUG [agent.transport.Request] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 1-317260064: Sending { Cmd , MgmtId: 187767034175903, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-3-8-VM","wait":0}}] }
2013-07-29 13:08:55,015 DEBUG [agent.transport.Request] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 1-317260064: Executing: { Cmd , MgmtId: 187767034175903, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-3-8-VM","wait":0}}] }
2013-07-29 13:08:55,015 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-335:null) Seq 1-317260064: Executing request
2013-07-29 13:08:55,015 INFO [vmware.resource.VmwareResource] (DirectAgent-335:10.102.192.18) Executing resource StopCommand: {"isProxy":false,"executeInSequence":false,"vmName":"i-3-8-VM","wait":0}
> [VMWAREDVS] Failed to Live Migrate instances across DVSwitch clusters
> ----------------------------------------------------------------------
>
> Key: CLOUDSTACK-3891
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3891
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Network Controller, VMware
> Affects Versions: 4.2.0
> Reporter: Sailaja Mada
> Priority: Critical
> Fix For: 4.2.0
>
>
> Setup: VMWARE 2 clusters (5.0 Update2) , DVS enabled at the zone level , Each cluster is using different dvSwitch
> Steps:
> 1. Configure Adv Zone with dvs enabled at the zone level
> 2. Configure Cluster1 with dvswitch1 (Traffic lables configured at the zone level with 2 physical networks ( Network1 - Mgmt with standard switch lable , Network2 - Guest and public with dvswitch lable )
> 3. Add second cluster with dvswitch2
> 4. Deploy instances on cluster2
> 5. Tried to live migrate instance from cluster2 to cluster1.
> Observation:
> Failed to Live Migrate instances across DVSwitch clusters
> 2013-07-29 13:08:53,026 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-54:null) Seq 5-669909134: Executing request
> 2013-07-29 13:08:53,036 INFO [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Executing resource MigrateWithStorageCommand: {"vm":{"id":8,"name":"i-3-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS 5.3 (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"8e56770c1bb49556","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"a604653b-3838-47ec-b17c-a57abf3bf4f8","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"11692616-9357-40e9-9163-cda1723a5290","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"3ecae8a1-c59a-33ee-b22b-6bd5758f10e8","id":1,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/dvsps1","port":2049}},"name":"ROOT-8","size":0,"path":"ROOT-8-13","volumeId":13,"vmName":"i-3-8-VM","accountId":3,"format":"OVA","id":13,"hypervisorType":"None"}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"af03f814-e66f-4367-9506-369aa00120ad","ip":"10.1.1.20","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:71:f8:00:03","dns1":"10.103.128.15","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://791","isolationUri":"vlan://791","isSecurityGroupEnabled":false,"name":"dvSwitch18,,vmwaredvs"}]},"volumeToFiler":{"Vol[13|ROOT|ROOT-8-13|0]":{"id":1,"uuid":"3ecae8a1-c59a-33ee-b22b-6bd5758f10e8","host":"10.102.192.100","path":"/cpg_vol/sailaja/dvsps1","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7279@10.102.192.248","wait":0}
> 2013-07-29 13:08:53,073 DEBUG [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Preparing spec for volume : ROOT-8
> 2013-07-29 13:08:53,111 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Look for disk device info from volume : [3ecae8a1c59a33eeb22b6bd5758f10e8] ROOT-8-13.vmdk
> 2013-07-29 13:08:53,112 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Test against disk device, controller key: 200, unit number: 1
> 2013-07-29 13:08:53,113 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Test against disk backing : [3ecae8a1c59a33eeb22b6bd5758f10e8] ROOT-8-13.vmdk
> 2013-07-29 13:08:53,114 INFO [vmware.mo.VirtualMachineMO] (DirectAgent-54:10.102.192.23) Disk backing : [3ecae8a1c59a33eeb22b6bd5758f10e8] ROOT-8-13.vmdk matches ==> ide0:1
> 2013-07-29 13:08:53,114 INFO [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Prepare network on vmwaredvs P[dvSwitch23:untagged] with name prefix: cloud.guest
> 2013-07-29 13:08:53,131 INFO [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Found distributed vSwitch com.vmware.vim25.ManagedObjectReference@9391607
> 2013-07-29 13:08:53,140 INFO [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Found Distributed Virtual Port group cloud.guest.791.200.1-dvSwitch23
> 2013-07-29 13:08:53,161 INFO [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Updating Distributed Virtual Port group cloud.guest.791.200.1-dvSwitch23
> 2013-07-29 13:08:53,240 DEBUG [vmware.mo.HypervisorHostHelper] (DirectAgent-54:10.102.192.23) Added custom field : cloud.gc.dvp
> 2013-07-29 13:08:53,539 DEBUG [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) Successfully migrated storage of VM i-3-8-VM to target datastore(s)
> 2013-07-29 13:08:53,849 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 1 routers to update status.
> 2013-07-29 13:08:53,851 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> 2013-07-29 13:08:53,998 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 1 routers to update status.
> 2013-07-29 13:08:54,000 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> 2013-07-29 13:08:54,973 WARN [vmware.resource.VmwareResource] (DirectAgent-54:10.102.192.23) MigrationCommand failed due to Exception: java.lang.RuntimeException
> Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
> java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
> at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
> at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
> at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
> at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
> at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-07-29 13:08:54,976 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-54:null) Seq 5-669909134: Response Received:
> 2013-07-29 13:08:54,977 DEBUG [agent.transport.Request] (DirectAgent-54:null) Seq 5-669909134: Processing: { Ans: , MgmtId: 187767034175903, via: 5, Ver: v1, Flags: 110, [{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"details":"Exception: java.lang.RuntimeException\nMessage: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.\nStack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.\n\tat com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)\n\tat com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)\n\tat com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)\n\tat com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)\n\tat com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:166)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
> 2013-07-29 13:08:54,977 DEBUG [agent.transport.Request] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 5-669909134: Received: { Ans: , MgmtId: 187767034175903, via: 5, Ver: v1, Flags: 110, { MigrateWithStorageAnswer } }
> 2013-07-29 13:08:54,978 ERROR [storage.motion.VmwareStorageMotionStrategy] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Migration with storage of vm VM[User|instance2] failed. Details: Exception: java.lang.RuntimeException
> Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
> Stack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
> at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
> at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
> at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
> at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
> at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-07-29 13:08:54,980 ERROR [storage.motion.VmwareStorageMotionStrategy] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) copy failed
> com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm VM[User|instance2] to host Host[-1-Routing]. Exception: java.lang.RuntimeException
> Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
> Stack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
> at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
> at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
> at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
> at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
> at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> at org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.migrateVmWithVolumesAcrossCluster(VmwareStorageMotionStrategy.java:140)
> at org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.copyAsync(VmwareStorageMotionStrategy.java:99)
> at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:67)
> at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:940)
> at com.cloud.storage.VolumeManagerImpl.migrateVolumes(VolumeManagerImpl.java:2274)
> at com.cloud.vm.VirtualMachineManagerImpl.migrateWithStorage(VirtualMachineManagerImpl.java:1782)
> at com.cloud.vm.UserVmManagerImpl.migrateVirtualMachineWithVolume(UserVmManagerImpl.java:4250)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd.execute(MigrateVirtualMachineWithVolumeCmd.java:137)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-07-29 13:08:54,985 DEBUG [agent.manager.AgentAttache] (DirectAgent-54:null) Seq 5-669909134: No more commands found
> 2013-07-29 13:08:54,994 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@447a99c)
> 2013-07-29 13:08:54,994 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Failed to migrated vm VM[User|instance2] along with its volumes. com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm VM[User|instance2] to host Host[-1-Routing]. Exception: java.lang.RuntimeException
> Message: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
> Stack: java.lang.RuntimeException: Currently connected network interface 'Network adapter 1' uses network 'Distributed Virtual Switch (uuid: a3 63 2f 50 f2 40 33 e6-de dd 8e 9d 0b 6a 38 53)', which is not accessible.
> is not accessible.
> at com.cloud.hypervisor.vmware.util.VmwareClient.waitForTask(VmwareClient.java:290)
> at com.cloud.hypervisor.vmware.mo.VirtualMachineMO.migrate(VirtualMachineMO.java:331)
> at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3953)
> at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:464)
> at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-07-29 13:08:54,995 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Migration was unsuccessful. Cleaning up: VM[User|instance2]
> 2013-07-29 13:08:54,996 WARN [apache.cloudstack.alerts] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) alertType:: 17 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to migrate vm i-3-8-VM from host 10.102.192.23 in zone DVSzone1 and pod DVSzone1
> 2013-07-29 13:08:55,014 DEBUG [agent.manager.AgentAttache] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 1-317260064: Routed from 187767034175903
> 2013-07-29 13:08:55,014 DEBUG [agent.transport.Request] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 1-317260064: Sending { Cmd , MgmtId: 187767034175903, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-3-8-VM","wait":0}}] }
> 2013-07-29 13:08:55,015 DEBUG [agent.transport.Request] (Job-Executor-33:job-30 = [ 6d3be833-e02b-4660-a842-50fb44b55b04 ]) Seq 1-317260064: Executing: { Cmd , MgmtId: 187767034175903, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-3-8-VM","wait":0}}] }
> 2013-07-29 13:08:55,015 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-335:null) Seq 1-317260064: Executing request
> 2013-07-29 13:08:55,015 INFO [vmware.resource.VmwareResource] (DirectAgent-335:10.102.192.18) Executing resource StopCommand: {"isProxy":false,"executeInSequence":false,"vmName":"i-3-8-VM","wait":0}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira