You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "angeline shen (JIRA)" <ji...@apache.org> on 2014/02/21 02:56:19 UTC
[jira] [Resolved] (CLOUDSTACK-5665) XEN patch/hotfix certification
- XS 6.0.2 XS602E030 patch installation fail
[ https://issues.apache.org/jira/browse/CLOUDSTACK-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
angeline shen resolved CLOUDSTACK-5665.
---------------------------------------
Resolution: Fixed
Add procedure to Reconnect master host and slave host.
> XEN patch/hotfix certification - XS 6.0.2 XS602E030 patch installation fail
> ---------------------------------------------------------------------------
>
> Key: CLOUDSTACK-5665
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5665
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Management Server
> Affects Versions: 4.2.1
> Reporter: angeline shen
> Priority: Critical
> Fix For: 4.2.1
>
> Attachments: management-server.log.gz
>
>
>
> XEN hotfix installation procedure
> 1. Hosts master slave installed with
> XS 6.0.2 base + Hotfix XS602E001 to XS602E029 + CSPs
> 2. Basic zone with cluster of above master & slave hosts
> security group default with NO security rules
> Deploy two VMs
> 3. MS: master host put to maintenance mode - VMs migrate to slave host
> 4. MS: unmanage cluster
> 5. master host: install hotfix XS602E030 + CSP , reboot master
> 6. MS: master host cancel maintenance mode
> 7. MS: manage cluster
> 8. MS: slave host put to maintenance mode, VMs FAIL migrate to master
> Unable to complete hotfix installation
> 2013-12-26 16:51:25,307 INFO [xen.resource.CitrixResourceBase] (DirectAgent-56:null) VM does not exist on XenServerc6c75483-b35b-4a7c-9e2d-2274e34dd437
> 2013-12-26 16:51:25,307 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-56:null) Seq 1-265617550: Response Received:
> 2013-12-26 16:51:25,307 DEBUG [agent.transport.Request] (DirectAgent-56:null) Seq 1-265617550: Processing: { Ans: , MgmtId: 7692017993539, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not exist","wait":0}}] }
> 2013-12-26 16:51:25,307 DEBUG [agent.manager.AgentAttache] (DirectAgent-56:null) Seq 1-265617550: Unable to find listener.
> 2013-12-26 16:51:25,307 DEBUG [agent.manager.AgentAttache] (DirectAgent-56:null) Seq 1-265617550: No more commands found
> 2013-12-26 16:51:25,313 DEBUG [vm.dao.VMInstanceDaoImpl] (HA-Worker-3:work-19) Unable to update VM[DomainRouter|r-4-VM]: DB Data={Host=4; State=Running; updated=13; time=Thu Dec 26 16:51:25 PST 2013} New Data: {Host=4; State=Stopping; updated=12; time=Thu Dec 26 16:51:25 PST 2013} Stale Data: {Host=4; State=Running; updated=11; time=Thu Dec 26 16:51:22 PST 2013}
> 2013-12-26 16:51:25,313 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-3:work-19) Unable to stop VM due to VM is being operated on.
> 2013-12-26 16:51:25,313 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-19) Unable to migrate vm from 4
> 2013-12-26 16:51:25,314 DEBUG [cloud.resource.ResourceManagerImpl] (HA-Worker-3:work-19) No next resource state for host 4 while current state is ErrorInMaintenance with event UnableToMigrate
> com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state =ErrorInMaintenance event =UnableToMigrate
> at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178)
> at com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313)
> at com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:602)
> at com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858)
> 2013-12-26 16:51:25,314 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-19) Completed HAWork[19-Migration-4-Running-Migrating]
> 2013-12-26 16:51:25,324 WARN [xen.resource.CitrixResourceBase] (DirectAgent-57:null) Task failed! Task record: uuid: 38ce6cd4-9ab2-7e46-4d10-dafe25a83084
> nameLabel: Async.VM.pool_migrate
> nameDescription:
> allowedOperations: []
> currentOperations: {}
> created: Thu Dec 26 16:55:03 PST 2013
> finished: Thu Dec 26 16:55:03 PST 2013
> status: failure
> residentOn: com.xensource.xenapi.Host@40561b9c
> progress: 1.0
> type: <none/>
> result:
> errorInfo: [VM_REQUIRES_SR, OpaqueRef:6e9c09e8-9ed5-1e21-2048-d848a084a15a, OpaqueRef:eec84d31-9691-1ba4-f7bb-39db42ac8628]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
> 2013-12-26 16:51:25,332 WARN [xen.resource.CitrixResourceBase] (DirectAgent-57:null) Unable to migrate VM(s-1-VM) from host(d044a044-1a30-4c01-ae83-11d2631a4ffe) due to Task failed! Task record: uuid: 38ce6cd4-9ab2-7e46-4d10-dafe25a83084
> nameLabel: Async.VM.pool_migrate
> nameDescription:
> allowedOperations: []
> currentOperations: {}
> created: Thu Dec 26 16:55:03 PST 2013
> finished: Thu Dec 26 16:55:03 PST 2013
> status: failure
> residentOn: com.xensource.xenapi.Host@40561b9c
> progress: 1.0
> type: <none/>
> result:
> errorInfo: [VM_REQUIRES_SR, OpaqueRef:6e9c09e8-9ed5-1e21-2048-d848a084a15a, OpaqueRef:eec84d31-9691-1ba4-f7bb-39db42ac8628]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
> Task failed! Task record: uuid: 38ce6cd4-9ab2-7e46-4d10-dafe25a83084
> nameLabel: Async.VM.pool_migrate
> nameDescription:
> :
> residentOn: com.xensource.xenapi.Host@40561b9c
> progress: 1.0
> type: <none/>
> result:
> errorInfo: [VM_REQUIRES_SR, OpaqueRef:6e9c09e8-9ed5-1e21-2048-d848a084a15a, OpaqueRef:eec84d31-9691-1ba4-f7bb-39db42ac8628]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.migrateVM(CitrixResourceBase.java:3789)
> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:3346)
> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:511)
> at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
> at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-12-26 16:51:25,342 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-57:null) 6. The VM s-1-VM is in Migrating state
> 2013-12-26 16:51:25,342 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-57:null) Seq 4-961806364: Response Received:
> 2013-12-26 16:51:25,342 DEBUG [agent.transport.Request] (DirectAgent-57:null) Seq 4-961806364: Processing: { Ans: , MgmtId: 7692017993539, via: 4, Ver: v1, Flags: 110, [{"com.cloud.agent.api.MigrateAnswer":{"result":false,"details":"Catch Exception com.cloud.utils.exception.CloudRuntimeException: Migration failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to migrate VM(s-1-VM) from host(d044a044-1a30-4c01-ae83-11d2631a4ffe) due to Task failed! Task record: uuid: 38ce6cd4-9ab2-7e46-4d10-dafe25a83084\n nameLabel: Async.VM.pool_migrate\n nameDescription: \n allowedOperations: []\n currentOperations: {}\n created: Thu Dec 26 16:55:03 PST 2013\n finished: Thu Dec 26 16:55:03 PST 2013\n status: failure\n residentOn: com.xensource.xenapi.Host@40561b9c\n progress: 1.0\n type: <none/>\n result: \n errorInfo: [VM_REQUIRES_SR, OpaqueRef:6e9c09e8-9ed5-1e21-2048-d848a084a15a, OpaqueRef:eec84d31-9691-1ba4-f7bb-39db42ac8628]\n otherConfig: {}\n subtaskOf: com.xensource.xenapi.Task@aaf13f6f\n subtasks: []\n","wait":0}}] }
> 2013-12-26 16:51:25,342 DEBUG [agent.transport.Request] (HA-Worker-0:work-16) Seq 4-961806364: Received: { Ans: , MgmtId: 7692017993539, via: 4, Ver: v1, Flags: 110, { MigrateAnswer } }
> 2013-12-26 16:51:25,342 ERROR [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-16) Unable to migrate due to Catch Exception com.cloud.utils.exception.CloudRuntimeException: Migration failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to migrate VM(s-1-VM) from host(d044a044-1a30-4c01-ae83-11d2631a4ffe) due to Task failed! Task record: uuid: 38ce6cd4-9ab2-7e46-4d10-dafe25a83084
> nameLabel: Async.VM.pool_migrate
> nameDescription:
> allowedOperations: []
> currentOperations: {}
> created: Thu Dec 26 16:55:03 PST 2013
> finished: Thu Dec 26 16:55:03 PST 2013
> status: failure
> residentOn: com.xensource.xenapi.Host@40561b9c
> progress: 1.0
> type: <none/>
> result:
> errorInfo: [VM_REQUIRES_SR, OpaqueRef:6e9c09e8-9ed5-1e21-2048-d848a084a15a, OpaqueRef:eec84d31-9691-1ba4-f7bb-39db42ac8628]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
> 2013-12-26 16:51:25,342 INFO [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-16) Migration was unsuccessful. Cleaning up: VM[SecondaryStorageVm|s-1-VM]
> 2013-12-26 16:51:25,346 DEBUG [agent.manager.AgentAttache] (DirectAgent-57:null) Seq 4-961806364: No more commands found
> 2013-12-26 16:51:25,355 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-16) alertType:: 17 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to migrate vm s-1-VM from host Rack2Host19.lab.vmops.com in zone z1 and pod z1p1
> 2013-12-26 16:51:25,356 DEBUG [cloud.alert.AlertManagerImpl] (HA-Worker-0:work-16) Have already sent: 1 emails for alert type '17' -- skipping send email
> 2013-12-26 16:51:25,357 DEBUG [agent.manager.AgentAttache] (HA-Worker-0:work-16) Seq 1-265617551: Routed from 7692017993539
> 2013-12-26 16:51:25,358 DEBUG [agent.transport.Request] (HA-Worker-0:work-16) Seq 1-265617551: Sending { Cmd , MgmtId: 7692017993539, via: 1, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"s-1-VM","wait":0}}] }
> 2013-12-26 16:51:25,358 DEBUG [agent.transport.Request] (HA-Worker-0:work-16) Seq 1-265617551: Executing: { Cmd , MgmtId: 7692017993539, via: 1, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"s-1-VM","wait":0}}] }
> 2013-12-26 16:51:25,358 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-62:null) Seq 1-265617551: Executing request
> 2013-12-26 16:51:25,368 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-0:work-16) VM state transitted from :Migrating to Running with event: OperationFailedvm's original host id: 4 new host id: 4 host id before state transition: 1
> 2013-12-26 16:51:25,375 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-0:work-16) Hosts's actual total CPU: 9044 and CPU after applying overprovisioning: 9044
> 2013-12-26 16:51:25,375 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-0:work-16) Hosts's actual total RAM: 15988871808 and RAM after applying overprovisioning: 15988872192
> 2013-12-26 16:51:25,375 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-0:work-16) release cpu from host: 1, old used: 500,reserved: 0, actual total: 9044, total with overprovisioning: 9044; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
> 2013-12-26 16:51:25,376 DEBUG [cloud.capacity.CapacityManagerImpl] (HA-Worker-0:work-16) release mem from host: 1, old used: 268435456,reserved: 0, total: 15988872192; new used: 0,reserved:0; movedfromreserved: false,moveToReserveredfalse
> 2013-12-26 16:51:25,385 INFO [xen.resource.CitrixResourceBase] (DirectAgent-62:null) VM does not exist on XenServerc6c75483-b35b-4a7c-9e2d-2274e34dd437
> 2013-12-26 16:51:25,385 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-62:null) Seq 1-265617551: Response Received:
> 2013-12-26 16:51:25,386 DEBUG [agent.transport.Request] (DirectAgent-62:null) Seq 1-265617551: Processing: { Ans: , MgmtId: 7692017993539, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not exist","wait":0}}] }
> 2013-12-26 16:51:25,386 DEBUG [agent.manager.AgentAttache] (DirectAgent-62:null) Seq 1-265617551: Unable to find listener.
> 2013-12-26 16:51:25,386 DEBUG [agent.manager.AgentAttache] (DirectAgent-62:null) Seq 1-265617551: No more commands found
> 2013-12-26 16:51:25,390 DEBUG [vm.dao.VMInstanceDaoImpl] (HA-Worker-0:work-16) Unable to update VM[SecondaryStorageVm|s-1-VM]: DB Data={Host=4; State=Running; updated=13; time=Thu Dec 26 16:51:25 PST 2013} New Data: {Host=4; State=Stopping; updated=12; time=Thu Dec 26 16:51:25 PST 2013} Stale Data: {Host=4; State=Running; updated=11; time=Thu Dec 26 16:51:23 PST 2013}
> 2013-12-26 16:51:25,390 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-16) Unable to stop VM due to VM is being operated on.
> 2013-12-26 16:51:25,390 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-16) Unable to migrate vm from 4
> 2013-12-26 16:51:25,391 DEBUG [cloud.resource.ResourceManagerImpl] (HA-Worker-0:work-16) No next resource state for host 4 while current state is ErrorInMaintenance with event UnableToMigrate
> com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state =ErrorInMaintenance event =UnableToMigrate
> at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178)
> at com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313)
> at com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:602)
> at com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858)
> 2013-12-26 16:51:25,392 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-16) Completed HAWork[16-Migration-1-Running-Migrating]
> 2013-12-26 16:51:26,339 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-10:null) SeqA 2-416: Processing Seq 2-416: { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] }
> 2013-12-26 16:51:26,346 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-10:null) SeqA 2-416: Sending Seq 2-416: { Ans: , MgmtId: 7692017993539, via: 2, Ver: v1, Flags: 100010, [{"com.cloud.agent.ap
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)