You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Chandan Purushothama (JIRA)" <ji...@apache.org> on 2013/06/27 23:36:20 UTC
[jira] [Created] (CLOUDSTACK-3248) XenServer Host got removed
successfully inspite of running VMs on the host
Chandan Purushothama created CLOUDSTACK-3248:
------------------------------------------------
Summary: XenServer Host got removed successfully inspite of running VMs on the host
Key: CLOUDSTACK-3248
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3248
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0
Reporter: Chandan Purushothama
Priority: Critical
Fix For: 4.2.0
==============
>From the API log:
==============
2013-06-27 11:48:36,855 INFO [cloud.api.ApiServer] (catalina-exec-15:null) (userId=2 accountId=2 sessionId=62A69AC91786781EA9BC2F5E534F5590) 10.252.120.177 -- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625 200 { "deletehostresponse" : { "success" : "true"} }
==========================
>From the Management Server Log:
==========================
2013-06-27 11:48:36,765 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===START=== 10.252.120.177 -- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625
2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to ConsoleProxyManagerImpl_EnhancerByCloudStack_edb8e808
2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to PremiumSecondaryStorageManagerImpl_EnhancerByCloudStack_9f8bf2f0
2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to JuniperSRXExternalFirewallElement_EnhancerByCloudStack_c0d55fca
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to KvmServerDiscoverer_EnhancerByCloudStack_6a93cc3b
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to VmwareServerDiscoverer_EnhancerByCloudStack_c0f46d01
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to NetscalerElement_EnhancerByCloudStack_2c0cf22
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to NetworkUsageManagerImpl_EnhancerByCloudStack_fed7434f
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to F5ExternalLoadBalancerElement_EnhancerByCloudStack_91202979
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to NiciraNvp
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to CiscoVnmcElement_EnhancerByCloudStack_439d7cf0
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to OvmDiscoverer_EnhancerByCloudStack_d6ea6366
2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to LxcServerDiscoverer_EnhancerByCloudStack_2f0a46f8
2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to OvmDiscoverer_EnhancerByCloudStack_d6ea6366
2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to LxcServerDiscoverer_EnhancerByCloudStack_2f0a46f8
2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to XcpServerDiscoverer_EnhancerByCloudStack_6d80ebbf
2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Deleting Host: 5 Guid:b213f6c1-16d6-4fa3-8998-dce12a65d1e3
2013-06-27 11:48:36,806 DEBUG [dc.dao.DataCenterIpAddressDaoImpl] (catalina-exec-15:null) Releasing ip address: 10.223.57.194 data center 1
2013-06-27 11:48:36,810 DEBUG [cloud.host.Status] (catalina-exec-15:null) Transition:[Resource state = Enabled, Agent event = Remove, Host id = 5, name = Rack3Host8.lab.vmops.com]
2013-06-27 11:48:36,812 DEBUG [cloud.host.Status] (catalina-exec-15:null) Agent status update: [id = 5; name = Rack3Host8.lab.vmops.com; old status = Disconnected; event = Remove; new status = Removed; old update count = 12; new update count = 13]
2013-06-27 11:48:36,847 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Cannot transmit host 5to Enabled state
com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state =Enabled event =DeleteHost
at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1367)
at com.cloud.resource.ResourceManagerImpl.doDeleteHost(ResourceManagerImpl.java:999)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at com.cloud.resource.ResourceManagerImpl.deleteHost(ResourceManagerImpl.java:1055)
at org.apache.cloudstack.api.command.admin.host.DeleteHostCmd.execute(DeleteHostCmd.java:83)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:528)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:371)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-06-27 11:48:36,855 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===END=== 10.252.120.177 -- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625
===============
>From the Database:
===============
mysql> select id,name,status,uuid,type,private_ip_address,created,removed from host where type="Routing";
+----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
| id | name | status | uuid | type | private_ip_address | created | removed |
+----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
| 1 | Rack3Host20.lab.vmops.com | Up | 9b22f04f-f59d-4145-bddb-138027e4cf56 | Routing | 10.223.59.66 | 2013-06-27 01:08:20 | NULL |
| 5 | Rack3Host8.lab.vmops.com | Removed | 405585d6-3579-4c8b-82a0-425158ac061a | Routing | 10.223.57.194 | 2013-06-27 01:27:34 | 2013-06-27 18:48:36 |
+----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
2 rows in set (0.00 sec)
mysql> select id,name,instance_name,state from vm_instance where host_id=5;
+----+------------+---------------+---------+
| id | name | instance_name | state |
+----+------------+---------------+---------+
| 4 | san | i-3-4-NTIERR | Running |
| 5 | r-5-NTIERR | r-5-NTIERR | Running |
| 6 | san-5 | i-3-6-NTIERR | Running |
| 7 | san-4 | i-3-7-NTIERR | Running |
| 8 | san-3 | i-3-8-NTIERR | Running |
| 9 | san-1 | i-3-9-NTIERR | Running |
| 10 | san-2 | i-3-10-NTIERR | Running |
| 11 | san-13 | i-3-11-NTIERR | Running |
| 14 | san-12 | i-3-14-NTIERR | Running |
| 15 | san-19 | i-3-15-NTIERR | Running |
| 17 | san-7 | i-3-17-NTIERR | Running |
| 18 | san-16 | i-3-18-NTIERR | Running |
| 21 | san-11 | i-3-21-NTIERR | Running |
+----+------------+---------------+---------+
13 rows in set (0.00 sec)
Any Agent Operation on the above VMs fail
2013-06-27 12:12:12,071 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===START=== 10.252.120.177 -- GET command=destroyVirtualMachine&id=c5e5a1c2-f830-4f0b-8250-e548b48681a1&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372360378188
2013-06-27 12:12:12,157 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-24:null) submit async job-35, details: AsyncJobVO {id:35, userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId: 4, cmd: org.apache.cloudstack.api.command.user.vm.DestroyVMCmd, cmdOriginator: null, cmdInfo: {"id":"c5e5a1c2-f830-4f0b-8250-e548b48681a1","response":"json","sessionkey":"LAHSBLWAfVc8ti8IX/CBeUyvaJQ\u003d","ctxUserId":"2","httpmethod":"GET","_":"1372360378188","ctxAccountId":"2","ctxStartEventId":"170"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 7471666038533, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-06-27 12:12:12,159 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===END=== 10.252.120.177 -- GET command=destroyVirtualMachine&id=c5e5a1c2-f830-4f0b-8250-e548b48681a1&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372360378188
2013-06-27 12:12:12,182 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Executing org.apache.cloudstack.api.command.user.vm.DestroyVMCmd for job-35
2013-06-27 12:12:12,253 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Destroying vm VM[User|san]
2013-06-27 12:12:12,285 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-35) VM state transitted from :Running to Stopping with event: StopRequestedvm's original host id: 5 new host id: 5 host id before state transition: 5
2013-06-27 12:12:12,288 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Unable to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException: Resource [Host:5] is unreachable: Host 5: Can't find the host
2013-06-27 12:12:12,288 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Unable to stop vm VM[User|san]
2013-06-27 12:12:12,294 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-35) VM state transitted from :Stopping to Running with event: OperationFailedvm's original host id: 5 new host id: 5 host id before state transition: 5
2013-06-27 12:12:12,294 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Unable to stop VM[User|san]
2013-06-27 12:12:12,300 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Unexpected exception while executing org.apache.cloudstack.api.command.user.vm.DestroyVMCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to destroy vm with specified vmId
at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:3386)
at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:1881)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DestroyVMCmd.execute(DestroyVMCmd.java:100)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-06-27 12:12:12,300 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Complete async job-35, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to destroy vm with specified vmId
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira