You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Koushik Das (JIRA)" <ji...@apache.org> on 2013/07/10 13:15:48 UTC
[jira] [Assigned] (CLOUDSTACK-3248) XenServer Host got removed
successfully inspite of running VMs on the host
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koushik Das reassigned CLOUDSTACK-3248:
---------------------------------------
Assignee: Koushik Das
> XenServer Host got removed successfully inspite of running VMs on the host
> --------------------------------------------------------------------------
>
> Key: CLOUDSTACK-3248
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3248
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Reporter: Chandan Purushothama
> Assignee: Koushik Das
> Priority: Critical
> Fix For: 4.2.0
>
>
> ==============
> From the API log:
> ==============
> 2013-06-27 11:48:36,855 INFO [cloud.api.ApiServer] (catalina-exec-15:null) (userId=2 accountId=2 sessionId=62A69AC91786781EA9BC2F5E534F5590) 10.252.120.177 -- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625 200 { "deletehostresponse" : { "success" : "true"} }
> ==========================
> From the Management Server Log:
> ==========================
> 2013-06-27 11:48:36,765 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===START=== 10.252.120.177 -- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625
> 2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to ConsoleProxyManagerImpl_EnhancerByCloudStack_edb8e808
> 2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to PremiumSecondaryStorageManagerImpl_EnhancerByCloudStack_9f8bf2f0
> 2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to JuniperSRXExternalFirewallElement_EnhancerByCloudStack_c0d55fca
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to KvmServerDiscoverer_EnhancerByCloudStack_6a93cc3b
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to VmwareServerDiscoverer_EnhancerByCloudStack_c0f46d01
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to NetscalerElement_EnhancerByCloudStack_2c0cf22
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to NetworkUsageManagerImpl_EnhancerByCloudStack_fed7434f
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to F5ExternalLoadBalancerElement_EnhancerByCloudStack_91202979
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to NiciraNvp
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to CiscoVnmcElement_EnhancerByCloudStack_439d7cf0
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to OvmDiscoverer_EnhancerByCloudStack_d6ea6366
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to LxcServerDiscoverer_EnhancerByCloudStack_2f0a46f8
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to OvmDiscoverer_EnhancerByCloudStack_d6ea6366
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to LxcServerDiscoverer_EnhancerByCloudStack_2f0a46f8
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to XcpServerDiscoverer_EnhancerByCloudStack_6d80ebbf
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Deleting Host: 5 Guid:b213f6c1-16d6-4fa3-8998-dce12a65d1e3
> 2013-06-27 11:48:36,806 DEBUG [dc.dao.DataCenterIpAddressDaoImpl] (catalina-exec-15:null) Releasing ip address: 10.223.57.194 data center 1
> 2013-06-27 11:48:36,810 DEBUG [cloud.host.Status] (catalina-exec-15:null) Transition:[Resource state = Enabled, Agent event = Remove, Host id = 5, name = Rack3Host8.lab.vmops.com]
> 2013-06-27 11:48:36,812 DEBUG [cloud.host.Status] (catalina-exec-15:null) Agent status update: [id = 5; name = Rack3Host8.lab.vmops.com; old status = Disconnected; event = Remove; new status = Removed; old update count = 12; new update count = 13]
> 2013-06-27 11:48:36,847 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Cannot transmit host 5to Enabled state
> com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state =Enabled event =DeleteHost
> at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1367)
> at com.cloud.resource.ResourceManagerImpl.doDeleteHost(ResourceManagerImpl.java:999)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at com.cloud.resource.ResourceManagerImpl.deleteHost(ResourceManagerImpl.java:1055)
> at org.apache.cloudstack.api.command.admin.host.DeleteHostCmd.execute(DeleteHostCmd.java:83)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:528)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:371)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-27 11:48:36,855 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===END=== 10.252.120.177 -- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625
> ===============
> From the Database:
> ===============
> mysql> select id,name,status,uuid,type,private_ip_address,created,removed from host where type="Routing";
> +----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
> | id | name | status | uuid | type | private_ip_address | created | removed |
> +----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
> | 1 | Rack3Host20.lab.vmops.com | Up | 9b22f04f-f59d-4145-bddb-138027e4cf56 | Routing | 10.223.59.66 | 2013-06-27 01:08:20 | NULL |
> | 5 | Rack3Host8.lab.vmops.com | Removed | 405585d6-3579-4c8b-82a0-425158ac061a | Routing | 10.223.57.194 | 2013-06-27 01:27:34 | 2013-06-27 18:48:36 |
> +----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
> 2 rows in set (0.00 sec)
> mysql> select id,name,instance_name,state from vm_instance where host_id=5;
> +----+------------+---------------+---------+
> | id | name | instance_name | state |
> +----+------------+---------------+---------+
> | 4 | san | i-3-4-NTIERR | Running |
> | 5 | r-5-NTIERR | r-5-NTIERR | Running |
> | 6 | san-5 | i-3-6-NTIERR | Running |
> | 7 | san-4 | i-3-7-NTIERR | Running |
> | 8 | san-3 | i-3-8-NTIERR | Running |
> | 9 | san-1 | i-3-9-NTIERR | Running |
> | 10 | san-2 | i-3-10-NTIERR | Running |
> | 11 | san-13 | i-3-11-NTIERR | Running |
> | 14 | san-12 | i-3-14-NTIERR | Running |
> | 15 | san-19 | i-3-15-NTIERR | Running |
> | 17 | san-7 | i-3-17-NTIERR | Running |
> | 18 | san-16 | i-3-18-NTIERR | Running |
> | 21 | san-11 | i-3-21-NTIERR | Running |
> +----+------------+---------------+---------+
> 13 rows in set (0.00 sec)
> Any Agent Operation on the above VMs fail
> 2013-06-27 12:12:12,071 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===START=== 10.252.120.177 -- GET command=destroyVirtualMachine&id=c5e5a1c2-f830-4f0b-8250-e548b48681a1&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372360378188
> 2013-06-27 12:12:12,157 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-24:null) submit async job-35, details: AsyncJobVO {id:35, userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId: 4, cmd: org.apache.cloudstack.api.command.user.vm.DestroyVMCmd, cmdOriginator: null, cmdInfo: {"id":"c5e5a1c2-f830-4f0b-8250-e548b48681a1","response":"json","sessionkey":"LAHSBLWAfVc8ti8IX/CBeUyvaJQ\u003d","ctxUserId":"2","httpmethod":"GET","_":"1372360378188","ctxAccountId":"2","ctxStartEventId":"170"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 7471666038533, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-06-27 12:12:12,159 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===END=== 10.252.120.177 -- GET command=destroyVirtualMachine&id=c5e5a1c2-f830-4f0b-8250-e548b48681a1&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372360378188
> 2013-06-27 12:12:12,182 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Executing org.apache.cloudstack.api.command.user.vm.DestroyVMCmd for job-35
> 2013-06-27 12:12:12,253 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Destroying vm VM[User|san]
> 2013-06-27 12:12:12,285 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-35) VM state transitted from :Running to Stopping with event: StopRequestedvm's original host id: 5 new host id: 5 host id before state transition: 5
> 2013-06-27 12:12:12,288 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Unable to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException: Resource [Host:5] is unreachable: Host 5: Can't find the host
> 2013-06-27 12:12:12,288 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Unable to stop vm VM[User|san]
> 2013-06-27 12:12:12,294 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-35) VM state transitted from :Stopping to Running with event: OperationFailedvm's original host id: 5 new host id: 5 host id before state transition: 5
> 2013-06-27 12:12:12,294 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Unable to stop VM[User|san]
> 2013-06-27 12:12:12,300 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Unexpected exception while executing org.apache.cloudstack.api.command.user.vm.DestroyVMCmd
> com.cloud.utils.exception.CloudRuntimeException: Failed to destroy vm with specified vmId
> at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:3386)
> at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:1881)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.api.command.user.vm.DestroyVMCmd.execute(DestroyVMCmd.java:100)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-27 12:12:12,300 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Complete async job-35, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to destroy vm with specified vmId
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira