You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Koushik Das (JIRA)" <ji...@apache.org> on 2013/07/10 13:19:58 UTC
[jira] [Commented] (CLOUDSTACK-3248) XenServer Host got removed
successfully inspite of running VMs on the host
[ https://issues.apache.org/jira/browse/CLOUDSTACK-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704441#comment-13704441 ]
Koushik Das commented on CLOUDSTACK-3248:
-----------------------------------------
The correct way to remove a host is to first put it in maintenance and then remove it. This ensures that all VMs running in it are either migrated or stopped. This check is currently only enforced in UI. Using the API directly a host can be deleted without putting it in maintenance. This leaves the system in a bad state wherein the host is no longer there but VMs running on it are still present. Not sure as to why this wasn't found earlier.
> XenServer Host got removed successfully inspite of running VMs on the host
> --------------------------------------------------------------------------
>
> Key: CLOUDSTACK-3248
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3248
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Reporter: Chandan Purushothama
> Assignee: Koushik Das
> Priority: Critical
> Fix For: 4.2.0
>
>
> ==============
> From the API log:
> ==============
> 2013-06-27 11:48:36,855 INFO [cloud.api.ApiServer] (catalina-exec-15:null) (userId=2 accountId=2 sessionId=62A69AC91786781EA9BC2F5E534F5590) 10.252.120.177 -- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625 200 { "deletehostresponse" : { "success" : "true"} }
> ==========================
> From the Management Server Log:
> ==========================
> 2013-06-27 11:48:36,765 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===START=== 10.252.120.177 -- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625
> 2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to ConsoleProxyManagerImpl_EnhancerByCloudStack_edb8e808
> 2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to PremiumSecondaryStorageManagerImpl_EnhancerByCloudStack_9f8bf2f0
> 2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to JuniperSRXExternalFirewallElement_EnhancerByCloudStack_c0d55fca
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to KvmServerDiscoverer_EnhancerByCloudStack_6a93cc3b
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to VmwareServerDiscoverer_EnhancerByCloudStack_c0f46d01
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to NetscalerElement_EnhancerByCloudStack_2c0cf22
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to NetworkUsageManagerImpl_EnhancerByCloudStack_fed7434f
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to F5ExternalLoadBalancerElement_EnhancerByCloudStack_91202979
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to NiciraNvp
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to CiscoVnmcElement_EnhancerByCloudStack_439d7cf0
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to OvmDiscoverer_EnhancerByCloudStack_d6ea6366
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to LxcServerDiscoverer_EnhancerByCloudStack_2f0a46f8
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to OvmDiscoverer_EnhancerByCloudStack_d6ea6366
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to LxcServerDiscoverer_EnhancerByCloudStack_2f0a46f8
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to XcpServerDiscoverer_EnhancerByCloudStack_6d80ebbf
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Deleting Host: 5 Guid:b213f6c1-16d6-4fa3-8998-dce12a65d1e3
> 2013-06-27 11:48:36,806 DEBUG [dc.dao.DataCenterIpAddressDaoImpl] (catalina-exec-15:null) Releasing ip address: 10.223.57.194 data center 1
> 2013-06-27 11:48:36,810 DEBUG [cloud.host.Status] (catalina-exec-15:null) Transition:[Resource state = Enabled, Agent event = Remove, Host id = 5, name = Rack3Host8.lab.vmops.com]
> 2013-06-27 11:48:36,812 DEBUG [cloud.host.Status] (catalina-exec-15:null) Agent status update: [id = 5; name = Rack3Host8.lab.vmops.com; old status = Disconnected; event = Remove; new status = Removed; old update count = 12; new update count = 13]
> 2013-06-27 11:48:36,847 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-15:null) Cannot transmit host 5to Enabled state
> com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state =Enabled event =DeleteHost
> at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1367)
> at com.cloud.resource.ResourceManagerImpl.doDeleteHost(ResourceManagerImpl.java:999)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at com.cloud.resource.ResourceManagerImpl.deleteHost(ResourceManagerImpl.java:1055)
> at org.apache.cloudstack.api.command.admin.host.DeleteHostCmd.execute(DeleteHostCmd.java:83)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:528)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:371)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-27 11:48:36,855 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===END=== 10.252.120.177 -- GET command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625
> ===============
> From the Database:
> ===============
> mysql> select id,name,status,uuid,type,private_ip_address,created,removed from host where type="Routing";
> +----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
> | id | name | status | uuid | type | private_ip_address | created | removed |
> +----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
> | 1 | Rack3Host20.lab.vmops.com | Up | 9b22f04f-f59d-4145-bddb-138027e4cf56 | Routing | 10.223.59.66 | 2013-06-27 01:08:20 | NULL |
> | 5 | Rack3Host8.lab.vmops.com | Removed | 405585d6-3579-4c8b-82a0-425158ac061a | Routing | 10.223.57.194 | 2013-06-27 01:27:34 | 2013-06-27 18:48:36 |
> +----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
> 2 rows in set (0.00 sec)
> mysql> select id,name,instance_name,state from vm_instance where host_id=5;
> +----+------------+---------------+---------+
> | id | name | instance_name | state |
> +----+------------+---------------+---------+
> | 4 | san | i-3-4-NTIERR | Running |
> | 5 | r-5-NTIERR | r-5-NTIERR | Running |
> | 6 | san-5 | i-3-6-NTIERR | Running |
> | 7 | san-4 | i-3-7-NTIERR | Running |
> | 8 | san-3 | i-3-8-NTIERR | Running |
> | 9 | san-1 | i-3-9-NTIERR | Running |
> | 10 | san-2 | i-3-10-NTIERR | Running |
> | 11 | san-13 | i-3-11-NTIERR | Running |
> | 14 | san-12 | i-3-14-NTIERR | Running |
> | 15 | san-19 | i-3-15-NTIERR | Running |
> | 17 | san-7 | i-3-17-NTIERR | Running |
> | 18 | san-16 | i-3-18-NTIERR | Running |
> | 21 | san-11 | i-3-21-NTIERR | Running |
> +----+------------+---------------+---------+
> 13 rows in set (0.00 sec)
> Any Agent Operation on the above VMs fail
> 2013-06-27 12:12:12,071 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===START=== 10.252.120.177 -- GET command=destroyVirtualMachine&id=c5e5a1c2-f830-4f0b-8250-e548b48681a1&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372360378188
> 2013-06-27 12:12:12,157 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-24:null) submit async job-35, details: AsyncJobVO {id:35, userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine, instanceId: 4, cmd: org.apache.cloudstack.api.command.user.vm.DestroyVMCmd, cmdOriginator: null, cmdInfo: {"id":"c5e5a1c2-f830-4f0b-8250-e548b48681a1","response":"json","sessionkey":"LAHSBLWAfVc8ti8IX/CBeUyvaJQ\u003d","ctxUserId":"2","httpmethod":"GET","_":"1372360378188","ctxAccountId":"2","ctxStartEventId":"170"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 7471666038533, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-06-27 12:12:12,159 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null) ===END=== 10.252.120.177 -- GET command=destroyVirtualMachine&id=c5e5a1c2-f830-4f0b-8250-e548b48681a1&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372360378188
> 2013-06-27 12:12:12,182 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Executing org.apache.cloudstack.api.command.user.vm.DestroyVMCmd for job-35
> 2013-06-27 12:12:12,253 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Destroying vm VM[User|san]
> 2013-06-27 12:12:12,285 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-35) VM state transitted from :Running to Stopping with event: StopRequestedvm's original host id: 5 new host id: 5 host id before state transition: 5
> 2013-06-27 12:12:12,288 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Unable to stop vm, agent unavailable: com.cloud.exception.AgentUnavailableException: Resource [Host:5] is unreachable: Host 5: Can't find the host
> 2013-06-27 12:12:12,288 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Unable to stop vm VM[User|san]
> 2013-06-27 12:12:12,294 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-1:job-35) VM state transitted from :Stopping to Running with event: OperationFailedvm's original host id: 5 new host id: 5 host id before state transition: 5
> 2013-06-27 12:12:12,294 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-35) Unable to stop VM[User|san]
> 2013-06-27 12:12:12,300 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Unexpected exception while executing org.apache.cloudstack.api.command.user.vm.DestroyVMCmd
> com.cloud.utils.exception.CloudRuntimeException: Failed to destroy vm with specified vmId
> at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:3386)
> at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:1881)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at org.apache.cloudstack.api.command.user.vm.DestroyVMCmd.execute(DestroyVMCmd.java:100)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
> at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-27 12:12:12,300 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-1:job-35) Complete async job-35, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to destroy vm with specified vmId
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira