SSVM、CPVM无法启动成功，然后xenServer集群下的所有主机自动重启。

[zt <zh...@qq.com>]

云平台部署环境：cloudstack4.0.2+xenServer6.0.2基本网络模式
问题描述：
    SSVM、CPVM无法启动成功，然后xenServer集群下的所有主机自动重启。
管理节点上报日志如下：
2015-02-03 10:09:45,562 INFO [utils.exception.CSExceptionErrorCode] (secstorage-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions
2015-02-03 10:09:45,562 WARN [agent.manager.AgentAttache] (secstorage-1:null) Seq 33-160759819: Timed out on Seq 33-160759819: { Cmd , MgmtId: 345052348430, via: 33, Ver: v1, Flags: 100111, [{"storage.CreateCommand":{"volId":3325,"pool":{"id":200,"uuid":"1a8e96b6-04a7-3b02-a04a-56ccdf4c54d3","host":"10.6.26.200","path":"/BMDPrimary1","port":2049,"type":"NetworkFilesystem"},"diskCharacteristics":{"size":565240320,"tags":[],"type":"ROOT","name":"ROOT-3251","useLocalStorage":false,"recreatable":true,"diskOfferingId":7,"volumeId":3325,"hyperType":"XenServer"},"templateUrl":"3651f783-6bf2-416b-8a8f-f5406c948ccd","wait":0}}] }
2015-02-03 10:09:45,562 DEBUG [agent.manager.AgentAttache] (secstorage-1:null) Seq 33-160759819: Cancelling.
2015-02-03 10:09:45,562 DEBUG [cloud.storage.StorageManagerImpl] (secstorage-1:null) Unable to send storage pool command to Pool[200|NetworkFilesystem] via 33
com.cloud.exception.OperationTimedoutException: Commands 160759819 to Host 33 timed out after 1800
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:421)
at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:505)
at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:458)
at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:2215)
at com.cloud.storage.StorageManagerImpl.sendToPool(StorageManagerImpl.java:482)
at com.cloud.storage.StorageManagerImpl.createVolume(StorageManagerImpl.java:3464)
at com.cloud.storage.StorageManagerImpl.prepare(StorageManagerImpl.java:3351)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:754)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:472)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:465)
at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:257)
at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:684)
at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1310)
at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:119)
at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:106)
at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:34)
at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:83)
at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:73)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2015-02-03 10:09:45,565 DEBUG [agent.transport.Request] (secstorage-1:null) Seq 1-1717043215: Sending { Cmd , MgmtId: 345052348430, via: 1, Ver: v1, Flags: 100111, [{"storage.CreateCommand":{"volId":3325,"pool":{"id":200,"uuid":"1a8e96b6-04a7-3b02-a04a-56ccdf4c54d3","host":"10.6.26.200","path":"/BMDPrimary1","port":2049,"type":"NetworkFilesystem"},"diskCharacteristics":{"size":565240320,"tags":[],"type":"ROOT","name":"ROOT-3251","useLocalStorage":false,"recreatable":true,"diskOfferingId":7,"volumeId":3325,"hyperType":"XenServer"},"templateUrl":"3651f783-6bf2-416b-8a8f-f5406c948ccd","wait":0}}] }
2015-02-03 10:09:45,565 DEBUG [agent.transport.Request] (secstorage-1:null) Seq 1-1717043215: Executing: { Cmd , MgmtId: 345052348430, via: 1, Ver: v1, Flags: 100111, [{"storage.CreateCommand":{"volId":3325,"pool":{"id":200,"uuid":"1a8e96b6-04a7-3b02-a04a-56ccdf4c54d3","host":"10.6.26.200","path":"/BMDPrimary1","port":2049,"type":"NetworkFilesystem"},"diskCharacteristics":{"size":565240320,"tags":[],"type":"ROOT","name":"ROOT-3251","useLocalStorage":false,"recreatable":true,"diskOfferingId":7,"volumeId":3325,"hyperType":"XenServer"},"templateUrl":"3651f783-6bf2-416b-8a8f-f5406c948ccd","wait":0}}] }
2015-02-03 10:09:45,565 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-160:null) Seq 1-1717043215: Executing request
2015-02-03 10:09:46,563 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager vm manager to shutdown.
2015-02-03 10:09:46,563 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager vm manager because it is not in a state to shutdown.
2015-02-03 10:09:46,563 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager upgrade manager to shutdown.
2015-02-03 10:09:46,563 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager StorageManager to shutdown.
2015-02-03 10:09:46,563 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager StorageManager because it is not in a state to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager Alert Manager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager Template Manager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager Template Manager because it is not in a state to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager Snapshot Manager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager Snapshot Manager because it is not in a state to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager SnapshotScheduler to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager SecurityGroupManager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager SecurityGroupManager because it is not in a state to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager EntityManager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager LoadBalancingRulesManager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager LoadBalancingRulesManager because it is not in a state to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager RulesManager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager RulesManager because it is not in a state to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager RemoteAccessVpnManager to shutdown.
2015-02-03 10:09:46,564 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager Capacity Manager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager VirtualMachineManager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager HypervisorGuruManager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager ResourceManager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager ResourceManager because it is not in a state to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager IdentityManager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager OCFS2Manager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager FirewallManager to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager FirewallManager because it is not in a state to shutdown.
2015-02-03 10:09:46,565 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager ConsoleProxyManager to shutdown.
2015-02-03 10:09:46,565 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl] (Thread-5:null) Stop console proxy manager
2015-02-03 10:09:46,566 DEBUG [agent.manager.AgentAttache] (consoleproxy-1:null) Seq 33-160759818: Interrupted
2015-02-03 10:09:46,566 DEBUG [agent.manager.AgentAttache] (consoleproxy-1:null) Seq 33-160759818: Waiting some more time because this is the current command
2015-02-03 10:09:47,566 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager ConsoleProxyManager because it is not in a state to shutdown.
2015-02-03 10:09:47,566 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager ProjectManager to shutdown.
2015-02-03 10:09:47,566 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager ProjectManager because it is not in a state to shutdown.
2015-02-03 10:09:47,566 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager SwiftManager to shutdown.
2015-02-03 10:09:47,566 INFO [storage.swift.SwiftManagerImpl] (Thread-5:null) Stop Swift Manager
2015-02-03 10:09:47,567 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager StorageNetworkManager to shutdown.
2015-02-03 10:09:47,567 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager StorageNetworkManager because it is not in a state to shutdown.
2015-02-03 10:09:47,567 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager ExternalLoadBalancerUsageManager to shutdown.
2015-02-03 10:09:47,567 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager HA Manager to shutdown.
2015-02-03 10:09:47,567 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:null) Time to go home!
2015-02-03 10:09:47,568 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:null) Time to go home!
2015-02-03 10:09:47,568 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:null) Time to go home!
2015-02-03 10:09:47,567 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager VPC Manager to shutdown.
2015-02-03 10:09:47,568 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:null) Time to go home!
2015-02-03 10:09:47,567 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-2:null) Time to go home!
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager VPC Manager because it is not in a state to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager VpcVirtualRouterManager to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager VpcVirtualRouterManager because it is not in a state to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager NetworkACLManager to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager NetworkACLManager because it is not in a state to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager TaggedResourcesManager to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager Site2SiteVpnManager to shutdown.
2015-02-03 10:09:47,568 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager Site2SiteVpnManager because it is not in a state to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager ExternalNetworkManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager BareMetalVmManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager BareMetalVmManager because it is not in a state to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager ExternalDhcpManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager PxeServerManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager NetworkUsageManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager OvsTunnelManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager NetappManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager ElasticLoadBalancerManager to shutdown.
2015-02-03 10:09:47,569 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager VmwareManager to shutdown.
2015-02-03 10:09:47,569 INFO [vmware.manager.VmwareManagerImpl] (Thread-5:null) Cleanup mounted NFS mount points used in current session
2015-02-03 10:09:47,990 DEBUG [xen.resource.XenServerConnectionPool] (DirectAgent-46:null) Unable to create slave connection to host(c568f879-25ee-4a2e-9cae-df53f39e5cc6) due to org.apache.xmlrpc.XmlRpcException: Failed to read server's response: connect timed out



zt