You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@cloudstack.apache.org by "Koushik Das (JIRA)" <ji...@apache.org> on 2013/12/11 07:12:13 UTC
[jira] [Resolved] (CLOUDSTACK-4371) [Performance Testing] Basic
zone with 20K Hosts, management server restart leaves the hosts in
disconnected state for very long time
[ https://issues.apache.org/jira/browse/CLOUDSTACK-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koushik Das resolved CLOUDSTACK-4371.
-------------------------------------
Resolution: Cannot Reproduce
> [Performance Testing] Basic zone with 20K Hosts, management server restart leaves the hosts in disconnected state for very long time
> ------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-4371
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4371
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Environment: Basic zone, with over 20K simulator hosts
> Reporter: Sowmya Krishnan
> Assignee: Koushik Das
> Labels: performance
> Fix For: 4.3.0
>
> Attachments: agenttaskpool_334.log, ms1_restartfail.log.gz, ms2_restartfail.log.gz, ms3_restartfail.log.gz
>
>
> Basic zone performance test bed:
> 20K simulator hosts,
> 3 Management servers
> 1 host/cluster
> Local storage
> Java heap size: 12GB
> db.cloud.maxActive=2000
> direct.agent.load.size=1000
> agent.lb.enabled=true
> Deploy around 20K simulator hosts with 3 Management servers clustered
> (Not deployed any VMs yet)
> After all hosts are deployed, stop all 3 Management servers and then start all 3 one after another
> Result
> =====
> Hosts don't get to connected state at all even after 10 minutes. While around 2K of them go into alert state while rest are in disconnected state.
> mysql> select count(*), status, resource_state, type, mgmt_server_id from host group by mgmt_server_id, status, type, resource_state;
> +----------+--------------+----------------+--------------------+----------------+
> | count(*) | status | resource_state | type | mgmt_server_id |
> +----------+--------------+----------------+--------------------+----------------+
> | 1946 | Alert | Enabled | Routing | NULL |
> | 18054 | Disconnected | Enabled | Routing | NULL |
> | 1 | Disconnected | Enabled | SecondaryStorageVM | NULL |
> +----------+--------------+----------------+--------------------+----------------+
> 3 rows in set (0.11 sec)
> MS Logs show lot of storage pool exceptions while hosts try to get connected:
> 2013-08-16 05:49:25,592 DEBUG [agent.transport.Request] (AgentTaskPool-12:null) Seq 13-32440322: Sending { Cmd , MgmtId: 206915885094132, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agen
> t.api.CleanupNetworkRulesCmd":{"interval":2028,"wait":0}}] }
> 2013-08-16 05:49:25,592 DEBUG [agent.transport.Request] (AgentTaskPool-12:null) Seq 13-32440322: Executing: { Cmd , MgmtId: 206915885094132, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.a
> gent.api.CleanupNetworkRulesCmd":{"interval":2028,"wait":0}}] }
> 2013-08-16 05:49:25,592 DEBUG [xen.discoverer.XcpServerDiscoverer] (AgentTaskPool-14:null) Not XenServer so moving on.
> 2013-08-16 05:49:25,592 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-14:null) Sending Connect to listener: DeploymentPlanningManagerImpl_EnhancerByCloudStack_76f3d8e4
> 2013-08-16 05:49:25,591 DEBUG [cloud.resource.AgentResourceBase] (ClusteredAgentManager Timer:null) Deserializing simulated agent on reconnect
> 2013-08-16 05:49:25,594 INFO [network.security.SecurityGroupListener] (AgentTaskPool-12:null) Scheduled network rules cleanup, interval=2028
> 2013-08-16 05:49:25,594 INFO [network.security.SecurityGroupListener] (AgentTaskPool-12:null) Received a host startup notification
> 2013-08-16 05:49:25,595 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null) Sending Connect to listener: StoragePoolMonitor
> ...
> ...
> 2013-08-16 05:49:25,761 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null) Sending Connect to listener: ClusteredVirtualMachineManagerImpl_EnhancerByCloudStack_b5459b7b
> 2013-08-16 05:49:25,764 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-12:null) Found 0 VMs for host 13
> 2013-08-16 05:49:25,765 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null) Sending Connect to listener: LocalStoragePoolListener
> 2013-08-16 05:49:25,768 DEBUG [datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl] (AgentTaskPool-12:null) createPool Params @ scheme - Filesystem storageHost - 172.1.3.131 hostPath - /mnt/2a2463b4-4fd2-4ac7-ad3f-040a3046e478 port - -1
> 2013-08-16 05:49:25,771 DEBUG [datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl] (AgentTaskPool-12:null) Another active pool with the same uuid already exists
> 2013-08-16 05:49:25,772 WARN [cloud.storage.StorageManagerImpl] (AgentTaskPool-12:null) Unable to setup the local storage pool for Host[-13-Routing]
> com.cloud.utils.exception.CloudRuntimeException: Another active pool with the same uuid already exists
> at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.initialize(CloudStackPrimaryDataStoreLifeCycleImpl.java:319)
> at com.cloud.storage.StorageManagerImpl.createLocalStorage(StorageManagerImpl.java:647)
> at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at com.cloud.storage.LocalStoragePoolListener.processConnect(LocalStoragePoolListener.java:86)
> at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:587)
> at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1479)
> at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1739)
> at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1901)
> at com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1130)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:679)
> 2013-08-16 05:49:25,773 INFO [utils.exception.CSExceptionErrorCode] (AgentTaskPool-12:null) Could not find exception: com.cloud.exception.ConnectionException in error code list for exceptions
> 2013-08-16 05:49:25,773 WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null) Monitor LocalStoragePoolListener says there is an error in the connect process for 13 due to Unable to setup the local storage pool for Host[-13-Routing]
> 2013-08-16 05:49:25,773 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null) Host 13 is disconnecting with event AgentDisconnected
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)