You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Koushik Das <ko...@citrix.com> on 2013/08/08 15:10:08 UTC
RE: [jira] [Commented] (CLOUDSTACK-4179) [Performance Testing] Time
taken for Deploy VM async job to complete is considerably higher
The increase in deployVM time is due to the dedicated resources feature. I have updated the issue with my analysis. Need some inputs from the original devs.
-Koushik
> -----Original Message-----
> From: Koushik Das (JIRA) [mailto:jira@apache.org]
> Sent: Thursday, August 08, 2013 6:37 PM
> To: cloudstack-issues@incubator.apache.org
> Subject: [jira] [Commented] (CLOUDSTACK-4179) [Performance Testing]
> Time taken for Deploy VM async job to complete is considerably higher
>
>
> [ https://issues.apache.org/jira/browse/CLOUDSTACK-
> 4179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=13733464#comment-13733464 ]
>
> Koushik Das commented on CLOUDSTACK-4179:
> -----------------------------------------
>
> The fix is to optimize the following code in
> checkForNonDedicatedResources(). Also I am not sure why it gets called
> twice during a VM deployment (have debugged through the code). Need to
> check with the developer for dedicated resources.
>
> List<HostPodVO> podsInDc = _podDao.listByDataCenterId(dc.getId());
> for (HostPodVO pod : podsInDc) {
> DedicatedResourceVO dedicatedPod =
> _dedicatedDao.findByPodId(pod.getId());
> if (dedicatedPod != null) {
> avoids.addPod(dedicatedPod.getPodId());
> if (s_logger.isDebugEnabled()) {
> s_logger.debug("Cannot use this dedicated pod " +
> pod.getName() + ".");
> }
> }
> }
> List<ClusterVO> clusterInDc =
> _clusterDao.listClustersByDcId(dc.getId());
> for (ClusterVO cluster : clusterInDc) {
> DedicatedResourceVO dedicatedCluster =
> _dedicatedDao.findByClusterId(cluster.getId());
> if (dedicatedCluster != null) {
> avoids.addCluster(dedicatedCluster.getClusterId());
> if (s_logger.isDebugEnabled()) {
> s_logger.debug("Cannot use this dedicated Cluster " +
> cluster.getName() + ".");
> }
> }
> }
> List<HostVO> hostInDc = _hostDao.listByDataCenterId(dc.getId());
> for (HostVO host : hostInDc) {
> DedicatedResourceVO dedicatedHost =
> _dedicatedDao.findByHostId(host.getId());
> if (dedicatedHost != null) {
> avoids.addHost(dedicatedHost.getHostId());
> if (s_logger.isDebugEnabled()) {
> s_logger.debug("Cannot use this dedicated host " +
> host.getName() + ".");
> }
> }
> }
>
> > [Performance Testing] Time taken for Deploy VM async job to complete
> > is considerably higher
> > ----------------------------------------------------------------------
> > ---------------------
> >
> > Key: CLOUDSTACK-4179
> > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4179
> > Project: CloudStack
> > Issue Type: Bug
> > Security Level: Public(Anyone can view this level - this is the default.)
> > Components: Management Server
> > Affects Versions: 4.2.0
> > Environment: Simulated VMs and hosts, advanced zone, RVR
> > Reporter: Sowmya Krishnan
> > Assignee: Koushik Das
> > Priority: Critical
> > Fix For: 4.2.0
> >
> > Attachments: async_job_time.out, job-109.log
> >
> >
> > Following setup is used to create performance test bed:
> > Advanced zone, 2 Management Servers, 124 Pods [Each Pod having 2
> > Clusters]
> > 248 Clusters [Each cluster having 8 hosts and one primary storage]
> > 2000 Hosts
> > 4000 User accounts [Each account having one network]
> > ~8000 Virtual Routers
> > Following config parameters were used in both the management servers
> > - Java heap size = 5 GB
> > - db.cloud.maxActive = 250
> > -
> >
> db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&prepS
> tmt
> >
> CacheSqlLimit=4096&includeInnodbStatusInDeadlockExceptions=true&logSlo
> > wQueries=true Tried deploying around 4000 Simulator VMs across 4K
> > accounts. Noticing considerably higher time for deploy VM async job to
> complete as compared to baseline results.
> > Some of the Baseline results are documented here:
> >
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Baseline+Report
> > s+for+Pre-4.x+Performance+Runs On an average, it is now taking 70-80
> > seconds for deployVM to complete as compared to 10-15 seconds earlier.
> > From the logs, it appears in some cases, DeploymentPlanningManager
> takes time to kick in.:
> > 2013-08-07 01:52:09,785 DEBUG [cloud.network.NetworkModelImpl]
> > (Job-Executor-105:job-109 = [ 19ccfecb-c815-497a-9ff1-70d590025af3 ])
> > Service SecurityGroup is not supported in the network id=304
> > 2013-08-07 01:52:37,288 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-105:job-
> 109 = [ 19ccfecb-c815-497a-9ff1-70d590025af3 ]) Deploy avoids pods: null,
> clusters: null, hosts: null ...
> > ...
> > 2013-08-07 01:52:41,199 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-105:job-109 = [ 19ccfecb-c815-497a-9ff1-70d590025af3 ])
> Deploy avoids pods:
> > null, clusters: null, hosts: null
> > 2013-08-07 01:53:19,886 DEBUG
> > [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-105:job-
> 109
> > = [ 19ccfecb-c815-497a-9ff1-70d590025af3 ]) Deploy avoid s pods: null,
> clusters: null, hosts: null Both VRs are running at the end of deploy VM and
> also the VM.
>
> --
> This message is automatically generated by JIRA.
> If you think it was sent incorrectly, please contact your JIRA administrators
> For more information on JIRA, see: http://www.atlassian.com/software/jira
RE: [jira] [Commented] (CLOUDSTACK-4179) [Performance Testing] Time
taken for Deploy VM async job to complete is considerably higher
Posted by Devdeep Singh <de...@citrix.com>.
Hi Koushik,
I have updated the bug with my comments. We don't have to loop through resources and check them one at a time. We need to run a better query and add all the dedicated resources to the avoid set.
Regards,
Devdeep
> -----Original Message-----
> From: Koushik Das
> Sent: Thursday, August 08, 2013 6:40 PM
> To: dev@cloudstack.apache.org; cloudstack-issues@incubator.apache.org
> Cc: Devdeep Singh; Prachi Damle
> Subject: RE: [jira] [Commented] (CLOUDSTACK-4179) [Performance Testing]
> Time taken for Deploy VM async job to complete is considerably higher
>
> The increase in deployVM time is due to the dedicated resources feature. I
> have updated the issue with my analysis. Need some inputs from the original
> devs.
>
> -Koushik
>
> > -----Original Message-----
> > From: Koushik Das (JIRA) [mailto:jira@apache.org]
> > Sent: Thursday, August 08, 2013 6:37 PM
> > To: cloudstack-issues@incubator.apache.org
> > Subject: [jira] [Commented] (CLOUDSTACK-4179) [Performance Testing]
> > Time taken for Deploy VM async job to complete is considerably higher
> >
> >
> > [ https://issues.apache.org/jira/browse/CLOUDSTACK-
> > 4179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> > tabpanel&focusedCommentId=13733464#comment-13733464 ]
> >
> > Koushik Das commented on CLOUDSTACK-4179:
> > -----------------------------------------
> >
> > The fix is to optimize the following code in
> > checkForNonDedicatedResources(). Also I am not sure why it gets called
> > twice during a VM deployment (have debugged through the code). Need to
> > check with the developer for dedicated resources.
> >
> > List<HostPodVO> podsInDc = _podDao.listByDataCenterId(dc.getId());
> > for (HostPodVO pod : podsInDc) {
> > DedicatedResourceVO dedicatedPod =
> > _dedicatedDao.findByPodId(pod.getId());
> > if (dedicatedPod != null) {
> > avoids.addPod(dedicatedPod.getPodId());
> > if (s_logger.isDebugEnabled()) {
> > s_logger.debug("Cannot use this dedicated pod
> > " +
> > pod.getName() + ".");
> > }
> > }
> > }
> > List<ClusterVO> clusterInDc =
> > _clusterDao.listClustersByDcId(dc.getId());
> > for (ClusterVO cluster : clusterInDc) {
> > DedicatedResourceVO dedicatedCluster =
> > _dedicatedDao.findByClusterId(cluster.getId());
> > if (dedicatedCluster != null) {
> > avoids.addCluster(dedicatedCluster.getClusterId());
> > if (s_logger.isDebugEnabled()) {
> > s_logger.debug("Cannot use this dedicated
> > Cluster " +
> > cluster.getName() + ".");
> > }
> > }
> > }
> > List<HostVO> hostInDc = _hostDao.listByDataCenterId(dc.getId());
> > for (HostVO host : hostInDc) {
> > DedicatedResourceVO dedicatedHost =
> > _dedicatedDao.findByHostId(host.getId());
> > if (dedicatedHost != null) {
> > avoids.addHost(dedicatedHost.getHostId());
> > if (s_logger.isDebugEnabled()) {
> > s_logger.debug("Cannot use this dedicated host
> > " +
> > host.getName() + ".");
> > }
> > }
> > }
> >
> > > [Performance Testing] Time taken for Deploy VM async job to complete
> > > is considerably higher
> > > --------------------------------------------------------------------
> > > --
> > > ---------------------
> > >
> > > Key: CLOUDSTACK-4179
> > > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4179
> > > Project: CloudStack
> > > Issue Type: Bug
> > > Security Level: Public(Anyone can view this level - this is the default.)
> > > Components: Management Server
> > > Affects Versions: 4.2.0
> > > Environment: Simulated VMs and hosts, advanced zone, RVR
> > > Reporter: Sowmya Krishnan
> > > Assignee: Koushik Das
> > > Priority: Critical
> > > Fix For: 4.2.0
> > >
> > > Attachments: async_job_time.out, job-109.log
> > >
> > >
> > > Following setup is used to create performance test bed:
> > > Advanced zone, 2 Management Servers, 124 Pods [Each Pod having 2
> > > Clusters]
> > > 248 Clusters [Each cluster having 8 hosts and one primary storage]
> > > 2000 Hosts
> > > 4000 User accounts [Each account having one network]
> > > ~8000 Virtual Routers
> > > Following config parameters were used in both the management servers
> > > - Java heap size = 5 GB
> > > - db.cloud.maxActive = 250
> > > -
> > >
> >
> db.cloud.url.params=prepStmtCacheSize=517&cachePrepStmts=true&prepS
> > tmt
> > >
> >
> CacheSqlLimit=4096&includeInnodbStatusInDeadlockExceptions=true&logSlo
> > > wQueries=true Tried deploying around 4000 Simulator VMs across 4K
> > > accounts. Noticing considerably higher time for deploy VM async job
> > > to
> > complete as compared to baseline results.
> > > Some of the Baseline results are documented here:
> > >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Baseline+Report
> > > s+for+Pre-4.x+Performance+Runs On an average, it is now taking 70-80
> > > seconds for deployVM to complete as compared to 10-15 seconds earlier.
> > > From the logs, it appears in some cases, DeploymentPlanningManager
> > takes time to kick in.:
> > > 2013-08-07 01:52:09,785 DEBUG [cloud.network.NetworkModelImpl]
> > > (Job-Executor-105:job-109 = [ 19ccfecb-c815-497a-9ff1-70d590025af3
> > > ]) Service SecurityGroup is not supported in the network id=304
> > > 2013-08-07 01:52:37,288 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-105:job-
> > 109 = [ 19ccfecb-c815-497a-9ff1-70d590025af3 ]) Deploy avoids pods:
> > null,
> > clusters: null, hosts: null ...
> > > ...
> > > 2013-08-07 01:52:41,199 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> > (Job-Executor-105:job-109 = [ 19ccfecb-c815-497a-9ff1-70d590025af3 ])
> > Deploy avoids pods:
> > > null, clusters: null, hosts: null
> > > 2013-08-07 01:53:19,886 DEBUG
> > > [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-105:job-
> > 109
> > > = [ 19ccfecb-c815-497a-9ff1-70d590025af3 ]) Deploy avoid s pods:
> > > null,
> > clusters: null, hosts: null Both VRs are running at the end of deploy
> > VM and also the VM.
> >
> > --
> > This message is automatically generated by JIRA.
> > If you think it was sent incorrectly, please contact your JIRA
> > administrators For more information on JIRA, see:
> http://www.atlassian.com/software/jira