You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Sowmya Krishnan <so...@citrix.com> on 2013/02/22 19:24:00 UTC
[ACS41][QA] Test plan for List API Performance Optimization
Hi,
I've posted a test plan for tracking the performance numbers for the set of List APIs which were optimized as mentioned in https://issues.apache.org/jira/browse/CLOUDSTACK-527
Test plan is here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Performance+Test+Plan
Please take a look and post comments if any.
Thanks,
Sowmya
Re: [ACS41][QA] Test plan for List API Performance Optimization
Posted by Min Chen <mi...@citrix.com>.
Thanks a lot Sowmya for trying with comparable hardware setup for this
before and after test.
-min
On 3/5/13 10:01 AM, "Sowmya Krishnan" <so...@citrix.com> wrote:
>
>
>> -----Original Message-----
>> From: David Nalley [mailto:david@gnsa.us]
>> Sent: Tuesday, March 05, 2013 9:36 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Re: [ACS41][QA] Test plan for List API Performance Optimization
>>
>> On Tue, Mar 5, 2013 at 7:48 AM, Sowmya Krishnan
>> <so...@citrix.com> wrote:
>> >
>> >> Great - perhaps we can talk with Prasanna about getting those
>> >> completely automated with jenkins.
>> > Sure. I'll get the scripts to some shape and will get in touch with
>> > Prasanna to automate with jenkins
>> >
>> >> Can we get the baseline numbers published somewhere? (as well as the
>> >> numbers you get in your tests of 4.1?
>> > Baselines Published here:
>> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Performance+Tes
>> > t+Baselines
>> > 4.1 Results being published here:
>> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Perfor
>> > mance+Test+Execution
>> > (this is in progress)
>> >
>>
>> Awesome, thanks for getting this up.
>> I do echo Chiradeep's concerns regarding differences in the baseline
>>and current
>> test hardware and configuration.
>>
>
>Sure. I gave an initial try with lesser resources and smaller
>configuration DB but it's not adding up. I've now started the runs with
>the DB configuration same as the one used for the baseline runs. And the
>results so far look good as compared to the baselines. Intermediate
>results are posted here:
>https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Perf+Resul
>ts+Iteration+2
>
>
>> --David
RE: [ACS41][QA] Test plan for List API Performance Optimization
Posted by Sowmya Krishnan <so...@citrix.com>.
> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Tuesday, March 05, 2013 9:36 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [ACS41][QA] Test plan for List API Performance Optimization
>
> On Tue, Mar 5, 2013 at 7:48 AM, Sowmya Krishnan
> <so...@citrix.com> wrote:
> >
> >> Great - perhaps we can talk with Prasanna about getting those
> >> completely automated with jenkins.
> > Sure. I'll get the scripts to some shape and will get in touch with
> > Prasanna to automate with jenkins
> >
> >> Can we get the baseline numbers published somewhere? (as well as the
> >> numbers you get in your tests of 4.1?
> > Baselines Published here:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Performance+Tes
> > t+Baselines
> > 4.1 Results being published here:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Perfor
> > mance+Test+Execution
> > (this is in progress)
> >
>
> Awesome, thanks for getting this up.
> I do echo Chiradeep's concerns regarding differences in the baseline and current
> test hardware and configuration.
>
Sure. I gave an initial try with lesser resources and smaller configuration DB but it's not adding up. I've now started the runs with the DB configuration same as the one used for the baseline runs. And the results so far look good as compared to the baselines. Intermediate results are posted here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Perf+Results+Iteration+2
> --David
Re: [ACS41][QA] Test plan for List API Performance Optimization
Posted by David Nalley <da...@gnsa.us>.
On Tue, Mar 5, 2013 at 7:48 AM, Sowmya Krishnan
<so...@citrix.com> wrote:
>
>> Great - perhaps we can talk with Prasanna about getting those completely
>> automated with jenkins.
> Sure. I'll get the scripts to some shape and will get in touch with Prasanna to automate with jenkins
>
>> Can we get the baseline numbers published somewhere? (as well as the numbers
>> you get in your tests of 4.1?
> Baselines Published here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Performance+Test+Baselines
> 4.1 Results being published here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Performance+Test+Execution
> (this is in progress)
>
Awesome, thanks for getting this up.
I do echo Chiradeep's concerns regarding differences in the baseline
and current test hardware and configuration.
--David
RE: [ACS41][QA] Test plan for List API Performance Optimization
Posted by Sowmya Krishnan <so...@citrix.com>.
> Great - perhaps we can talk with Prasanna about getting those completely
> automated with jenkins.
Sure. I'll get the scripts to some shape and will get in touch with Prasanna to automate with jenkins
> Can we get the baseline numbers published somewhere? (as well as the numbers
> you get in your tests of 4.1?
Baselines Published here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Performance+Test+Baselines
4.1 Results being published here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Performance+Test+Execution
(this is in progress)
Thanks,
Sowmya
Re: [ACS41][QA] Test plan for List API Performance Optimization
Posted by David Nalley <da...@gnsa.us>.
Inline reply.
On Thu, Feb 28, 2013 at 2:09 AM, Sowmya Krishnan
<so...@citrix.com> wrote:
> Thanks for taking time to review the plan David. Answers inline.
>
>> -----Original Message-----
>> From: David Nalley [mailto:david@gnsa.us]
>> Sent: Thursday, February 28, 2013 3:26 AM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Re: [ACS41][QA] Test plan for List API Performance Optimization
>>
>> On Fri, Feb 22, 2013 at 1:24 PM, Sowmya Krishnan
>> <so...@citrix.com> wrote:
>> > Hi,
>> >
>> > I've posted a test plan for tracking the performance numbers for the
>> > set of List APIs which were optimized as mentioned in
>> > https://issues.apache.org/jira/browse/CLOUDSTACK-527
>> > Test plan is here:
>> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Perfor
>> > mance+Test+Plan
>> >
>> > Please take a look and post comments if any.
>> >
>>
>>
>> Thanks for writing this up, I have a couple of questions for you.
>>
>> I understand that you are running these tests and recording performance, but it
>> seems like you are measuring time. Is this time from query leaving the client to
>> answer? Is the client on the management server or not?
>>
> Yes. Time measured from the instant the query is fired from the client till the complete response has been received. Client is not in the management server. I'll fire queries from a different server than MS or the DB.
>
>> I assume you are going to use the simulator, and not just have a populated DB?
>> (If that isn't the case, perhaps you can share the db
>> dump.)
>>
> Plan is to use simulator to create Hosts, Routers, VMs etc... to generate the load on DB and management server rather than populate DB.
> Unless, at a later stage, there's a need to run much higher loads than what I've mentioned in the test plan and if it's beyond what my test servers can sustain... Then I might switch to using a DB dump. But I don't foresee this for now.
>
>> Are you going to take a baseline from 4.0.{0,1}?
>>
> I have some numbers for List APIs pre-API optimization. I'll use those as baseline.
>
>> Can this test be written up as a script and generate these statistics as we get
>> near a release to ensure we don't regress?
>>
>
> Sure. I already have some scripts for generating load. I'll write few more scripts to track time taken for List APIs.
>
Great - perhaps we can talk with Prasanna about getting those
completely automated with jenkins.
>> Are we assuming there will be no slow/long running queries? If there are, it
>> might be interesting to see what those are and if there are database issues we
>> can further work on?
>>
> I usually have the DB log slow queries. I can publish those too as part of the results.
>
>> What is 'failure' of this test? (slower than 4.0.x?, slower than n-percent-faster
>> than 4.0.x?)
>>
> I have some numbers recorded for few List APIs before the API re-factoring was done. I'll take those as baseline and call out failures based on that for a start. Going forward, I'll try to automate the regressions so that we catch issues due to regressions if any.
Can we get the baseline numbers published somewhere? (as well as the
numbers you get in your tests of 4.1?
--David
RE: [ACS41][QA] Test plan for List API Performance Optimization
Posted by Sowmya Krishnan <so...@citrix.com>.
Thanks for taking time to review the plan David. Answers inline.
> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Thursday, February 28, 2013 3:26 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: [ACS41][QA] Test plan for List API Performance Optimization
>
> On Fri, Feb 22, 2013 at 1:24 PM, Sowmya Krishnan
> <so...@citrix.com> wrote:
> > Hi,
> >
> > I've posted a test plan for tracking the performance numbers for the
> > set of List APIs which were optimized as mentioned in
> > https://issues.apache.org/jira/browse/CLOUDSTACK-527
> > Test plan is here:
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Perfor
> > mance+Test+Plan
> >
> > Please take a look and post comments if any.
> >
>
>
> Thanks for writing this up, I have a couple of questions for you.
>
> I understand that you are running these tests and recording performance, but it
> seems like you are measuring time. Is this time from query leaving the client to
> answer? Is the client on the management server or not?
>
Yes. Time measured from the instant the query is fired from the client till the complete response has been received. Client is not in the management server. I'll fire queries from a different server than MS or the DB.
> I assume you are going to use the simulator, and not just have a populated DB?
> (If that isn't the case, perhaps you can share the db
> dump.)
>
Plan is to use simulator to create Hosts, Routers, VMs etc... to generate the load on DB and management server rather than populate DB.
Unless, at a later stage, there's a need to run much higher loads than what I've mentioned in the test plan and if it's beyond what my test servers can sustain... Then I might switch to using a DB dump. But I don't foresee this for now.
> Are you going to take a baseline from 4.0.{0,1}?
>
I have some numbers for List APIs pre-API optimization. I'll use those as baseline.
> Can this test be written up as a script and generate these statistics as we get
> near a release to ensure we don't regress?
>
Sure. I already have some scripts for generating load. I'll write few more scripts to track time taken for List APIs.
> Are we assuming there will be no slow/long running queries? If there are, it
> might be interesting to see what those are and if there are database issues we
> can further work on?
>
I usually have the DB log slow queries. I can publish those too as part of the results.
> What is 'failure' of this test? (slower than 4.0.x?, slower than n-percent-faster
> than 4.0.x?)
>
I have some numbers recorded for few List APIs before the API re-factoring was done. I'll take those as baseline and call out failures based on that for a start. Going forward, I'll try to automate the regressions so that we catch issues due to regressions if any.
> --David
Re: [ACS41][QA] Test plan for List API Performance Optimization
Posted by David Nalley <da...@gnsa.us>.
On Fri, Feb 22, 2013 at 1:24 PM, Sowmya Krishnan
<so...@citrix.com> wrote:
> Hi,
>
> I've posted a test plan for tracking the performance numbers for the set of List APIs which were optimized as mentioned in https://issues.apache.org/jira/browse/CLOUDSTACK-527
> Test plan is here: https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Performance+Test+Plan
>
> Please take a look and post comments if any.
>
Thanks for writing this up, I have a couple of questions for you.
I understand that you are running these tests and recording
performance, but it seems like you are measuring time. Is this time
from query leaving the client to answer? Is the client on the
management server or not?
I assume you are going to use the simulator, and not just have a
populated DB? (If that isn't the case, perhaps you can share the db
dump.)
Are you going to take a baseline from 4.0.{0,1}?
Can this test be written up as a script and generate these statistics
as we get near a release to ensure we don't regress?
Are we assuming there will be no slow/long running queries? If there
are, it might be interesting to see what those are and if there are
database issues we can further work on?
What is 'failure' of this test? (slower than 4.0.x?, slower than
n-percent-faster than 4.0.x?)
--David
RE: [ACS41][QA] Test plan for List API Performance Optimization
Posted by Sowmya Krishnan <so...@citrix.com>.
Thanks for the helpful pointers Chiradeep! I'll keep these in mind during the further performance runs.
On the innodb_buffer_pool_size, I usually set it to 80% of the RAM size for all my runs.
Thanks for the pointers on the tools as well - I'll try out these for my next performance run and will include details in the test plan too once I start using.
> -----Original Message-----
> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
> Sent: Tuesday, March 05, 2013 4:09 AM
> To: cloudstack-users@incubator.apache.org; cloudstack-
> dev@incubator.apache.org
> Subject: Re: [ACS41][QA] Test plan for List API Performance Optimization
>
>
>
> On 2/22/13 10:24 AM, "Sowmya Krishnan" <so...@citrix.com>
> wrote:
>
> >Hi,
> >
> >I've posted a test plan for tracking the performance numbers for the
> >set of List APIs which were optimized as mentioned in
> >https://issues.apache.org/jira/browse/CLOUDSTACK-527
> >Test plan is here:
> >https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Perform
> >anc
> >e+Test+Plan
> >
> >Please take a look and post comments if any.
> >
> >Thanks,
> >Sowmya
>
> Thanks for this. Performance involves a number of different factors and
> changing any one of them can result in dramatically different results.
> So
>
> Client -> client http library -> client OS -> client cpu -> client nic
> -> network switch -> server nic -> server cpu -> server OS -> jvm ->
> -> tomcat cloudstack -> server OS -> nic -> server nic -> network switch
> -> DB nic -> DB CPU -> DB OS -> mysql -> DB OS -> DB hard drive
>
> So, a lot of variables, some more pertinent than others.
>
> One tip is to make sure that MySQL runs entirely from memory, so if you set
> innodb_buffer_pool_size [1] to the maximum possible, queries will likely run
> entirely from RAM and isolate any hard drive dependency.
>
> Another tip is to make sure that there is no other traffic on the network.
>
> Another tip is to instrument the server with a tool like AppDynamics[2] or
> NewRelic [3]. This will help us isolate the bottlenecks within the application code
> and give us useful statistics.
>
> [1]<http://dev.mysql.com/doc/refman/5.1/en/innodb-
> parameters.html#sysvar_in
> nodb_buffer_pool_size>
> [2] http://info.appdynamics.com/AppDynamicsLiteJavaNet.html
> [3] http://newrelic.com/
RE: [ACS41][QA] Test plan for List API Performance Optimization
Posted by Sowmya Krishnan <so...@citrix.com>.
Thanks for the helpful pointers Chiradeep! I'll keep these in mind during the further performance runs.
On the innodb_buffer_pool_size, I usually set it to 80% of the RAM size for all my runs.
Thanks for the pointers on the tools as well - I'll try out these for my next performance run and will include details in the test plan too once I start using.
> -----Original Message-----
> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
> Sent: Tuesday, March 05, 2013 4:09 AM
> To: cloudstack-users@incubator.apache.org; cloudstack-
> dev@incubator.apache.org
> Subject: Re: [ACS41][QA] Test plan for List API Performance Optimization
>
>
>
> On 2/22/13 10:24 AM, "Sowmya Krishnan" <so...@citrix.com>
> wrote:
>
> >Hi,
> >
> >I've posted a test plan for tracking the performance numbers for the
> >set of List APIs which were optimized as mentioned in
> >https://issues.apache.org/jira/browse/CLOUDSTACK-527
> >Test plan is here:
> >https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Perform
> >anc
> >e+Test+Plan
> >
> >Please take a look and post comments if any.
> >
> >Thanks,
> >Sowmya
>
> Thanks for this. Performance involves a number of different factors and
> changing any one of them can result in dramatically different results.
> So
>
> Client -> client http library -> client OS -> client cpu -> client nic
> -> network switch -> server nic -> server cpu -> server OS -> jvm ->
> -> tomcat cloudstack -> server OS -> nic -> server nic -> network switch
> -> DB nic -> DB CPU -> DB OS -> mysql -> DB OS -> DB hard drive
>
> So, a lot of variables, some more pertinent than others.
>
> One tip is to make sure that MySQL runs entirely from memory, so if you set
> innodb_buffer_pool_size [1] to the maximum possible, queries will likely run
> entirely from RAM and isolate any hard drive dependency.
>
> Another tip is to make sure that there is no other traffic on the network.
>
> Another tip is to instrument the server with a tool like AppDynamics[2] or
> NewRelic [3]. This will help us isolate the bottlenecks within the application code
> and give us useful statistics.
>
> [1]<http://dev.mysql.com/doc/refman/5.1/en/innodb-
> parameters.html#sysvar_in
> nodb_buffer_pool_size>
> [2] http://info.appdynamics.com/AppDynamicsLiteJavaNet.html
> [3] http://newrelic.com/
Re: [ACS41][QA] Test plan for List API Performance Optimization
Posted by Chiradeep Vittal <Ch...@citrix.com>.
On 2/22/13 10:24 AM, "Sowmya Krishnan" <so...@citrix.com> wrote:
>Hi,
>
>I've posted a test plan for tracking the performance numbers for the set
>of List APIs which were optimized as mentioned in
>https://issues.apache.org/jira/browse/CLOUDSTACK-527
>Test plan is here:
>https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Performanc
>e+Test+Plan
>
>Please take a look and post comments if any.
>
>Thanks,
>Sowmya
Thanks for this. Performance involves a number of different factors and
changing any one of them can result in dramatically different results.
So
Client -> client http library -> client OS -> client cpu -> client nic
-> network switch -> server nic -> server cpu -> server OS -> jvm -> tomcat
-> cloudstack -> server OS -> nic -> server nic -> network switch
-> DB nic -> DB CPU -> DB OS -> mysql -> DB OS -> DB hard drive
So, a lot of variables, some more pertinent than others.
One tip is to make sure that MySQL runs entirely from memory, so
if you set innodb_buffer_pool_size [1] to the maximum possible,
queries will likely run entirely from RAM and isolate any hard drive
dependency.
Another tip is to make sure that there is no other traffic on the network.
Another tip is to instrument the server with a tool like AppDynamics[2] or
NewRelic [3]. This will help us isolate the bottlenecks within the
application code and give us useful statistics.
[1]<http://dev.mysql.com/doc/refman/5.1/en/innodb-parameters.html#sysvar_in
nodb_buffer_pool_size>
[2] http://info.appdynamics.com/AppDynamicsLiteJavaNet.html
[3] http://newrelic.com/
Re: [ACS41][QA] Test plan for List API Performance Optimization
Posted by Chiradeep Vittal <Ch...@citrix.com>.
On 2/22/13 10:24 AM, "Sowmya Krishnan" <so...@citrix.com> wrote:
>Hi,
>
>I've posted a test plan for tracking the performance numbers for the set
>of List APIs which were optimized as mentioned in
>https://issues.apache.org/jira/browse/CLOUDSTACK-527
>Test plan is here:
>https://cwiki.apache.org/confluence/display/CLOUDSTACK/List+API+Performanc
>e+Test+Plan
>
>Please take a look and post comments if any.
>
>Thanks,
>Sowmya
Thanks for this. Performance involves a number of different factors and
changing any one of them can result in dramatically different results.
So
Client -> client http library -> client OS -> client cpu -> client nic
-> network switch -> server nic -> server cpu -> server OS -> jvm -> tomcat
-> cloudstack -> server OS -> nic -> server nic -> network switch
-> DB nic -> DB CPU -> DB OS -> mysql -> DB OS -> DB hard drive
So, a lot of variables, some more pertinent than others.
One tip is to make sure that MySQL runs entirely from memory, so
if you set innodb_buffer_pool_size [1] to the maximum possible,
queries will likely run entirely from RAM and isolate any hard drive
dependency.
Another tip is to make sure that there is no other traffic on the network.
Another tip is to instrument the server with a tool like AppDynamics[2] or
NewRelic [3]. This will help us isolate the bottlenecks within the
application code and give us useful statistics.
[1]<http://dev.mysql.com/doc/refman/5.1/en/innodb-parameters.html#sysvar_in
nodb_buffer_pool_size>
[2] http://info.appdynamics.com/AppDynamicsLiteJavaNet.html
[3] http://newrelic.com/