You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by nmq <nm...@gmail.com> on 2013/07/01 21:33:14 UTC
Establishing baseline metrics
Hi all
This is not a JMeter specific questions but since this user list comprises
of experts in performance testing, I figured it would be a good place to
ask this question.
My question is how do you establish baselines for a website's performance
if you do not have any historic data? Lets say this is a new website and
its for a limited number of customers.
How do you determine what should be the number of concurrent users you
should simulate.
Lets say the executives say off at the top of their heads, that the maximum
number of concurrent users would be 50 at peak times. Does that mean I
should not go beyond 50 or should I still do tests with a higher number?
How can I go about establishing baselines for page load times, if I do not
have any historic data and have no industry benchmarks or competitor data.
Would it make sense to say let's see how the website is doing throughout
the development phase and establish our baseline using the current response
times?
I would appreciate any input.
Regards
Sam
Re: Establishing baseline metrics
Posted by Sergio Boso <se...@bosoconsulting.it>.
Il 01/07/2013 22:32, Robin D. Wilson ha scritto:
> I'm thinking I look at performance testing differently than a lot of people... For me, the objective of performance testing is to...
.. me too! I have an entirely different point of view about base line.
I agree with Robin, user acceptance is hard to define: When I ask, in general, about user response, customers tend to give generica
answer, like "all page must be loaded in 3 secs....etc".
IN reality, user acceptance has very much to do with the value of the thing users are doing: in one case, they were doing document
uploading and retrieval,
and they found perfectly normal to wait for more than 20 s to get the documents they were looking for.
Previously, it took more than one hour to get the paper version in a dusty archive, so 20 s was a perfect time.
So, I'm doing a baseline to get the "best possible behaviour" for the script: usually it is run with a single user, with relaxed
think time, when very little load (or no load at all) is running on the system.
Often I take many samples, (i.e. many loops), in order to understand variability.
This way, you can go back to the management and say: "hey, your software is running with this response time and this operation is
done in X ms. Are your user OK with that?"
If the answer is OK, you can start the load test looking at how response time grows with load...
Surely, you also have to make calculation about the _*target load *_you want to bear. According to my experience, the customer has
no interest into knowing
the maximum load for the system: they want to know if the system can bear the business they are running.
SO, I start with increase the load (i.e. the thread, i.,e. the virtual users), and look at the way response time increase
> establish what your system _can_ do, not what you need to accomplish. So when you are setting up your tests, you are trying to drive
> your systems at maximum capacity for some extended period of time. Then you measure that capacity as your 'baseline'.
>
> For every subsequent release of your code, you measure it against the 'baseline', and determine whether the code got faster or
> slower. If you determine that the slower (or faster) response is acceptable to your end users (because you were nowhere near the
> user's acceptable standard), you can reset your baseline to that standard. If your slower standard is encroaching on the usability
> of the system - you can declare that baseline as the minimum spec, and then fail any code that exceeds that standard.
>
> As for how you determine what is acceptable to a 'user', that can be handled in a number of ways - without actually improving the
> 'real' performance of the system. Consider a web page that loads a bunch of rows of data in a big table. For most users, if you can
> start reading the table within 1-2 seconds, that is acceptable for a system's performance. But if there are hundreds of rows of
> data, you would not need to load _all_ the rows within 1-2 seconds to actually meet their performance criteria. You only need to
> load enough rows that the table fills the browser - so they can start reading - within the 1-2 second period. JMeter cannot really
> measure this timing, it can only measure the 'overall response time' (indeed, I don't know any testing tool that can do it). So
> trying to define a performance benchmark in terms of what 'users' experience is really difficult, and nearly useless (to me anyway).
>
> I look at performance testing as a way to cross-check my development team against the perpetual tendency to gum-up the code and slow
> things down. So in order to make the testing effective for the developers, I need to perf test _very_specific_ things. Trying to
> performance test the "system" as a whole is nearly an impossible task - not only because there are so many variables that influence
> the tests, but precisely because "all of those variables" make it impossible to debug which one causes the bottleneck when there is
> a change in performance from one release to the next. (Have you ever sent your programmers off to 'fix' a performance problem that
> turned out to be caused by an O/S update on your server? I have...)
>
> Instead, we create performance tests that test specific functional systems. That is, the "login" perf test. The "registration" perf
> test. The "..." perf test. Each one of these tests is run independently, so that when we encounter a slower benchmark - we can tell
> the developers immediately where to concentrate their efforts in fixing the problem. (We also monitor all parts of the system (CPU,
> IO, Database Transactions (reads, writes, full table scans, etc.) from all servers involved in the test. The goal is not to simulate
> 'real user activity', it is to max out the capacity of at least 1 of the servers in the test (specifically the one executing the
> 'application logic'). If we max out that one server, we know that our 'benchmark' is the most we can expect of a single member of
> our cluster of machines. (We also test a cluster of 2 machines - and measure the fall-off in capacity between a 1-member cluster and
> 2-member cluster, this gives us an idea of how much impact our 'clustering' system has on performance as well.) I suppose you could
> say that I look at it as if, we measure the 'maximum capacity', and so long as the number of users doesn't exceed that - we will
> perform OK.
>
> We do run some 'all-encompassing' system tests as well, but those are more for 'stress' testing than for performance benchmarking.
> We are specifically looking for things that start to break-down after hours of continuous operation at peak capacity. So we monitor
> error logs and look to make sure that we aren't throwing errors while under stress.
>
> The number one thing to keep in mind about performance testing is that you have to use 'real data'. We actually download our
> production database every weekend, and strip out any 'personal information' (stuff that we protect in our production environment) by
> either nulling it out, or replacing it with bogus data. This allows us to run our performance tests against a database that has 100s
> of millions of rows of data. Nearly all of our performance 'bugs' have been caused by poor data handling in the code (SQL requests
> that don't use indices (causing a full table scan), badly formed joins, fetching a few rows of data and then looping through them in
> the code (when the 'few rows of data' from your 'dev' environment become 100,000 rows with the production data, this tends to bog
> the code down a lot), etc.). So if you are testing with 'faked' data, odds are good you will miss a lot of performance issues - no
> matter what form of performance testing you use.
>
> I will say that we have served over 130M web pages in 1 month, using only 5 servers (4 tomcats, and 1 DB server)... Those pages
> represented about 10X in "GET" requests to our servers...
>
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> http://www.kingsisle.com
>
>
--
Ing. Sergio Boso
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Establishing baseline metrics
Posted by nmq <nm...@gmail.com>.
Hi folks
Interesting approaches. Thank you for the information, very useful for me.
Regards
Sam
On Mon, Jul 1, 2013 at 11:45 PM, Deepak Goel <de...@gmail.com> wrote:
> For baseline metrics, do a stress test till the system crashes, and
> then the 'Number of Users' supported can be the baseline metrics for
> you along with other data (processor and memory usage)
>
> On 7/2/13, Deepak Shetty <sh...@gmail.com> wrote:
> > Hi
> > We seem to have differing philosophies :)
> >>For me, the objective of performance testing is to establish what your
> > system _can_ do, not what you need to accomplish.
> > One(but not all) of the goals of performance testing is indeed whether or
> > not your system does what you set out to accomplish. If its the first
> time
> > you are releasing something (As the poster implies) you will have a set
> of
> > goals you need to accomplish , and if your perf tests indicate that those
> > goals are not met then you do have to make some change (infrastructure or
> > code or whatever) and rerun your tests till you meet those goals. You
> also
> > have to know when to stop (you stop the optimisations when the goals are
> > met).
> >
> >>So trying to define a performance benchmark in terms of what 'users'
> > experience is really difficult, and nearly useless (to me anyway).
> > You are missing the point here. If you think that X seconds is an
> > acceptable experience but a majority of users report your website as
> "slow"
> > are you going to dismiss their experience as nearly useless? You have to
> > have some way of validating your baseline is good enough for your users.
> >
> >
> >>Trying to
> >>performance test the "system" as a whole is nearly an impossible task -
> > not only because there are so many variables that influence
> >>the tests, but precisely because "all of those variables" make it
> > impossible to debug which one causes the bottleneck when there is
> >>a change in performance from one release to the next.
> > You are mixing the detection part (the load test) with the analytic or
> > profiling part. Once you know there is a problem , then you can analyse
> it
> > in a way that works for you. But it is absolutely necessary to test the
> > system as a whole precisely because of those variables.
> >
> >
> >
> > On Mon, Jul 1, 2013 at 1:32 PM, Robin D. Wilson <rw...@gmail.com>
> wrote:
> >
> >> I'm thinking I look at performance testing differently than a lot of
> >> people... For me, the objective of performance testing is to
> >> establish what your system _can_ do, not what you need to accomplish. So
> >> when you are setting up your tests, you are trying to drive
> >> your systems at maximum capacity for some extended period of time. Then
> >> you measure that capacity as your 'baseline'.
> >>
> >> For every subsequent release of your code, you measure it against the
> >> 'baseline', and determine whether the code got faster or
> >> slower. If you determine that the slower (or faster) response is
> >> acceptable to your end users (because you were nowhere near the
> >> user's acceptable standard), you can reset your baseline to that
> >> standard.
> >> If your slower standard is encroaching on the usability
> >> of the system - you can declare that baseline as the minimum spec, and
> >> then fail any code that exceeds that standard.
> >>
> >> As for how you determine what is acceptable to a 'user', that can be
> >> handled in a number of ways - without actually improving the
> >> 'real' performance of the system. Consider a web page that loads a bunch
> >> of rows of data in a big table. For most users, if you can
> >> start reading the table within 1-2 seconds, that is acceptable for a
> >> system's performance. But if there are hundreds of rows of
> >> data, you would not need to load _all_ the rows within 1-2 seconds to
> >> actually meet their performance criteria. You only need to
> >> load enough rows that the table fills the browser - so they can start
> >> reading - within the 1-2 second period. JMeter cannot really
> >> measure this timing, it can only measure the 'overall response time'
> >> (indeed, I don't know any testing tool that can do it). So
> >> trying to define a performance benchmark in terms of what 'users'
> >> experience is really difficult, and nearly useless (to me anyway).
> >>
> >> I look at performance testing as a way to cross-check my development
> team
> >> against the perpetual tendency to gum-up the code and slow
> >> things down. So in order to make the testing effective for the
> >> developers,
> >> I need to perf test _very_specific_ things. Trying to
> >> performance test the "system" as a whole is nearly an impossible task -
> >> not only because there are so many variables that influence
> >> the tests, but precisely because "all of those variables" make it
> >> impossible to debug which one causes the bottleneck when there is
> >> a change in performance from one release to the next. (Have you ever
> sent
> >> your programmers off to 'fix' a performance problem that
> >> turned out to be caused by an O/S update on your server? I have...)
> >>
> >> Instead, we create performance tests that test specific functional
> >> systems. That is, the "login" perf test. The "registration" perf
> >> test. The "..." perf test. Each one of these tests is run independently,
> >> so that when we encounter a slower benchmark - we can tell
> >> the developers immediately where to concentrate their efforts in fixing
> >> the problem. (We also monitor all parts of the system (CPU,
> >> IO, Database Transactions (reads, writes, full table scans, etc.) from
> >> all
> >> servers involved in the test. The goal is not to simulate
> >> 'real user activity', it is to max out the capacity of at least 1 of the
> >> servers in the test (specifically the one executing the
> >> 'application logic'). If we max out that one server, we know that our
> >> 'benchmark' is the most we can expect of a single member of
> >> our cluster of machines. (We also test a cluster of 2 machines - and
> >> measure the fall-off in capacity between a 1-member cluster and
> >> 2-member cluster, this gives us an idea of how much impact our
> >> 'clustering' system has on performance as well.) I suppose you could
> >> say that I look at it as if, we measure the 'maximum capacity', and so
> >> long as the number of users doesn't exceed that - we will
> >> perform OK.
> >>
> >> We do run some 'all-encompassing' system tests as well, but those are
> >> more
> >> for 'stress' testing than for performance benchmarking.
> >> We are specifically looking for things that start to break-down after
> >> hours of continuous operation at peak capacity. So we monitor
> >> error logs and look to make sure that we aren't throwing errors while
> >> under stress.
> >>
> >> The number one thing to keep in mind about performance testing is that
> >> you
> >> have to use 'real data'. We actually download our
> >> production database every weekend, and strip out any 'personal
> >> information' (stuff that we protect in our production environment) by
> >> either nulling it out, or replacing it with bogus data. This allows us
> to
> >> run our performance tests against a database that has 100s
> >> of millions of rows of data. Nearly all of our performance 'bugs' have
> >> been caused by poor data handling in the code (SQL requests
> >> that don't use indices (causing a full table scan), badly formed joins,
> >> fetching a few rows of data and then looping through them in
> >> the code (when the 'few rows of data' from your 'dev' environment become
> >> 100,000 rows with the production data, this tends to bog
> >> the code down a lot), etc.). So if you are testing with 'faked' data,
> >> odds
> >> are good you will miss a lot of performance issues - no
> >> matter what form of performance testing you use.
> >>
> >> I will say that we have served over 130M web pages in 1 month, using
> only
> >> 5 servers (4 tomcats, and 1 DB server)... Those pages
> >> represented about 10X in "GET" requests to our servers...
> >>
> >> --
> >> Robin D. Wilson
> >> Sr. Director of Web Development
> >> KingsIsle Entertainment, Inc.
> >> http://www.kingsisle.com
> >>
> >>
> >> -----Original Message-----
> >> From: nmq [mailto:nmq0607@gmail.com]
> >> Sent: Monday, July 01, 2013 2:33 PM
> >> To: JMeter Users List
> >> Subject: Establishing baseline metrics
> >>
> >> Hi all
> >>
> >> This is not a JMeter specific questions but since this user list
> >> comprises
> >> of experts in performance testing, I figured it would be
> >> a good place to ask this question.
> >>
> >> My question is how do you establish baselines for a website's
> performance
> >> if you do not have any historic data? Lets say this is a
> >> new website and its for a limited number of customers.
> >>
> >> How do you determine what should be the number of concurrent users you
> >> should simulate.
> >>
> >> Lets say the executives say off at the top of their heads, that the
> >> maximum number of concurrent users would be 50 at peak times.
> >> Does that mean I should not go beyond 50 or should I still do tests with
> >> a
> >> higher number?
> >>
> >> How can I go about establishing baselines for page load times, if I do
> >> not
> >> have any historic data and have no industry benchmarks or
> >> competitor data.
> >>
> >> Would it make sense to say let's see how the website is doing throughout
> >> the development phase and establish our baseline using the
> >> current response times?
> >>
> >> I would appreciate any input.
> >>
> >>
> >> Regards
> >> Sam
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >> For additional commands, e-mail: user-help@jmeter.apache.org
> >>
> >>
> >
>
>
> --
> Namaskara~Nalama~Guten Tag~Bonjour
>
>
> --
> Keigu
>
> Deepak
> 7350012833
> deicool@gmail.com
> http://www.simtree.net
>
> Skype: thumsupdeicool
> Google talk: deicool
> Blog: http://loveandfearless.wordpress.com
> Facebook: http://www.facebook.com/deicool
>
> "Contribute to the world, environment and more :
> http://www.gridrepublic.org
> "
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
Re: Establishing baseline metrics
Posted by Deepak Goel <de...@gmail.com>.
For baseline metrics, do a stress test till the system crashes, and
then the 'Number of Users' supported can be the baseline metrics for
you along with other data (processor and memory usage)
On 7/2/13, Deepak Shetty <sh...@gmail.com> wrote:
> Hi
> We seem to have differing philosophies :)
>>For me, the objective of performance testing is to establish what your
> system _can_ do, not what you need to accomplish.
> One(but not all) of the goals of performance testing is indeed whether or
> not your system does what you set out to accomplish. If its the first time
> you are releasing something (As the poster implies) you will have a set of
> goals you need to accomplish , and if your perf tests indicate that those
> goals are not met then you do have to make some change (infrastructure or
> code or whatever) and rerun your tests till you meet those goals. You also
> have to know when to stop (you stop the optimisations when the goals are
> met).
>
>>So trying to define a performance benchmark in terms of what 'users'
> experience is really difficult, and nearly useless (to me anyway).
> You are missing the point here. If you think that X seconds is an
> acceptable experience but a majority of users report your website as "slow"
> are you going to dismiss their experience as nearly useless? You have to
> have some way of validating your baseline is good enough for your users.
>
>
>>Trying to
>>performance test the "system" as a whole is nearly an impossible task -
> not only because there are so many variables that influence
>>the tests, but precisely because "all of those variables" make it
> impossible to debug which one causes the bottleneck when there is
>>a change in performance from one release to the next.
> You are mixing the detection part (the load test) with the analytic or
> profiling part. Once you know there is a problem , then you can analyse it
> in a way that works for you. But it is absolutely necessary to test the
> system as a whole precisely because of those variables.
>
>
>
> On Mon, Jul 1, 2013 at 1:32 PM, Robin D. Wilson <rw...@gmail.com> wrote:
>
>> I'm thinking I look at performance testing differently than a lot of
>> people... For me, the objective of performance testing is to
>> establish what your system _can_ do, not what you need to accomplish. So
>> when you are setting up your tests, you are trying to drive
>> your systems at maximum capacity for some extended period of time. Then
>> you measure that capacity as your 'baseline'.
>>
>> For every subsequent release of your code, you measure it against the
>> 'baseline', and determine whether the code got faster or
>> slower. If you determine that the slower (or faster) response is
>> acceptable to your end users (because you were nowhere near the
>> user's acceptable standard), you can reset your baseline to that
>> standard.
>> If your slower standard is encroaching on the usability
>> of the system - you can declare that baseline as the minimum spec, and
>> then fail any code that exceeds that standard.
>>
>> As for how you determine what is acceptable to a 'user', that can be
>> handled in a number of ways - without actually improving the
>> 'real' performance of the system. Consider a web page that loads a bunch
>> of rows of data in a big table. For most users, if you can
>> start reading the table within 1-2 seconds, that is acceptable for a
>> system's performance. But if there are hundreds of rows of
>> data, you would not need to load _all_ the rows within 1-2 seconds to
>> actually meet their performance criteria. You only need to
>> load enough rows that the table fills the browser - so they can start
>> reading - within the 1-2 second period. JMeter cannot really
>> measure this timing, it can only measure the 'overall response time'
>> (indeed, I don't know any testing tool that can do it). So
>> trying to define a performance benchmark in terms of what 'users'
>> experience is really difficult, and nearly useless (to me anyway).
>>
>> I look at performance testing as a way to cross-check my development team
>> against the perpetual tendency to gum-up the code and slow
>> things down. So in order to make the testing effective for the
>> developers,
>> I need to perf test _very_specific_ things. Trying to
>> performance test the "system" as a whole is nearly an impossible task -
>> not only because there are so many variables that influence
>> the tests, but precisely because "all of those variables" make it
>> impossible to debug which one causes the bottleneck when there is
>> a change in performance from one release to the next. (Have you ever sent
>> your programmers off to 'fix' a performance problem that
>> turned out to be caused by an O/S update on your server? I have...)
>>
>> Instead, we create performance tests that test specific functional
>> systems. That is, the "login" perf test. The "registration" perf
>> test. The "..." perf test. Each one of these tests is run independently,
>> so that when we encounter a slower benchmark - we can tell
>> the developers immediately where to concentrate their efforts in fixing
>> the problem. (We also monitor all parts of the system (CPU,
>> IO, Database Transactions (reads, writes, full table scans, etc.) from
>> all
>> servers involved in the test. The goal is not to simulate
>> 'real user activity', it is to max out the capacity of at least 1 of the
>> servers in the test (specifically the one executing the
>> 'application logic'). If we max out that one server, we know that our
>> 'benchmark' is the most we can expect of a single member of
>> our cluster of machines. (We also test a cluster of 2 machines - and
>> measure the fall-off in capacity between a 1-member cluster and
>> 2-member cluster, this gives us an idea of how much impact our
>> 'clustering' system has on performance as well.) I suppose you could
>> say that I look at it as if, we measure the 'maximum capacity', and so
>> long as the number of users doesn't exceed that - we will
>> perform OK.
>>
>> We do run some 'all-encompassing' system tests as well, but those are
>> more
>> for 'stress' testing than for performance benchmarking.
>> We are specifically looking for things that start to break-down after
>> hours of continuous operation at peak capacity. So we monitor
>> error logs and look to make sure that we aren't throwing errors while
>> under stress.
>>
>> The number one thing to keep in mind about performance testing is that
>> you
>> have to use 'real data'. We actually download our
>> production database every weekend, and strip out any 'personal
>> information' (stuff that we protect in our production environment) by
>> either nulling it out, or replacing it with bogus data. This allows us to
>> run our performance tests against a database that has 100s
>> of millions of rows of data. Nearly all of our performance 'bugs' have
>> been caused by poor data handling in the code (SQL requests
>> that don't use indices (causing a full table scan), badly formed joins,
>> fetching a few rows of data and then looping through them in
>> the code (when the 'few rows of data' from your 'dev' environment become
>> 100,000 rows with the production data, this tends to bog
>> the code down a lot), etc.). So if you are testing with 'faked' data,
>> odds
>> are good you will miss a lot of performance issues - no
>> matter what form of performance testing you use.
>>
>> I will say that we have served over 130M web pages in 1 month, using only
>> 5 servers (4 tomcats, and 1 DB server)... Those pages
>> represented about 10X in "GET" requests to our servers...
>>
>> --
>> Robin D. Wilson
>> Sr. Director of Web Development
>> KingsIsle Entertainment, Inc.
>> http://www.kingsisle.com
>>
>>
>> -----Original Message-----
>> From: nmq [mailto:nmq0607@gmail.com]
>> Sent: Monday, July 01, 2013 2:33 PM
>> To: JMeter Users List
>> Subject: Establishing baseline metrics
>>
>> Hi all
>>
>> This is not a JMeter specific questions but since this user list
>> comprises
>> of experts in performance testing, I figured it would be
>> a good place to ask this question.
>>
>> My question is how do you establish baselines for a website's performance
>> if you do not have any historic data? Lets say this is a
>> new website and its for a limited number of customers.
>>
>> How do you determine what should be the number of concurrent users you
>> should simulate.
>>
>> Lets say the executives say off at the top of their heads, that the
>> maximum number of concurrent users would be 50 at peak times.
>> Does that mean I should not go beyond 50 or should I still do tests with
>> a
>> higher number?
>>
>> How can I go about establishing baselines for page load times, if I do
>> not
>> have any historic data and have no industry benchmarks or
>> competitor data.
>>
>> Would it make sense to say let's see how the website is doing throughout
>> the development phase and establish our baseline using the
>> current response times?
>>
>> I would appreciate any input.
>>
>>
>> Regards
>> Sam
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>
>
--
Namaskara~Nalama~Guten Tag~Bonjour
--
Keigu
Deepak
7350012833
deicool@gmail.com
http://www.simtree.net
Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool
"Contribute to the world, environment and more : http://www.gridrepublic.org
"
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Establishing baseline metrics
Posted by Deepak Shetty <sh...@gmail.com>.
Hi
We seem to have differing philosophies :)
>For me, the objective of performance testing is to establish what your
system _can_ do, not what you need to accomplish.
One(but not all) of the goals of performance testing is indeed whether or
not your system does what you set out to accomplish. If its the first time
you are releasing something (As the poster implies) you will have a set of
goals you need to accomplish , and if your perf tests indicate that those
goals are not met then you do have to make some change (infrastructure or
code or whatever) and rerun your tests till you meet those goals. You also
have to know when to stop (you stop the optimisations when the goals are
met).
>So trying to define a performance benchmark in terms of what 'users'
experience is really difficult, and nearly useless (to me anyway).
You are missing the point here. If you think that X seconds is an
acceptable experience but a majority of users report your website as "slow"
are you going to dismiss their experience as nearly useless? You have to
have some way of validating your baseline is good enough for your users.
>Trying to
>performance test the "system" as a whole is nearly an impossible task -
not only because there are so many variables that influence
>the tests, but precisely because "all of those variables" make it
impossible to debug which one causes the bottleneck when there is
>a change in performance from one release to the next.
You are mixing the detection part (the load test) with the analytic or
profiling part. Once you know there is a problem , then you can analyse it
in a way that works for you. But it is absolutely necessary to test the
system as a whole precisely because of those variables.
On Mon, Jul 1, 2013 at 1:32 PM, Robin D. Wilson <rw...@gmail.com> wrote:
> I'm thinking I look at performance testing differently than a lot of
> people... For me, the objective of performance testing is to
> establish what your system _can_ do, not what you need to accomplish. So
> when you are setting up your tests, you are trying to drive
> your systems at maximum capacity for some extended period of time. Then
> you measure that capacity as your 'baseline'.
>
> For every subsequent release of your code, you measure it against the
> 'baseline', and determine whether the code got faster or
> slower. If you determine that the slower (or faster) response is
> acceptable to your end users (because you were nowhere near the
> user's acceptable standard), you can reset your baseline to that standard.
> If your slower standard is encroaching on the usability
> of the system - you can declare that baseline as the minimum spec, and
> then fail any code that exceeds that standard.
>
> As for how you determine what is acceptable to a 'user', that can be
> handled in a number of ways - without actually improving the
> 'real' performance of the system. Consider a web page that loads a bunch
> of rows of data in a big table. For most users, if you can
> start reading the table within 1-2 seconds, that is acceptable for a
> system's performance. But if there are hundreds of rows of
> data, you would not need to load _all_ the rows within 1-2 seconds to
> actually meet their performance criteria. You only need to
> load enough rows that the table fills the browser - so they can start
> reading - within the 1-2 second period. JMeter cannot really
> measure this timing, it can only measure the 'overall response time'
> (indeed, I don't know any testing tool that can do it). So
> trying to define a performance benchmark in terms of what 'users'
> experience is really difficult, and nearly useless (to me anyway).
>
> I look at performance testing as a way to cross-check my development team
> against the perpetual tendency to gum-up the code and slow
> things down. So in order to make the testing effective for the developers,
> I need to perf test _very_specific_ things. Trying to
> performance test the "system" as a whole is nearly an impossible task -
> not only because there are so many variables that influence
> the tests, but precisely because "all of those variables" make it
> impossible to debug which one causes the bottleneck when there is
> a change in performance from one release to the next. (Have you ever sent
> your programmers off to 'fix' a performance problem that
> turned out to be caused by an O/S update on your server? I have...)
>
> Instead, we create performance tests that test specific functional
> systems. That is, the "login" perf test. The "registration" perf
> test. The "..." perf test. Each one of these tests is run independently,
> so that when we encounter a slower benchmark - we can tell
> the developers immediately where to concentrate their efforts in fixing
> the problem. (We also monitor all parts of the system (CPU,
> IO, Database Transactions (reads, writes, full table scans, etc.) from all
> servers involved in the test. The goal is not to simulate
> 'real user activity', it is to max out the capacity of at least 1 of the
> servers in the test (specifically the one executing the
> 'application logic'). If we max out that one server, we know that our
> 'benchmark' is the most we can expect of a single member of
> our cluster of machines. (We also test a cluster of 2 machines - and
> measure the fall-off in capacity between a 1-member cluster and
> 2-member cluster, this gives us an idea of how much impact our
> 'clustering' system has on performance as well.) I suppose you could
> say that I look at it as if, we measure the 'maximum capacity', and so
> long as the number of users doesn't exceed that - we will
> perform OK.
>
> We do run some 'all-encompassing' system tests as well, but those are more
> for 'stress' testing than for performance benchmarking.
> We are specifically looking for things that start to break-down after
> hours of continuous operation at peak capacity. So we monitor
> error logs and look to make sure that we aren't throwing errors while
> under stress.
>
> The number one thing to keep in mind about performance testing is that you
> have to use 'real data'. We actually download our
> production database every weekend, and strip out any 'personal
> information' (stuff that we protect in our production environment) by
> either nulling it out, or replacing it with bogus data. This allows us to
> run our performance tests against a database that has 100s
> of millions of rows of data. Nearly all of our performance 'bugs' have
> been caused by poor data handling in the code (SQL requests
> that don't use indices (causing a full table scan), badly formed joins,
> fetching a few rows of data and then looping through them in
> the code (when the 'few rows of data' from your 'dev' environment become
> 100,000 rows with the production data, this tends to bog
> the code down a lot), etc.). So if you are testing with 'faked' data, odds
> are good you will miss a lot of performance issues - no
> matter what form of performance testing you use.
>
> I will say that we have served over 130M web pages in 1 month, using only
> 5 servers (4 tomcats, and 1 DB server)... Those pages
> represented about 10X in "GET" requests to our servers...
>
> --
> Robin D. Wilson
> Sr. Director of Web Development
> KingsIsle Entertainment, Inc.
> http://www.kingsisle.com
>
>
> -----Original Message-----
> From: nmq [mailto:nmq0607@gmail.com]
> Sent: Monday, July 01, 2013 2:33 PM
> To: JMeter Users List
> Subject: Establishing baseline metrics
>
> Hi all
>
> This is not a JMeter specific questions but since this user list comprises
> of experts in performance testing, I figured it would be
> a good place to ask this question.
>
> My question is how do you establish baselines for a website's performance
> if you do not have any historic data? Lets say this is a
> new website and its for a limited number of customers.
>
> How do you determine what should be the number of concurrent users you
> should simulate.
>
> Lets say the executives say off at the top of their heads, that the
> maximum number of concurrent users would be 50 at peak times.
> Does that mean I should not go beyond 50 or should I still do tests with a
> higher number?
>
> How can I go about establishing baselines for page load times, if I do not
> have any historic data and have no industry benchmarks or
> competitor data.
>
> Would it make sense to say let's see how the website is doing throughout
> the development phase and establish our baseline using the
> current response times?
>
> I would appreciate any input.
>
>
> Regards
> Sam
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>
RE: Establishing baseline metrics
Posted by "Robin D. Wilson" <rw...@gmail.com>.
I'm thinking I look at performance testing differently than a lot of people... For me, the objective of performance testing is to
establish what your system _can_ do, not what you need to accomplish. So when you are setting up your tests, you are trying to drive
your systems at maximum capacity for some extended period of time. Then you measure that capacity as your 'baseline'.
For every subsequent release of your code, you measure it against the 'baseline', and determine whether the code got faster or
slower. If you determine that the slower (or faster) response is acceptable to your end users (because you were nowhere near the
user's acceptable standard), you can reset your baseline to that standard. If your slower standard is encroaching on the usability
of the system - you can declare that baseline as the minimum spec, and then fail any code that exceeds that standard.
As for how you determine what is acceptable to a 'user', that can be handled in a number of ways - without actually improving the
'real' performance of the system. Consider a web page that loads a bunch of rows of data in a big table. For most users, if you can
start reading the table within 1-2 seconds, that is acceptable for a system's performance. But if there are hundreds of rows of
data, you would not need to load _all_ the rows within 1-2 seconds to actually meet their performance criteria. You only need to
load enough rows that the table fills the browser - so they can start reading - within the 1-2 second period. JMeter cannot really
measure this timing, it can only measure the 'overall response time' (indeed, I don't know any testing tool that can do it). So
trying to define a performance benchmark in terms of what 'users' experience is really difficult, and nearly useless (to me anyway).
I look at performance testing as a way to cross-check my development team against the perpetual tendency to gum-up the code and slow
things down. So in order to make the testing effective for the developers, I need to perf test _very_specific_ things. Trying to
performance test the "system" as a whole is nearly an impossible task - not only because there are so many variables that influence
the tests, but precisely because "all of those variables" make it impossible to debug which one causes the bottleneck when there is
a change in performance from one release to the next. (Have you ever sent your programmers off to 'fix' a performance problem that
turned out to be caused by an O/S update on your server? I have...)
Instead, we create performance tests that test specific functional systems. That is, the "login" perf test. The "registration" perf
test. The "..." perf test. Each one of these tests is run independently, so that when we encounter a slower benchmark - we can tell
the developers immediately where to concentrate their efforts in fixing the problem. (We also monitor all parts of the system (CPU,
IO, Database Transactions (reads, writes, full table scans, etc.) from all servers involved in the test. The goal is not to simulate
'real user activity', it is to max out the capacity of at least 1 of the servers in the test (specifically the one executing the
'application logic'). If we max out that one server, we know that our 'benchmark' is the most we can expect of a single member of
our cluster of machines. (We also test a cluster of 2 machines - and measure the fall-off in capacity between a 1-member cluster and
2-member cluster, this gives us an idea of how much impact our 'clustering' system has on performance as well.) I suppose you could
say that I look at it as if, we measure the 'maximum capacity', and so long as the number of users doesn't exceed that - we will
perform OK.
We do run some 'all-encompassing' system tests as well, but those are more for 'stress' testing than for performance benchmarking.
We are specifically looking for things that start to break-down after hours of continuous operation at peak capacity. So we monitor
error logs and look to make sure that we aren't throwing errors while under stress.
The number one thing to keep in mind about performance testing is that you have to use 'real data'. We actually download our
production database every weekend, and strip out any 'personal information' (stuff that we protect in our production environment) by
either nulling it out, or replacing it with bogus data. This allows us to run our performance tests against a database that has 100s
of millions of rows of data. Nearly all of our performance 'bugs' have been caused by poor data handling in the code (SQL requests
that don't use indices (causing a full table scan), badly formed joins, fetching a few rows of data and then looping through them in
the code (when the 'few rows of data' from your 'dev' environment become 100,000 rows with the production data, this tends to bog
the code down a lot), etc.). So if you are testing with 'faked' data, odds are good you will miss a lot of performance issues - no
matter what form of performance testing you use.
I will say that we have served over 130M web pages in 1 month, using only 5 servers (4 tomcats, and 1 DB server)... Those pages
represented about 10X in "GET" requests to our servers...
--
Robin D. Wilson
Sr. Director of Web Development
KingsIsle Entertainment, Inc.
http://www.kingsisle.com
-----Original Message-----
From: nmq [mailto:nmq0607@gmail.com]
Sent: Monday, July 01, 2013 2:33 PM
To: JMeter Users List
Subject: Establishing baseline metrics
Hi all
This is not a JMeter specific questions but since this user list comprises of experts in performance testing, I figured it would be
a good place to ask this question.
My question is how do you establish baselines for a website's performance if you do not have any historic data? Lets say this is a
new website and its for a limited number of customers.
How do you determine what should be the number of concurrent users you should simulate.
Lets say the executives say off at the top of their heads, that the maximum number of concurrent users would be 50 at peak times.
Does that mean I should not go beyond 50 or should I still do tests with a higher number?
How can I go about establishing baselines for page load times, if I do not have any historic data and have no industry benchmarks or
competitor data.
Would it make sense to say let's see how the website is doing throughout the development phase and establish our baseline using the
current response times?
I would appreciate any input.
Regards
Sam
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Establishing baseline metrics
Posted by nmq <nm...@gmail.com>.
Sorry, I just hit reply instead of directing my question.
This is in response to Deepak S's comment:
*"Ill also add that the times are usually as seen by the end user - so the
times as seen by Jmeter usually have to be lesser (since these are max
times you need to consider empty cache for first time access or slower
networks etc)."*
I have added a Cache manager to my test which is cleared on each iteration
and checked the option of retrieving all embedded resources from HTML
files. I'm also using concurrent pool with a size of 8.
Doesn't that give higher or closer times to those seen by end user?
Regards
Sam
> On Mon, Jul 1, 2013 at 3:44 PM, Deepak Shetty <sh...@gmail.com> wrote:
>
>> Ill also add that the times are usually as seen by the end user - so the
>> times as seen by Jmeter usually have to be lesser (since these are max
>> times you need to consider empty cache for first time access or slower
>> networks etc). You also usually use something like 90 or 95 percentile
>>
>>
>>
Re: Establishing baseline metrics
Posted by nmq <nm...@gmail.com>.
I have added a Cache manager to my test which is cleared on each iteration
and checked the option of retrieving all embedded resources from HTML
files. I'm also using concurrent pool with a size of 8.
Doesn't that give higher or closer times to those seen by end user?
Regards
Sam
On Mon, Jul 1, 2013 at 3:44 PM, Deepak Shetty <sh...@gmail.com> wrote:
> Ill also add that the times are usually as seen by the end user - so the
> times as seen by Jmeter usually have to be lesser (since these are max
> times you need to consider empty cache for first time access or slower
> networks etc). You also usually use something like 90 or 95 percentile
>
>
> On Mon, Jul 1, 2013 at 12:42 PM, Deepak Shetty <sh...@gmail.com> wrote:
>
> > >Does that mean I should not go beyond 50 or should I still do tests with
> > a higher number?
> > You usually have to factor in growth (growing at a rate of X users per
> > month or whatever with whenever is your next scheduled release where you
> > could make reasonable amount of optimisations) - i.e. capacity planning.
> >
> >
> > >How can I go about establishing baselines for page load times, if I do
> > not have any historic data and have no industry benchmarks or >competitor
> > data.
> > There are some usability tests as to what times cause user's to perceive
> > as slow. (Normal navigation has different values then login or search or
> > more secure pages like checkout) - you can google these. These are
> specific
> > to industry and functionality. (for e.g. we have 2 seconds for search
> page
> > (most heavily used part of the website) , 4 seconds for catalog pages,
> > 6seconds for login/secure pages/ pages that have any ERP integration) -
> but
> > its better if you actually conduct some usability tests with end usersor
> if
> > you can find equivalents for your industry.
> >
> >
> > >Would it make sense to say let's see how the website is doing throughout
> > >the development phase and establish our baseline using the current
> > response
> > >times?
> > No! you have to determine the times you want(as above) and see that your
> > current response satisfies those (suppose your current page takes 60
> > seconds are you satisfied that , that is your baseline?)
> >
> >
> > On Mon, Jul 1, 2013 at 12:33 PM, nmq <nm...@gmail.com> wrote:
> >
> >> Hi all
> >>
> >> This is not a JMeter specific questions but since this user list
> comprises
> >> of experts in performance testing, I figured it would be a good place to
> >> ask this question.
> >>
> >> My question is how do you establish baselines for a website's
> performance
> >> if you do not have any historic data? Lets say this is a new website
> and
> >> its for a limited number of customers.
> >>
> >> How do you determine what should be the number of concurrent users you
> >> should simulate.
> >>
> >> Lets say the executives say off at the top of their heads, that the
> >> maximum
> >> number of concurrent users would be 50 at peak times. Does that mean I
> >> should not go beyond 50 or should I still do tests with a higher number?
> >>
> >> How can I go about establishing baselines for page load times, if I do
> not
> >> have any historic data and have no industry benchmarks or competitor
> data.
> >>
> >> Would it make sense to say let's see how the website is doing throughout
> >> the development phase and establish our baseline using the current
> >> response
> >> times?
> >>
> >> I would appreciate any input.
> >>
> >>
> >> Regards
> >> Sam
> >>
> >
> >
>
Re: Establishing baseline metrics
Posted by Deepak Shetty <sh...@gmail.com>.
Ill also add that the times are usually as seen by the end user - so the
times as seen by Jmeter usually have to be lesser (since these are max
times you need to consider empty cache for first time access or slower
networks etc). You also usually use something like 90 or 95 percentile
On Mon, Jul 1, 2013 at 12:42 PM, Deepak Shetty <sh...@gmail.com> wrote:
> >Does that mean I should not go beyond 50 or should I still do tests with
> a higher number?
> You usually have to factor in growth (growing at a rate of X users per
> month or whatever with whenever is your next scheduled release where you
> could make reasonable amount of optimisations) - i.e. capacity planning.
>
>
> >How can I go about establishing baselines for page load times, if I do
> not have any historic data and have no industry benchmarks or >competitor
> data.
> There are some usability tests as to what times cause user's to perceive
> as slow. (Normal navigation has different values then login or search or
> more secure pages like checkout) - you can google these. These are specific
> to industry and functionality. (for e.g. we have 2 seconds for search page
> (most heavily used part of the website) , 4 seconds for catalog pages,
> 6seconds for login/secure pages/ pages that have any ERP integration) - but
> its better if you actually conduct some usability tests with end usersor if
> you can find equivalents for your industry.
>
>
> >Would it make sense to say let's see how the website is doing throughout
> >the development phase and establish our baseline using the current
> response
> >times?
> No! you have to determine the times you want(as above) and see that your
> current response satisfies those (suppose your current page takes 60
> seconds are you satisfied that , that is your baseline?)
>
>
> On Mon, Jul 1, 2013 at 12:33 PM, nmq <nm...@gmail.com> wrote:
>
>> Hi all
>>
>> This is not a JMeter specific questions but since this user list comprises
>> of experts in performance testing, I figured it would be a good place to
>> ask this question.
>>
>> My question is how do you establish baselines for a website's performance
>> if you do not have any historic data? Lets say this is a new website and
>> its for a limited number of customers.
>>
>> How do you determine what should be the number of concurrent users you
>> should simulate.
>>
>> Lets say the executives say off at the top of their heads, that the
>> maximum
>> number of concurrent users would be 50 at peak times. Does that mean I
>> should not go beyond 50 or should I still do tests with a higher number?
>>
>> How can I go about establishing baselines for page load times, if I do not
>> have any historic data and have no industry benchmarks or competitor data.
>>
>> Would it make sense to say let's see how the website is doing throughout
>> the development phase and establish our baseline using the current
>> response
>> times?
>>
>> I would appreciate any input.
>>
>>
>> Regards
>> Sam
>>
>
>
Re: Establishing baseline metrics
Posted by Deepak Shetty <sh...@gmail.com>.
>Does that mean I should not go beyond 50 or should I still do tests with a
higher number?
You usually have to factor in growth (growing at a rate of X users per
month or whatever with whenever is your next scheduled release where you
could make reasonable amount of optimisations) - i.e. capacity planning.
>How can I go about establishing baselines for page load times, if I do not
have any historic data and have no industry benchmarks or >competitor data.
There are some usability tests as to what times cause user's to perceive as
slow. (Normal navigation has different values then login or search or more
secure pages like checkout) - you can google these. These are specific to
industry and functionality. (for e.g. we have 2 seconds for search page
(most heavily used part of the website) , 4 seconds for catalog pages,
6seconds for login/secure pages/ pages that have any ERP integration) - but
its better if you actually conduct some usability tests with end usersor if
you can find equivalents for your industry.
>Would it make sense to say let's see how the website is doing throughout
>the development phase and establish our baseline using the current response
>times?
No! you have to determine the times you want(as above) and see that your
current response satisfies those (suppose your current page takes 60
seconds are you satisfied that , that is your baseline?)
On Mon, Jul 1, 2013 at 12:33 PM, nmq <nm...@gmail.com> wrote:
> Hi all
>
> This is not a JMeter specific questions but since this user list comprises
> of experts in performance testing, I figured it would be a good place to
> ask this question.
>
> My question is how do you establish baselines for a website's performance
> if you do not have any historic data? Lets say this is a new website and
> its for a limited number of customers.
>
> How do you determine what should be the number of concurrent users you
> should simulate.
>
> Lets say the executives say off at the top of their heads, that the maximum
> number of concurrent users would be 50 at peak times. Does that mean I
> should not go beyond 50 or should I still do tests with a higher number?
>
> How can I go about establishing baselines for page load times, if I do not
> have any historic data and have no industry benchmarks or competitor data.
>
> Would it make sense to say let's see how the website is doing throughout
> the development phase and establish our baseline using the current response
> times?
>
> I would appreciate any input.
>
>
> Regards
> Sam
>