You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@jmeter.apache.org by Rajiv Nanduani <ra...@gmail.com> on 2011/05/28 02:39:51 UTC

How are throughput and response time related?

Hi All,

I was reading about performance testing and gone through below this.. But I
am not clear Is this true or false..? please explain your thoughts/Inputs on
this paragraph.


*How are throughput and response time related?*

The graph for throughput displays the quantity of data that the server sent
to the Vusers in one second. When throughput is compared to the response
time of transaction, it decreased as the response time decreased. In
addition to this, the highest throughput and response times occur at
approximately the same time.



it decreased as the response time decreased  Is it True ??

-- 

Regards

*RAJIV <http://testeverythingrajiv.com/>
KUMAR<http://rajivkumarnandvani.wordpress.com/>
NANDVANI <http://testeverythingqtp.blogspot.com>*

http://rajivkumarnandvani.wordpress.com
http://testeverythingqtp.blogspot.com/

Re: How are throughput and response time related?

Posted by oliver <ol...@hotmail.com>.

No, it's not. Not in the sense you want it to be.

There is no direct relation between throughput and response times, they are
just two metrics that form a part of the bigger picture. Yes, sure, if one
metric were to change it may correlate with the other one and how this
happens tells you something that you may be able to use, but only in
conjunction with other metrics.

The text you have quoted appears to be describing an example situation,
something that happened once when John Doe was testing X application with Y
load, but it is not a prediction (it is trying to use past tense although it
fails at this in certain points). So in that example, maybe "it decreased as
the response time decreased" but that does not make it a hard and fast rule
that you can quote and rely on. You can NOT say "if throughput drops then so
will resp. times". This is simply false.

To fully get this, you need to gain a further understanding of what these
metrics actually are. You've already written the definitions yourself, so
now you need to imagine what situations could *potentially* cause throughput
to affect resp. times. - there's a lot of information out there on this,
shouldn't be hard. Google it and find an example that does not use an
electronic analogy; the logic remains the same.

--
View this message in context: http://jmeter.512774.n5.nabble.com/How-are-throughput-and-response-time-related-tp4433821p4434643.html
Sent from the JMeter - User mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: How are throughput and response time related?

Posted by deicool <de...@gmail.com>.

Hey

Throughput = Number Of Concurrent Users
                   =====================
                    (Response Time + Think Time)

Number Of Concurrent Users = Customers logged into the system at the same
time
Think Time = Time one user or customer takes to respond between two
consecutive transactions
Throughput = Number of Hits/ sec or Transactions/sec
Response Time = Time taken by the system to respond to a transaction
request.

Throughput is inversely proportional to Response Time. More the Throughput,
less the Response Time and Less the Throughput, more the Response Time for
the same Number Of Concurrent Users and Think Time

Have fun
:)
Deepak
http://www.simtree.net

--
View this message in context: http://jmeter.512774.n5.nabble.com/How-are-throughput-and-response-time-related-tp4433821p4435229.html
Sent from the JMeter - User mailing list archive at Nabble.com.

Re: How are throughput and response time related?

Posted by oliver <ol...@hotmail.com>.

Hi Deepak,

Right, I think I see where I got confused.

You are talking about the situation where the system is running at full
capacity - maximum throughput. So, yes, in this situation then it is logical
that it is not possible to increase the load any further without suffering
longer response times.

I was talking about situation where a system is not running at capacity or
saturation point and still has headroom left to increase throughput without
incurring any or negligible cost in terms of response times. You know, it's
that classic knee-bend scenario where you have a steady increase in load
overlayed with flat response times over time until they suddenly increase
exponentially at a certain point and your throughput levels off...

Hmmm, I have a strong urge to draw a graph.

Instead, I spent 30 seconds on Google and found this:
http://www.1202performance.com/2009/08/how-do-you-know-if-your-load-test-has-a-bottleneck/
an excellent little blog post

Andrew - our guest blogger - has rather succinctly summarised what this
thread is talking about.

Notice how Andrew has used 'transactions/per second' as his label for
Throughput and not 'bytes' - good man Andrew, this guy knows his stuff.

--
View this message in context: http://jmeter.512774.n5.nabble.com/How-are-throughput-and-response-time-related-tp4433821p4437962.html
Sent from the JMeter - User mailing list archive at Nabble.com.

Re: How are throughput and response time related?

Posted by Kirk <ki...@gmail.com>.

Think of your last trip to disney or your favorite amusement park. Lets define capacity of the ride to be the number of people that can sit on the ride per turn (think roller coaster). Throughput will be the number of people that exit the ride per unit of time. Lets define service time the the amount of time you get to sit on the ride. Lets define response time or latency to be your time queuing for the ride (dead time) plus service time. 

So, if there are no lineups at your favorite ride, latency is your time on the ride. Capacity remains the same but measured throughput will be minimal. As people start to arrive you may monetarily have more people than can fit in the roller coaster so they will start to queue. So, throughput will start to go up but average latency will also start to increase as a few people might have to queue. But as you can imagine, there will still be plenty of spare capacity (empty seats on the roller coaster). As the crowd moves in, the situation will change, system point of view, capacity will be better if not perfectly utilized which will maximize throughput. But, your latency is now time on ride + all that dead time in the queue. So from an individual's put of view, the rate at which you can work (rides per hour) will have decreased but from a system point of view, it's transactional rate is maximal.

Hardware is a bucket of non-shareable (seats on a roller coaster) things. Number of seats in a CPU == number of cores, number of seats on the network == number of bytes in the packet's payload... bus, memory, disk channels... and so on. So, there is a relationship between latency and throughput but in a computer is tends to be a lot more complicated than the simple roller coaster story. That said, I think the analogy can be used to help you sort out the concepts.

Regards,
Kirk

> Hey
> 
> You will have to improve the system to get the same performance time (Even
> Google for that matter). If the system remains the same, and if you increase
> the number of request even by one, the response time will increase. You will
> have to increase the hardware or tune the system to get the same performance
> time.
> 
> :)
> Deepak
> 
> --
> View this message in context: http://jmeter.512774.n5.nabble.com/How-are-throughput-and-response-time-related-tp4433821p4437611.html
> Sent from the JMeter - User mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: How are throughput and response time related?

Posted by deicool <de...@gmail.com>.

Hey

You will have to improve the system to get the same performance time (Even
Google for that matter). If the system remains the same, and if you increase
the number of request even by one, the response time will increase. You will
have to increase the hardware or tune the system to get the same performance
time.

:)
Deepak

--
View this message in context: http://jmeter.512774.n5.nabble.com/How-are-throughput-and-response-time-related-tp4433821p4437611.html
Sent from the JMeter - User mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: How are throughput and response time related?

Posted by oliver <ol...@hotmail.com>.

Do I really want to argue this? Where to start? Sigh, look Deepak, to be fair
to you, I think you might have misunderstood something, sometimes the
terminology for this sort of thing can be different from one person to the
next so perhaps throughput means something else to you - this happens a lot
in perf. testing, 'stress' = this, 'soak' = that - everyone thinks they know
the right answer.

If not, then maybe if you think of it like this:

I am google. I have a website where people make X number of searches a
second and I am deadly serious about every request being fast. I know that
when I am getting 10 search requests a second the response time is 300ms. I
also know that when things are busy, I sometimes get 20 search requests a
second and when this happens the response times still stay at 300ms because,
hey, I wrote a pretty performant system and it can handle large volumes of
requests. I also know if I get 3000 requests a second is STILL stays at
300ms. Because I am Google. If what you say is true then, man, google would
be slooooow, but it's not slow, QED.

I would wager you are not testing at google but it's the same logic. If your
system really gets proportionally slower as the rate of requests increases,
then, well, it sounds like you might have a problem with your system.

--
View this message in context: http://jmeter.512774.n5.nabble.com/How-are-throughput-and-response-time-related-tp4433821p4437547.html
Sent from the JMeter - User mailing list archive at Nabble.com.

Re: How are throughput and response time related?

Posted by deicool <de...@gmail.com>.

Hey 
(Line 6) 
The response time will never stay at 300 ms if you double your 
throughput from 10 to 20. It will be always more than 300 ms  (For the 
same system). 

Infact if you just increase from 10 to 11, you will see an increase in 
response time. 

Try it on any system :) 

Have fun 
Deepak 

--
View this message in context: http://jmeter.512774.n5.nabble.com/How-are-throughput-and-response-time-related-tp4433821p4437507.html
Sent from the JMeter - User mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: How are throughput and response time related?

Posted by oliver <ol...@hotmail.com>.

"Throughput is inversely proportional to Response Time." Er, that's not true.

It is worrying how often people fail to grasp this basic concept. It's
really not rocket science guys.

I'm feeling generous so let's break it down. 'Throughput' is actually formed
of two words, 'through' and 'put', as in how much can I 'put' 'though'. It
is a *rate* so it is measured in an amount over time. Typically, in
computing it is a rate of bytes per second. But you can also think about
gallons of water flowing through a pipe if this helps.

In every system, electronic or otherwise, there is a limit - a maximum
throughput of some sort. Once this is reached then you would expect the
system to start behaving badly, this usually results in longer response
times but not necessarily. You might also see errors, HTTP503s and the like.
It depends.

Look, if my test is making 10 requests a second and each of those requests
is 10kb in size then my current throughput is 100kb/ps. At the same time I
might see avg. response times of 300ms.

If I double the load to 20 requests a second then my throughput increases to
200kb/ps. What happens to my avg. response time? Will it automatically
increase to 600ms? Of course not! It might, sure, who knows right? This is
why we test stuff. But it could equally just stay the same at 300ms.

But at some point, if you keep on increasing throughput you will eventually
reach a limit. Say I ramp up the test to 100 reqs/per sec and at this point
I start to see response times grow. Now, typically, even if I try to
increase the rate or requests I will not be able to make the throughput go
any higher - this is called a bottleneck. All that will happen if I increase
the load is I will get longer and longer response times until it all gets a
bit messy. In this scenario, my maximum throughput was shown to be
1000kb/ps.

But the affect of reaching a bottleneck could be anything, not JUST greater
response times. You might continue to see response times of 300ms but start
to get failures. Or, maybe a back end process starts to slow down or fail,
or memory may start to grow, or images no longer get rendered properly, or
blah, blah, blah. If you only focus on response times you are possibly
(probably) not testing your system properly.

But what if I change the test I ran earlier to make a call to a different
page? This time the page is bigger with a size of 20kbs - twice the size of
the old one. Now, I rerun the test again and I again reach 100 reqs/per sec
and then I hit a bottleneck. But this time the maximum throughput I got to
was 2000kb/ps! But hang on, I thought the maximum throughput was 1000kb/ps!
Not neccessarily, what if the bottleneck was actually not bandwidth but
somewhere else, like say, in the database.

You are surely going to fail if you try to analyse your system by only
looking at throughput in bytes per sec vs. response times. You could run the
first test and proclaim to the world that you've found the maximum
throughput! You would run around the office shouting "this system can only
reach 1000kb/ps! Let's buy a bigger Switch box, I want 100Gbit ethernet
cards! I demand fiber optics everywhere!" And then your company might spend
10 gadzillion dollars installing all this crap and you rerun your test and
still get the same 1000kb/ps bottleneck. You see how you might look a bit
stupid at this point?

This is why it is eternally better to talk about throughput in business
terms. Stop poring over at those sexy bytes per second graphs, go on, stop
it, it'll make you go blind staring at that stuff. Instead, start looking at
some real data that tells you the rate of requests for your *business*
transactions. Like: logins per second, registrations per second, payments
per second. It's unlikely that you will actually reach a bandwidth limit
(I've had this twice in 15 years vs. 100s of other types of bottlenecks) so
what you should really care about is throughput in terms of business
transactions per second (hour, minute, whatever). Or API calls a second, or
page views, or whatever works for you best.

I'm bored now but if you don't get these simple concepts then you should not
be trying to run performance tests. My advice to anyone who feels
enlightened by this text is to stop what you are doing, take your hands away
from JMeter, you are not helping anyone. Instead, open a browser and spend a
few hours reading wikipedia and friends. I promise you, if you teach
yourself these basics you will find that you start to enjoy your job more,
your sense of self worth will increase and you will become a happier person.

--
View this message in context: http://jmeter.512774.n5.nabble.com/How-are-throughput-and-response-time-related-tp4433821p4437493.html
Sent from the JMeter - User mailing list archive at Nabble.com.

Re: How are throughput and response time related?

Posted by Deepak Shetty <sh...@gmail.com>.

> it decreased as the response time decreased  Is it True ??
Its possible when you reduce the number of concurrent hits (lower throughput
and lower response time because of reduced load).
Since you seem to be reading someones report you need to see what steps were
performed.

You should also distinguish between "throughput" of a system (usually taken
to be the *maximum* requests per unit of time that the system can take
without erroring and with acceptable response time) and the throughput as
seen by the client (which is the current value that you are getting based on
how many requests you run concurrently, the pages you are accessing in the
test, the think times you have provided etc)

regards
deepak

On Fri, May 27, 2011 at 5:39 PM, Rajiv Nanduani <
rajivkumarnandvani@gmail.com> wrote:

> Hi All,
>
> I was reading about performance testing and gone through below this.. But I
> am not clear Is this true or false..? please explain your thoughts/Inputs
> on
> this paragraph.
>
>
> *How are throughput and response time related?*
>
> The graph for throughput displays the quantity of data that the server sent
> to the Vusers in one second. When throughput is compared to the response
> time of transaction, it decreased as the response time decreased. In
> addition to this, the highest throughput and response times occur at
> approximately the same time.
>
>
>
> it decreased as the response time decreased  Is it True ??
>
> --
>
> Regards
>
> *RAJIV <http://testeverythingrajiv.com/>
> KUMAR<http://rajivkumarnandvani.wordpress.com/>
> NANDVANI <http://testeverythingqtp.blogspot.com>*
>
> http://rajivkumarnandvani.wordpress.com
> http://testeverythingqtp.blogspot.com/
>