90% line calculation

[m mat <pe...@yahoo.com>]

I want to understand the explaination of the 90% line in aggregate report. 
 
What I think it should be: Average of samples with 90% significance, i.e. if you assume a bell shaped distribution around the mean, this number should be an average of the middle 90% numbers (as in average after top 5% and bottom 5% outliers are removed). So that this number could be inferred as "the response time 90% of your users are likely to see". For example if you have 10 samples 1,2,3,4,5,6,7,8,9,10. 90th percentile should be (2+3+4+5+6+7+8+9)/8 = 5.5
 
it seems to me JMeter reports this number as the 90% value in the sample set. So for the above sample set JMeter would report 9
 
Which one of the above two is correct for JMeter?
Matt

		
---------------------------------
 Yahoo! Music Unlimited - Access over 1 million songs. Try it free.

Re: 90% line calculation

[Michael Stover <ms...@apache.org>]

Right.  I'm wondering what a graph might look like in non-gui mode :-)

On Tue, 2005-10-11 at 10:14 +0100, sebb wrote:
> Or just load the results file in the relevant Listener after the test
> has finished.
> 
> S.
> On 10/10/05, Peter Lin <wo...@gmail.com> wrote:
> > currently there isn't a way to do that in console mode. I've been working on
> > a reporting tool for jmeter, but the last two weeks haven't been too
> > productive. I got the flu 2 weeks back and last week I didn't work on the
> > reporting tool.
> >
> > once we get Jmeter moved to Subversion, I will continue working on the
> > reporting tool.
> >
> > hope that answers your question
> >
> > peter
> >
> >
> > On 10/10/05, m mat <pe...@yahoo.com> wrote:
> > >
> > > Peter/Michael
> > >
> > > Thanks that explains the number and the rational.
> > >
> > > One more related question, I am running the tests in non GUI mode, is
> > > there a way to get the aggregate report/aggregate graph report in non GUI
> > > mode?
> > >
> > > Matt
> > >
> > >
> >
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: 90% line calculation

[sebb <se...@gmail.com>]

Or just load the results file in the relevant Listener after the test
has finished.

S.
On 10/10/05, Peter Lin <wo...@gmail.com> wrote:
> currently there isn't a way to do that in console mode. I've been working on
> a reporting tool for jmeter, but the last two weeks haven't been too
> productive. I got the flu 2 weeks back and last week I didn't work on the
> reporting tool.
>
> once we get Jmeter moved to Subversion, I will continue working on the
> reporting tool.
>
> hope that answers your question
>
> peter
>
>
> On 10/10/05, m mat <pe...@yahoo.com> wrote:
> >
> > Peter/Michael
> >
> > Thanks that explains the number and the rational.
> >
> > One more related question, I am running the tests in non GUI mode, is
> > there a way to get the aggregate report/aggregate graph report in non GUI
> > mode?
> >
> > Matt
> >
> >
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: 90% line calculation

[Peter Lin <wo...@gmail.com>]

currently there isn't a way to do that in console mode. I've been working on
a reporting tool for jmeter, but the last two weeks haven't been too
productive. I got the flu 2 weeks back and last week I didn't work on the
reporting tool.

once we get Jmeter moved to Subversion, I will continue working on the
reporting tool.

hope that answers your question

peter


On 10/10/05, m mat <pe...@yahoo.com> wrote:
>
> Peter/Michael
>
> Thanks that explains the number and the rational.
>
> One more related question, I am running the tests in non GUI mode, is
> there a way to get the aggregate report/aggregate graph report in non GUI
> mode?
>
> Matt
>
>

Re: 90% line calculation

[m mat <pe...@yahoo.com>]

Peter/Michael
 
Thanks that explains the number and the rational.
 
One more related question, I am running the tests in non GUI mode, is there a way to get the aggregate report/aggregate graph report in non GUI mode?
 
Matt

Peter Lin <wo...@gmail.com> wrote:
as mike already explained, the 90% is "90% of the samples finished within
0-x time".

Since I instigated this feature, I'll explain the purpose for it. In my line
work, requirements are expressed in terms of "90% of the requests must
finish within X milliseconds". This is driven by SLA (service level
agreements) contracts, which tell the customer "you can expect 90% of the
requests to finish within x time." if 11% of the requests for a given month
exceed the gaurantee, the customer gets a discount.

The average response time is a good number to know, but from a performance
requirement perspective, it normally doesn't drive SLA's. In a
business-to-business service application, it doesn't matter if the average
response time is 500ms, if the 90% line is 1.5 seconds.

from a developer perspective, if the 50% and 90% line are far appart, ie
(50% - 550ms, 90% - 5000ms), it tells you the application fluctuates a lot.
In an ideal situtation, the 90% is less than double the 50% line. Depending
on the application, this might not be possible. hope that helps

peter lin


On 10/10/05, Michael Stover wrote:
>
> The 90% line tells you that 90% of the samples fell at or below that
> number. It works like median rather than like mean. The advantage of
> such a measure is it allows you to assert something like "90% of
> requests were handled in x amount of time". With an average, you can
> make no such assertion about what users experience - you have no handle
> on what users experience during busy periods.
>
> -Mike
>
> On Mon, 2005-10-10 at 12:15 -0700, m mat wrote:
> > I want to understand the explaination of the 90% line in aggregate
> report.
> >
> > What I think it should be: Average of samples with 90% significance, i.e.
> if you assume a bell shaped distribution around the mean, this number should
> be an average of the middle 90% numbers (as in average after top 5% and
> bottom 5% outliers are removed). So that this number could be inferred as
> "the response time 90% of your users are likely to see". For example if you
> have 10 samples 1,2,3,4,5,6,7,8,9,10. 90th percentile should be
> (2+3+4+5+6+7+8+9)/8 = 5.5
> >
> > it seems to me JMeter reports this number as the 90% value in the sample
> set. So for the above sample set JMeter would report 9
> >
> > Which one of the above two is correct for JMeter?
> > Matt
> >
> >
> > ---------------------------------
> > Yahoo! Music Unlimited - Access over 1 million songs. Try it free.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
>
>

		
---------------------------------
 Yahoo! Music Unlimited - Access over 1 million songs. Try it free.

Re: 90% line calculation

[Peter Lin <wo...@gmail.com>]

as mike already explained, the 90% is "90% of the samples finished within
0-x time".

Since I instigated this feature, I'll explain the purpose for it. In my line
work, requirements are expressed in terms of "90% of the requests must
finish within X milliseconds". This is driven by SLA (service level
agreements) contracts, which tell the customer "you can expect 90% of the
requests to finish within x time." if 11% of the requests for a given month
exceed the gaurantee, the customer gets a discount.

The average response time is a good number to know, but from a performance
requirement perspective, it normally doesn't drive SLA's. In a
business-to-business service application, it doesn't matter if the average
response time is 500ms, if the 90% line is 1.5 seconds.

from a developer perspective, if the 50% and 90% line are far appart, ie
(50% - 550ms, 90% - 5000ms), it tells you the application fluctuates a lot.
In an ideal situtation, the 90% is less than double the 50% line. Depending
on the application, this might not be possible. hope that helps

peter lin


On 10/10/05, Michael Stover <ms...@apache.org> wrote:
>
> The 90% line tells you that 90% of the samples fell at or below that
> number. It works like median rather than like mean. The advantage of
> such a measure is it allows you to assert something like "90% of
> requests were handled in x amount of time". With an average, you can
> make no such assertion about what users experience - you have no handle
> on what users experience during busy periods.
>
> -Mike
>
> On Mon, 2005-10-10 at 12:15 -0700, m mat wrote:
> > I want to understand the explaination of the 90% line in aggregate
> report.
> >
> > What I think it should be: Average of samples with 90% significance, i.e.
> if you assume a bell shaped distribution around the mean, this number should
> be an average of the middle 90% numbers (as in average after top 5% and
> bottom 5% outliers are removed). So that this number could be inferred as
> "the response time 90% of your users are likely to see". For example if you
> have 10 samples 1,2,3,4,5,6,7,8,9,10. 90th percentile should be
> (2+3+4+5+6+7+8+9)/8 = 5.5
> >
> > it seems to me JMeter reports this number as the 90% value in the sample
> set. So for the above sample set JMeter would report 9
> >
> > Which one of the above two is correct for JMeter?
> > Matt
> >
> >
> > ---------------------------------
> > Yahoo! Music Unlimited - Access over 1 million songs. Try it free.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: jmeter-user-help@jakarta.apache.org
>
>

Re: 90% line calculation

[Michael Stover <ms...@apache.org>]

The 90% line tells you that 90% of the samples fell at or below that
number.  It works like median rather than like mean.  The advantage of
such a measure is it allows you to assert something like "90% of
requests were handled in x amount of time".  With an average, you can
make no such assertion about what users experience - you have no handle
on what users experience during busy periods.

-Mike

On Mon, 2005-10-10 at 12:15 -0700, m mat wrote:
> I want to understand the explaination of the 90% line in aggregate report. 
>  
> What I think it should be: Average of samples with 90% significance, i.e. if you assume a bell shaped distribution around the mean, this number should be an average of the middle 90% numbers (as in average after top 5% and bottom 5% outliers are removed). So that this number could be inferred as "the response time 90% of your users are likely to see". For example if you have 10 samples 1,2,3,4,5,6,7,8,9,10. 90th percentile should be (2+3+4+5+6+7+8+9)/8 = 5.5
>  
> it seems to me JMeter reports this number as the 90% value in the sample set. So for the above sample set JMeter would report 9
>  
> Which one of the above two is correct for JMeter?
> Matt
> 
> 		
> ---------------------------------
>  Yahoo! Music Unlimited - Access over 1 million songs. Try it free.


---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org