Failed samples and aggregate report

[Vladimir Sitnikov <si...@gmail.com>]

Hi,

I believe JMeter support of tracking failed samples can be
significantly improved.
Each and every manual suggests using "Aggregate Report" ([1]) or
friends, however it silently returns wrong data without any warning.

For instance: (sample, response time, status)
Sample1, 9 sec, OK
Sample1, 0 sec, ERR
Sample1, 0 sec, ERR

JMeter would show "average response time" as 3 seconds, and same for
percentiles (median should be 9 seconds, not 0).
However, it is common for failed requests to run much faster.
It does not matter very much how fast you can crash, but it does
matter how fast the successful responses are.

I see two problems here:
1) Default configuration averages/computes quantiles for both OK and
ERR responses. So users get wrong values in the report.
2) There is no easy way to track OK and ERR separately. Well, one can
add _two_ copies of AggregateReport (one for success, another one for
fails), however is that ever suggested in the documentation? Is that a
good user experience? One would have to switch back and forth from one
to another.

Any ideas/plans to fix that?

Well, I do not suggest implementing full set of reporting yet, but it
is a pity to see how performance engineers report wrong results unless
you pinpoint all those JMeter pitfalls.

[1]: http://jmeter.apache.org/usermanual/component_reference.html#Aggregate_Report

-- 
Regards,
Vladimir Sitnikov

Re: Failed samples and aggregate report

[Vladimir Sitnikov <si...@gmail.com>]

> Can you open a bugzilla, should be easy to fix

I've filed it as https://issues.apache.org/bugzilla/show_bug.cgi?id=57545

>- https://issues.apache.org/bugzilla/show_bug.cgi?id=57516

This looks nice!
Vladimir

Re: Failed samples and aggregate report

[Philippe Mouawad <ph...@gmail.com>]

Hi Vladimir,

On Fri, Feb 6, 2015 at 1:38 PM, Vladimir Sitnikov <
sitnikov.vladimir@gmail.com> wrote:

> Hi,
>
> I believe JMeter support of tracking failed samples can be
> significantly improved.
> Each and every manual suggests using "Aggregate Report" ([1]) or
> friends, however it silently returns wrong data without any warning.
>
> For instance: (sample, response time, status)
> Sample1, 9 sec, OK
> Sample1, 0 sec, ERR
> Sample1, 0 sec, ERR
>
> JMeter would show "average response time" as 3 seconds, and same for
> percentiles (median should be 9 seconds, not 0).
> However, it is common for failed requests to run much faster.
> It does not matter very much how fast you can crash, but it does
> matter how fast the successful responses are.
>
> I see two problems here:
> 1) Default configuration averages/computes quantiles for both OK and
> ERR responses. So users get wrong values in the report.
>

Can you open a bugzilla, should be easy to fix

> 2) There is no easy way to track OK and ERR separately. Well, one can
> add _two_ copies of AggregateReport (one for success, another one for
> fails), however is that ever suggested in the documentation? Is that a
> good user experience? One would have to switch back and forth from one
> to another.
>
> Any ideas/plans to fix that?
>
Have a look at that and give us your feedback. Your code contributions are
also very welcome :
- https://issues.apache.org/bugzilla/show_bug.cgi?id=57516


> Well, I do not suggest implementing full set of reporting yet, but it
> is a pity to see how performance engineers report wrong results unless
> you pinpoint all those JMeter pitfalls.
>
> [1]:
> http://jmeter.apache.org/usermanual/component_reference.html#Aggregate_Report
>
>
Not sure this can only be fixed by JMeter :-)

> --
> Regards,
> Vladimir Sitnikov
>



-- 
Cordialement.
Philippe Mouawad.