You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Malith Jayasinghe <ma...@gmail.com> on 2017/11/23 04:04:02 UTC
Issues in JMETER Dashboard Latency Percentile Calculations
Dear All,
We have been using JMETER extensively for performance testing.
All our performance reports are generated using the JMETER Dashboard.
Recently, we have found out that there is an issue in
the JMETER dashboard's *latency percentile calculation. *
There is a significant difference in the actual percentile values and the
values shown in the dashboard. We have verified this using "R" and in fact,
the values shown the JMETER aggregate report are correct. The values that
appear in the JMETER Dashboard are not.
We understand that JMETER Dashboard may be using a different algorithm
(approximation) for percentile calculation. However, the problem is that
that latency percentiles values calculated using this algorithm are not
accurate (at least for certain scenarios).
We have had a case where the Dashboard showing a 95% percentile value 1000
ms. However, the correct value was 321 ms.
The accuracy of percentile values are of utmost importance to us and we
would kindly ask you to look into this issue and release a patch if
possible.
If you can implement the same algorithm used in the aggregate report within
the dashboard that will resolve this issue.
Thanks
Malith Jayasinghe
Re: Issues in JMETER Dashboard Latency Percentile Calculations
Posted by Malith Jayasinghe <ma...@gmail.com>.
Thanks a lot Philippe. We will check on this and get back.
On Fri, Nov 24, 2017 at 4:02 AM, Philippe Mouawad <
philippe.mouawad@gmail.com> wrote:
> Hello,
> You can adjust accuracy by setting in user.properties:
>
> - jmeter.reportgenerator.statistic_window = 20000
>
> Caution : higher value provides a better accuracy but needs more memory.
>
> This is not a bug.
>
> We use DescriptiveStatistics from commons-math with a sliding window.
>
> So if after modifying this value you still face issue, please report and
> open a bug at commons-math.
>
> Regards
>
>
> On Thu, Nov 23, 2017 at 5:04 AM, Malith Jayasinghe <
> malith.jayasinghe@gmail.com> wrote:
>
> > Dear All,
> >
> > We have been using JMETER extensively for performance testing.
> >
> > All our performance reports are generated using the JMETER Dashboard.
> >
> > Recently, we have found out that there is an issue in
> > the JMETER dashboard's *latency percentile calculation. *
> >
> > There is a significant difference in the actual percentile values and the
> > values shown in the dashboard. We have verified this using "R" and in
> fact,
> > the values shown the JMETER aggregate report are correct. The values that
> > appear in the JMETER Dashboard are not.
> >
> > We understand that JMETER Dashboard may be using a different algorithm
> > (approximation) for percentile calculation. However, the problem is that
> > that latency percentiles values calculated using this algorithm are not
> > accurate (at least for certain scenarios).
> >
> > We have had a case where the Dashboard showing a 95% percentile value
> 1000
> > ms. However, the correct value was 321 ms.
> >
> > The accuracy of percentile values are of utmost importance to us and we
> > would kindly ask you to look into this issue and release a patch if
> > possible.
> >
> > If you can implement the same algorithm used in the aggregate report
> within
> > the dashboard that will resolve this issue.
> >
> >
> > Thanks
> >
> > Malith Jayasinghe
> >
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
Re: Issues in JMETER Dashboard Latency Percentile Calculations
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello,
What about contributing a documentation PR based on your blog in this doc:
- http://jmeter.apache.org/usermanual/generating-dashboard.html
Which is generated from :
xdocs/usermanual/generating-dashboard.xml
It would be nice.
Thanks
On Monday, December 11, 2017, Malith Jayasinghe <ma...@gmail.com>
wrote:
> A detailed discussion of this issue can be found in here
> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-
> percenti
>
> On Wed, Nov 29, 2017 at 5:43 AM, Malith Jayasinghe <
> malith.jayasinghe@gmail.com> wrote:
>
>> .Hi Phiilppe,
>>
>> No, I did not create this issue. There is a significant increase in
>> the accuracy when using "large" window sizes.
>>
>> Thanks
>>
>> Malith
>>
>>
>> On Wed, Nov 29, 2017 at 3:53 AM, Philippe Mouawad <
>> philippe.mouawad@gmail.com> wrote:
>>
>>> Hello,
>>> Any feedback on this ?
>>> Are you the opener of this ?:
>>>
>>> - https://bz.apache.org/bugzilla/show_bug.cgi?id=61725
>>>
>>> Thanks
>>>
>>> On Thu, Nov 23, 2017 at 11:32 PM, Philippe Mouawad <
>>> philippe.mouawad@gmail.com> wrote:
>>>
>>> > Hello,
>>> > You can adjust accuracy by setting in user.properties:
>>> >
>>> > - jmeter.reportgenerator.statistic_window = 20000
>>> >
>>> > Caution : higher value provides a better accuracy but needs more
>>> memory.
>>> >
>>> > This is not a bug.
>>> >
>>> > We use DescriptiveStatistics from commons-math with a sliding window.
>>> >
>>> > So if after modifying this value you still face issue, please report
>>> and
>>> > open a bug at commons-math.
>>> >
>>> > Regards
>>> >
>>> >
>>> > On Thu, Nov 23, 2017 at 5:04 AM, Malith Jayasinghe <
>>> > malith.jayasinghe@gmail.com> wrote:
>>> >
>>> >> Dear All,
>>> >>
>>> >> We have been using JMETER extensively for performance testing.
>>> >>
>>> >> All our performance reports are generated using the JMETER Dashboard.
>>> >>
>>> >> Recently, we have found out that there is an issue in
>>> >> the JMETER dashboard's *latency percentile calculation. *
>>> >>
>>> >> There is a significant difference in the actual percentile values and
>>> the
>>> >> values shown in the dashboard. We have verified this using "R" and in
>>> >> fact,
>>> >> the values shown the JMETER aggregate report are correct. The values
>>> that
>>> >> appear in the JMETER Dashboard are not.
>>> >>
>>> >> We understand that JMETER Dashboard may be using a different algorithm
>>> >> (approximation) for percentile calculation. However, the problem is
>>> that
>>> >> that latency percentiles values calculated using this algorithm are
>>> not
>>> >> accurate (at least for certain scenarios).
>>> >>
>>> >> We have had a case where the Dashboard showing a 95% percentile value
>>> 1000
>>> >> ms. However, the correct value was 321 ms.
>>> >>
>>> >> The accuracy of percentile values are of utmost importance to us and
>>> we
>>> >> would kindly ask you to look into this issue and release a patch if
>>> >> possible.
>>> >>
>>> >> If you can implement the same algorithm used in the aggregate report
>>> >> within
>>> >> the dashboard that will resolve this issue.
>>> >>
>>> >>
>>> >> Thanks
>>> >>
>>> >> Malith Jayasinghe
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Cordialement.
>>> > Philippe Mouawad.
>>> >
>>> >
>>> >
>>>
>>>
>>> --
>>> Cordialement.
>>> Philippe Mouawad.
>>>
>>
>>
>
--
Cordialement.
Philippe Mouawad.
Re: Issues in JMETER Dashboard Latency Percentile Calculations
Posted by Malith Jayasinghe <ma...@gmail.com>.
A detailed discussion of this issue can be found in here
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
On Wed, Nov 29, 2017 at 5:43 AM, Malith Jayasinghe <
malith.jayasinghe@gmail.com> wrote:
> .Hi Phiilppe,
>
> No, I did not create this issue. There is a significant increase in the
> accuracy when using "large" window sizes.
>
> Thanks
>
> Malith
>
>
> On Wed, Nov 29, 2017 at 3:53 AM, Philippe Mouawad <
> philippe.mouawad@gmail.com> wrote:
>
>> Hello,
>> Any feedback on this ?
>> Are you the opener of this ?:
>>
>> - https://bz.apache.org/bugzilla/show_bug.cgi?id=61725
>>
>> Thanks
>>
>> On Thu, Nov 23, 2017 at 11:32 PM, Philippe Mouawad <
>> philippe.mouawad@gmail.com> wrote:
>>
>> > Hello,
>> > You can adjust accuracy by setting in user.properties:
>> >
>> > - jmeter.reportgenerator.statistic_window = 20000
>> >
>> > Caution : higher value provides a better accuracy but needs more memory.
>> >
>> > This is not a bug.
>> >
>> > We use DescriptiveStatistics from commons-math with a sliding window.
>> >
>> > So if after modifying this value you still face issue, please report and
>> > open a bug at commons-math.
>> >
>> > Regards
>> >
>> >
>> > On Thu, Nov 23, 2017 at 5:04 AM, Malith Jayasinghe <
>> > malith.jayasinghe@gmail.com> wrote:
>> >
>> >> Dear All,
>> >>
>> >> We have been using JMETER extensively for performance testing.
>> >>
>> >> All our performance reports are generated using the JMETER Dashboard.
>> >>
>> >> Recently, we have found out that there is an issue in
>> >> the JMETER dashboard's *latency percentile calculation. *
>> >>
>> >> There is a significant difference in the actual percentile values and
>> the
>> >> values shown in the dashboard. We have verified this using "R" and in
>> >> fact,
>> >> the values shown the JMETER aggregate report are correct. The values
>> that
>> >> appear in the JMETER Dashboard are not.
>> >>
>> >> We understand that JMETER Dashboard may be using a different algorithm
>> >> (approximation) for percentile calculation. However, the problem is
>> that
>> >> that latency percentiles values calculated using this algorithm are not
>> >> accurate (at least for certain scenarios).
>> >>
>> >> We have had a case where the Dashboard showing a 95% percentile value
>> 1000
>> >> ms. However, the correct value was 321 ms.
>> >>
>> >> The accuracy of percentile values are of utmost importance to us and we
>> >> would kindly ask you to look into this issue and release a patch if
>> >> possible.
>> >>
>> >> If you can implement the same algorithm used in the aggregate report
>> >> within
>> >> the dashboard that will resolve this issue.
>> >>
>> >>
>> >> Thanks
>> >>
>> >> Malith Jayasinghe
>> >>
>> >
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>> >
>> >
>> >
>>
>>
>> --
>> Cordialement.
>> Philippe Mouawad.
>>
>
>
Re: Issues in JMETER Dashboard Latency Percentile Calculations
Posted by Malith Jayasinghe <ma...@gmail.com>.
.Hi Phiilppe,
No, I did not create this issue. There is a significant increase in the
accuracy when using "large" window sizes.
Thanks
Malith
On Wed, Nov 29, 2017 at 3:53 AM, Philippe Mouawad <
philippe.mouawad@gmail.com> wrote:
> Hello,
> Any feedback on this ?
> Are you the opener of this ?:
>
> - https://bz.apache.org/bugzilla/show_bug.cgi?id=61725
>
> Thanks
>
> On Thu, Nov 23, 2017 at 11:32 PM, Philippe Mouawad <
> philippe.mouawad@gmail.com> wrote:
>
> > Hello,
> > You can adjust accuracy by setting in user.properties:
> >
> > - jmeter.reportgenerator.statistic_window = 20000
> >
> > Caution : higher value provides a better accuracy but needs more memory.
> >
> > This is not a bug.
> >
> > We use DescriptiveStatistics from commons-math with a sliding window.
> >
> > So if after modifying this value you still face issue, please report and
> > open a bug at commons-math.
> >
> > Regards
> >
> >
> > On Thu, Nov 23, 2017 at 5:04 AM, Malith Jayasinghe <
> > malith.jayasinghe@gmail.com> wrote:
> >
> >> Dear All,
> >>
> >> We have been using JMETER extensively for performance testing.
> >>
> >> All our performance reports are generated using the JMETER Dashboard.
> >>
> >> Recently, we have found out that there is an issue in
> >> the JMETER dashboard's *latency percentile calculation. *
> >>
> >> There is a significant difference in the actual percentile values and
> the
> >> values shown in the dashboard. We have verified this using "R" and in
> >> fact,
> >> the values shown the JMETER aggregate report are correct. The values
> that
> >> appear in the JMETER Dashboard are not.
> >>
> >> We understand that JMETER Dashboard may be using a different algorithm
> >> (approximation) for percentile calculation. However, the problem is that
> >> that latency percentiles values calculated using this algorithm are not
> >> accurate (at least for certain scenarios).
> >>
> >> We have had a case where the Dashboard showing a 95% percentile value
> 1000
> >> ms. However, the correct value was 321 ms.
> >>
> >> The accuracy of percentile values are of utmost importance to us and we
> >> would kindly ask you to look into this issue and release a patch if
> >> possible.
> >>
> >> If you can implement the same algorithm used in the aggregate report
> >> within
> >> the dashboard that will resolve this issue.
> >>
> >>
> >> Thanks
> >>
> >> Malith Jayasinghe
> >>
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
> >
> >
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
Re: Issues in JMETER Dashboard Latency Percentile Calculations
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello,
Any feedback on this ?
Are you the opener of this ?:
- https://bz.apache.org/bugzilla/show_bug.cgi?id=61725
Thanks
On Thu, Nov 23, 2017 at 11:32 PM, Philippe Mouawad <
philippe.mouawad@gmail.com> wrote:
> Hello,
> You can adjust accuracy by setting in user.properties:
>
> - jmeter.reportgenerator.statistic_window = 20000
>
> Caution : higher value provides a better accuracy but needs more memory.
>
> This is not a bug.
>
> We use DescriptiveStatistics from commons-math with a sliding window.
>
> So if after modifying this value you still face issue, please report and
> open a bug at commons-math.
>
> Regards
>
>
> On Thu, Nov 23, 2017 at 5:04 AM, Malith Jayasinghe <
> malith.jayasinghe@gmail.com> wrote:
>
>> Dear All,
>>
>> We have been using JMETER extensively for performance testing.
>>
>> All our performance reports are generated using the JMETER Dashboard.
>>
>> Recently, we have found out that there is an issue in
>> the JMETER dashboard's *latency percentile calculation. *
>>
>> There is a significant difference in the actual percentile values and the
>> values shown in the dashboard. We have verified this using "R" and in
>> fact,
>> the values shown the JMETER aggregate report are correct. The values that
>> appear in the JMETER Dashboard are not.
>>
>> We understand that JMETER Dashboard may be using a different algorithm
>> (approximation) for percentile calculation. However, the problem is that
>> that latency percentiles values calculated using this algorithm are not
>> accurate (at least for certain scenarios).
>>
>> We have had a case where the Dashboard showing a 95% percentile value 1000
>> ms. However, the correct value was 321 ms.
>>
>> The accuracy of percentile values are of utmost importance to us and we
>> would kindly ask you to look into this issue and release a patch if
>> possible.
>>
>> If you can implement the same algorithm used in the aggregate report
>> within
>> the dashboard that will resolve this issue.
>>
>>
>> Thanks
>>
>> Malith Jayasinghe
>>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>
--
Cordialement.
Philippe Mouawad.
Re: Issues in JMETER Dashboard Latency Percentile Calculations
Posted by Philippe Mouawad <ph...@gmail.com>.
Hello,
You can adjust accuracy by setting in user.properties:
- jmeter.reportgenerator.statistic_window = 20000
Caution : higher value provides a better accuracy but needs more memory.
This is not a bug.
We use DescriptiveStatistics from commons-math with a sliding window.
So if after modifying this value you still face issue, please report and
open a bug at commons-math.
Regards
On Thu, Nov 23, 2017 at 5:04 AM, Malith Jayasinghe <
malith.jayasinghe@gmail.com> wrote:
> Dear All,
>
> We have been using JMETER extensively for performance testing.
>
> All our performance reports are generated using the JMETER Dashboard.
>
> Recently, we have found out that there is an issue in
> the JMETER dashboard's *latency percentile calculation. *
>
> There is a significant difference in the actual percentile values and the
> values shown in the dashboard. We have verified this using "R" and in fact,
> the values shown the JMETER aggregate report are correct. The values that
> appear in the JMETER Dashboard are not.
>
> We understand that JMETER Dashboard may be using a different algorithm
> (approximation) for percentile calculation. However, the problem is that
> that latency percentiles values calculated using this algorithm are not
> accurate (at least for certain scenarios).
>
> We have had a case where the Dashboard showing a 95% percentile value 1000
> ms. However, the correct value was 321 ms.
>
> The accuracy of percentile values are of utmost importance to us and we
> would kindly ask you to look into this issue and release a patch if
> possible.
>
> If you can implement the same algorithm used in the aggregate report within
> the dashboard that will resolve this issue.
>
>
> Thanks
>
> Malith Jayasinghe
>
--
Cordialement.
Philippe Mouawad.