Re: Incorrect 90th percentile values in JMeter dashboard HTML report

[A P <ab...@gmail.com>]

@Philippe
I tried different values like 200000, 2000000 for
"jmeter.reportgenerator.statistic_window"
But it doesn't work.

On Sat, Mar 6, 2021 at 2:26 AM Philippe Mouawad <ph...@gmail.com>
wrote:

> -1 does not work
> Did you try increasing it ?
>
> Thanks
>
> On Friday, March 5, 2021, A P <ab...@gmail.com> wrote:
>
>> Yes, I tried " jmeter.reportgenerator.statistic_window = -1"
>> But it does not work.
>>
>> On Fri, Mar 5, 2021 at 11:46 PM Philippe Mouawad <
>> philippe.mouawad@gmail.com> wrote:
>>
>>> Hello,
>>> Have you tried increasing this value in user.properties which defaults
>>> to 20000
>>>
>>>
>>>    - jmeter.reportgenerator.statistic_window
>>>
>>>
>>> See:
>>>
>>>
>>>    -
>>>    https://jmeter.apache.org/usermanual/properties_reference.html#reporting
>>>
>>>
>>> Thanks
>>>
>>> Regards
>>>
>>> Philippe M.
>>>
>>> https://leanpub.com/master-jmeter-from-load-test-to-devops
>>>
>>> On Fri, Mar 5, 2021 at 7:12 PM A P <ab...@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I'm eagerly waiting for solution to this issue. If anyone has a fix
>>>> please reply back.
>>>>
>>>> AP
>>>>
>>>> On Sun, Feb 28, 2021 at 11:59 PM A P <ab...@gmail.com> wrote:
>>>>
>>>>> @ Mariusz
>>>>> Yes.
>>>>>
>>>>> On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <ma...@gmail.com> wrote:
>>>>>
>>>>>> @Abhitosh
>>>>>> To be sure: So you loaded manually JTL file in Aggregate Report
>>>>>> listener ( without CMDRunner ) and compared the results with html report?
>>>>>>
>>>>>> Mariusz
>>>>>>
>>>>>> On Sun, 28 Feb 2021 at 12:27, A P <ab...@gmail.com> wrote:
>>>>>>
>>>>>>> *@Mariusz*
>>>>>>> As per below post, HTML report has wrong values:
>>>>>>>
>>>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>>>
>>>>>>> I automated aggregate report creation using JMeter plugin:
>>>>>>> *CMDRunner*
>>>>>>>
>>>>>>> Also I tried creating HTML and Aggregate reports manually using a
>>>>>>> JTL file. But the results are showing the same difference.
>>>>>>>
>>>>>>> My suggestion to reproduce this issue is:
>>>>>>> Execute a long time JMeter load test with a considerable number of
>>>>>>> samples. Then compare HTML and aggregate reports for the test.
>>>>>>> You can see the difference between them.
>>>>>>>
>>>>>>> *@Felix*
>>>>>>> I saw the post: https://github.com/apache/jmeter/pull/380
>>>>>>> But not getting a clear idea how to implement this.
>>>>>>>
>>>>>>> On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <
>>>>>>> felix.schumacher@internetallee.de> wrote:
>>>>>>>
>>>>>>>> Hi Abhitosh,
>>>>>>>>
>>>>>>>> it would have been a good idea to share sample data, that shows
>>>>>>>> that behaviour.
>>>>>>>>
>>>>>>>> I think you stumbled upon
>>>>>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=61130
>>>>>>>>
>>>>>>>> One problem is, that JMeter uses two different algorithms to
>>>>>>>> calculate a percentile. One for the report and one for the dashboard. They
>>>>>>>> differ in the point, which they use to display the percentile.
>>>>>>>>
>>>>>>>> We did some work to use the same algorithm for both components in
>>>>>>>> https://github.com/apache/jmeter/pull/380 which got closed when
>>>>>>>> converting our repo from subversion to git.
>>>>>>>>
>>>>>>>> In my eyes, that big discrepancy in your values shows, that you are
>>>>>>>> having bad data in the first place. But of course, it would be nice, if
>>>>>>>> JMeter would show consistent values, or even better an indicator of how
>>>>>>>> close the value is to its neighbours.
>>>>>>>>
>>>>>>>> Felix
>>>>>>>> Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
>>>>>>>>
>>>>>>>> @Kieran Lynch
>>>>>>>> I'm using JMeter 5.4
>>>>>>>> I already tried "jmeter.reportgenerator.statistic_window = -1". But
>>>>>>>> it doesn't work.
>>>>>>>>
>>>>>>>> @Mariusz W
>>>>>>>> In my case it's a very huge difference (> 40 seconds) which is not
>>>>>>>> acceptable at all as there are client facing reports.
>>>>>>>>
>>>>>>>> @All
>>>>>>>> I had a discussion with my colleague. As per him there is
>>>>>>>> difference in formula how Aggregate report and HTMl report calculates 90th
>>>>>>>> percentile.
>>>>>>>> Aggregate report taking 5th value out of 6 as 90 th percentile. But
>>>>>>>> HTML report is picking 6th value (which is the highest one) as 90 th
>>>>>>>> percentile.
>>>>>>>> I need to figure it out ASAP now.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes, I have seen this small discrepancies also in my tests but I
>>>>>>>>> don't adhere to exact values and this reports are just first step during
>>>>>>>>> analysis to find an anomaly or something else to focus on , e.g. Is it
>>>>>>>>> worth to look at the mean when data is not symmetrically distributed. If it
>>>>>>>>> is not symmetrical should I care about standard deviation at all. If my
>>>>>>>>> histogram is strange - what was wrong.
>>>>>>>>> But surely the meaning of the report depends on the client's need:)
>>>>>>>>>
>>>>>>>>> Mariusz
>>>>>>>>>
>>>>>>>>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Just chiming in with a higher concurrency test example - not as
>>>>>>>>>> severe but still a difference.
>>>>>>>>>>
>>>>>>>>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to
>>>>>>>>>> me, however. I can give that a go and see if it improves things, but these
>>>>>>>>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>>>>>>>>> and are a bit of a nuisance as they essentially limit me to the output
>>>>>>>>>> format I choose to share with clients...
>>>>>>>>>>
>>>>>>>>>> HTML Output:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Aggregate Report:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Kieran Lynch
>>>>>>>>>>
>>>>>>>>>> WM Reply
>>>>>>>>>> Nova South
>>>>>>>>>> 160 Victoria Street, Westminster
>>>>>>>>>> London SW1E 5LB - UK
>>>>>>>>>> phone: +44 (0)20 7730 6000
>>>>>>>>>> k.lynch@reply.com
>>>>>>>>>> www.reply.com
>>>>>>>>>>
>>>>>>>>>> [image: WM Reply]
>>>>>>>>>> ------------------------------
>>>>>>>>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>>>>>>>>> *Sent:* 26 February 2021 14:28
>>>>>>>>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>>>>>>>>> *Subject:* Re: Incorrect 90th percentile values in JMeter
>>>>>>>>>> dashboard HTML report
>>>>>>>>>>
>>>>>>>>>> This is just an example. Actual test was executed with thousands
>>>>>>>>>> of samples with different transactions.
>>>>>>>>>> It's a transaction controller.
>>>>>>>>>> Also I'm using "Result1.csv" format only.
>>>>>>>>>> I have automated Aggregate report and HTML report creation.
>>>>>>>>>> Manual calculations are not allowed.
>>>>>>>>>>
>>>>>>>>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> Yes, indeed results are strange... You have only 6 samples and
>>>>>>>>>> very wide distribution. Did You try to get more samples? I think that there
>>>>>>>>>> is too small amount of samples to do any descriptive statistics and draw
>>>>>>>>>> conclusions about test results and system. What is this sampler in report?
>>>>>>>>>> Is it http sampler or Transaction Controller? You can always make your own
>>>>>>>>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>>>>>>>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>>>>>>>>> however.
>>>>>>>>>>
>>>>>>>>>> Mariusz
>>>>>>>>>>
>>>>>>>>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <
>>>>>>>>>> abhitosh.qa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I'm generating HTML dashboard report and aggregate report during
>>>>>>>>>> a load test.
>>>>>>>>>> There is a huge difference in 90th percentile values in aggregate
>>>>>>>>>> report and HTML report.
>>>>>>>>>> It seems that HTML dashboard report is showing incorrect values.
>>>>>>>>>> Can someone help me on this?
>>>>>>>>>>
>>>>>>>>>> Screenshots:
>>>>>>>>>> Aggregate report (90th percentile is 123):
>>>>>>>>>> [image: image.png]
>>>>>>>>>>
>>>>>>>>>> HTML report (90th percentile is 44266):
>>>>>>>>>> [image: image.png]
>>>>>>>>>>
>>>>>>>>>> I tried the solution provided in the post below. But it didn't
>>>>>>>>>> work:
>>>>>>>>>>
>>>>>>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> AP
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>>>>>
>>>>>>>>>
>>>
>>> --
>>> Cordialement.
>>> Philippe Mouawad.
>>>
>>>
>>>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>
>