You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Abhitosh Patil <ab...@gmail.com> on 2021/02/26 13:15:18 UTC

Incorrect 90th percentile values in JMeter dashboard HTML report

Hi,

I'm generating HTML dashboard report and aggregate report during a load
test.
There is a huge difference in 90th percentile values in aggregate report
and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
[image: image.png]

HTML report (90th percentile is 44266):
[image: image.png]

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by Barcode S K <ba...@gmail.com>.
I've also had this problem for a while now. The percentile values never
come out correct as my results files grow. My results file usually grows to
over 3 GBs.

As a sample, I'm sharing a stripped down version of my results file
containing a little over 207k samples. I usually validate numbers using
awk. I wrote a Python script ("py_calc.py") which does the same thing, to
share with you guys. It looks simpler than the awk command I run.

From the script, these are the numbers I get for the given results file:

D:\> python py_calc.py mytxn.csv MyTXN
File to be analyzed: mytxn.csv
207557 lines in the file
207556 lines contain the sample
90th index: 186800
90th percentile (ms): 1868
Next value after 90th: 1868
Average (ms): 1049.18

So, I know that my 90th percentile should be 1868 ms. JMeter never reports
it correctly with the default reportgenerator values. I've tried with
JMeter 5.2 and 5.4.1.

90th percentile value with:
statistic_window default: 1783.90
statistic_window 60k: 1839.00
statistic_window 120k: 1900.00
statistic_window 200k: 1884.00
statistic_window 207556 (number of matching samples in my file): 1868.00
statistic_window 240k: 1868.00

So, to get the accurate number, my statistic_window has to be larger than
or equal to the number of matching samples. Is the need to utilize less
memory- as the reportgenerator.properties files says:

> # Caution : higher value provides a better accuracy but needs more memory.

really worth the inaccurate percentile numbers? I don't suppose load
generators nowadays are equipped with small memory modules.

-SK
PS: Unable to attach results CSV; file size over 2 MB even after
compression.

On Sat, Mar 6, 2021 at 2:26 AM Philippe Mouawad <ph...@gmail.com>
wrote:

> -1 does not work
> Did you try increasing it ?
>
> Thanks
>
> On Friday, March 5, 2021, A P <ab...@gmail.com> wrote:
>
>> Yes, I tried " jmeter.reportgenerator.statistic_window = -1"
>> But it does not work.
>>
>> On Fri, Mar 5, 2021 at 11:46 PM Philippe Mouawad <
>> philippe.mouawad@gmail.com> wrote:
>>
>>> Hello,
>>> Have you tried increasing this value in user.properties which defaults
>>> to 20000
>>>
>>>
>>>    - jmeter.reportgenerator.statistic_window
>>>
>>>
>>> See:
>>>
>>>
>>>    -
>>>    https://jmeter.apache.org/usermanual/properties_reference.html#reporting
>>>
>>>
>>> Thanks
>>>
>>> Regards
>>>
>>> Philippe M.
>>>
>>> https://leanpub.com/master-jmeter-from-load-test-to-devops
>>>
>>> On Fri, Mar 5, 2021 at 7:12 PM A P <ab...@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I'm eagerly waiting for solution to this issue. If anyone has a fix
>>>> please reply back.
>>>>
>>>> AP
>>>>
>>>> On Sun, Feb 28, 2021 at 11:59 PM A P <ab...@gmail.com> wrote:
>>>>
>>>>> @ Mariusz
>>>>> Yes.
>>>>>
>>>>> On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <ma...@gmail.com> wrote:
>>>>>
>>>>>> @Abhitosh
>>>>>> To be sure: So you loaded manually JTL file in Aggregate Report
>>>>>> listener ( without CMDRunner ) and compared the results with html report?
>>>>>>
>>>>>> Mariusz
>>>>>>
>>>>>> On Sun, 28 Feb 2021 at 12:27, A P <ab...@gmail.com> wrote:
>>>>>>
>>>>>>> *@Mariusz*
>>>>>>> As per below post, HTML report has wrong values:
>>>>>>>
>>>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>>>
>>>>>>> I automated aggregate report creation using JMeter plugin:
>>>>>>> *CMDRunner*
>>>>>>>
>>>>>>> Also I tried creating HTML and Aggregate reports manually using a
>>>>>>> JTL file. But the results are showing the same difference.
>>>>>>>
>>>>>>> My suggestion to reproduce this issue is:
>>>>>>> Execute a long time JMeter load test with a considerable number of
>>>>>>> samples. Then compare HTML and aggregate reports for the test.
>>>>>>> You can see the difference between them.
>>>>>>>
>>>>>>> *@Felix*
>>>>>>> I saw the post: https://github.com/apache/jmeter/pull/380
>>>>>>> But not getting a clear idea how to implement this.
>>>>>>>
>>>>>>> On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <
>>>>>>> felix.schumacher@internetallee.de> wrote:
>>>>>>>
>>>>>>>> Hi Abhitosh,
>>>>>>>>
>>>>>>>> it would have been a good idea to share sample data, that shows
>>>>>>>> that behaviour.
>>>>>>>>
>>>>>>>> I think you stumbled upon
>>>>>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=61130
>>>>>>>>
>>>>>>>> One problem is, that JMeter uses two different algorithms to
>>>>>>>> calculate a percentile. One for the report and one for the dashboard. They
>>>>>>>> differ in the point, which they use to display the percentile.
>>>>>>>>
>>>>>>>> We did some work to use the same algorithm for both components in
>>>>>>>> https://github.com/apache/jmeter/pull/380 which got closed when
>>>>>>>> converting our repo from subversion to git.
>>>>>>>>
>>>>>>>> In my eyes, that big discrepancy in your values shows, that you are
>>>>>>>> having bad data in the first place. But of course, it would be nice, if
>>>>>>>> JMeter would show consistent values, or even better an indicator of how
>>>>>>>> close the value is to its neighbours.
>>>>>>>>
>>>>>>>> Felix
>>>>>>>> Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
>>>>>>>>
>>>>>>>> @Kieran Lynch
>>>>>>>> I'm using JMeter 5.4
>>>>>>>> I already tried "jmeter.reportgenerator.statistic_window = -1". But
>>>>>>>> it doesn't work.
>>>>>>>>
>>>>>>>> @Mariusz W
>>>>>>>> In my case it's a very huge difference (> 40 seconds) which is not
>>>>>>>> acceptable at all as there are client facing reports.
>>>>>>>>
>>>>>>>> @All
>>>>>>>> I had a discussion with my colleague. As per him there is
>>>>>>>> difference in formula how Aggregate report and HTMl report calculates 90th
>>>>>>>> percentile.
>>>>>>>> Aggregate report taking 5th value out of 6 as 90 th percentile. But
>>>>>>>> HTML report is picking 6th value (which is the highest one) as 90 th
>>>>>>>> percentile.
>>>>>>>> I need to figure it out ASAP now.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes, I have seen this small discrepancies also in my tests but I
>>>>>>>>> don't adhere to exact values and this reports are just first step during
>>>>>>>>> analysis to find an anomaly or something else to focus on , e.g. Is it
>>>>>>>>> worth to look at the mean when data is not symmetrically distributed. If it
>>>>>>>>> is not symmetrical should I care about standard deviation at all. If my
>>>>>>>>> histogram is strange - what was wrong.
>>>>>>>>> But surely the meaning of the report depends on the client's need:)
>>>>>>>>>
>>>>>>>>> Mariusz
>>>>>>>>>
>>>>>>>>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Just chiming in with a higher concurrency test example - not as
>>>>>>>>>> severe but still a difference.
>>>>>>>>>>
>>>>>>>>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to
>>>>>>>>>> me, however. I can give that a go and see if it improves things, but these
>>>>>>>>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>>>>>>>>> and are a bit of a nuisance as they essentially limit me to the output
>>>>>>>>>> format I choose to share with clients...
>>>>>>>>>>
>>>>>>>>>> HTML Output:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Aggregate Report:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Kieran Lynch
>>>>>>>>>>
>>>>>>>>>> WM Reply
>>>>>>>>>> Nova South
>>>>>>>>>> 160 Victoria Street, Westminster
>>>>>>>>>> London SW1E 5LB - UK
>>>>>>>>>> phone: +44 (0)20 7730 6000
>>>>>>>>>> k.lynch@reply.com
>>>>>>>>>> www.reply.com
>>>>>>>>>>
>>>>>>>>>> [image: WM Reply]
>>>>>>>>>> ------------------------------
>>>>>>>>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>>>>>>>>> *Sent:* 26 February 2021 14:28
>>>>>>>>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>>>>>>>>> *Subject:* Re: Incorrect 90th percentile values in JMeter
>>>>>>>>>> dashboard HTML report
>>>>>>>>>>
>>>>>>>>>> This is just an example. Actual test was executed with thousands
>>>>>>>>>> of samples with different transactions.
>>>>>>>>>> It's a transaction controller.
>>>>>>>>>> Also I'm using "Result1.csv" format only.
>>>>>>>>>> I have automated Aggregate report and HTML report creation.
>>>>>>>>>> Manual calculations are not allowed.
>>>>>>>>>>
>>>>>>>>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> Yes, indeed results are strange... You have only 6 samples and
>>>>>>>>>> very wide distribution. Did You try to get more samples? I think that there
>>>>>>>>>> is too small amount of samples to do any descriptive statistics and draw
>>>>>>>>>> conclusions about test results and system. What is this sampler in report?
>>>>>>>>>> Is it http sampler or Transaction Controller? You can always make your own
>>>>>>>>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>>>>>>>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>>>>>>>>> however.
>>>>>>>>>>
>>>>>>>>>> Mariusz
>>>>>>>>>>
>>>>>>>>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <
>>>>>>>>>> abhitosh.qa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I'm generating HTML dashboard report and aggregate report during
>>>>>>>>>> a load test.
>>>>>>>>>> There is a huge difference in 90th percentile values in aggregate
>>>>>>>>>> report and HTML report.
>>>>>>>>>> It seems that HTML dashboard report is showing incorrect values.
>>>>>>>>>> Can someone help me on this?
>>>>>>>>>>
>>>>>>>>>> Screenshots:
>>>>>>>>>> Aggregate report (90th percentile is 123):
>>>>>>>>>> [image: image.png]
>>>>>>>>>>
>>>>>>>>>> HTML report (90th percentile is 44266):
>>>>>>>>>> [image: image.png]
>>>>>>>>>>
>>>>>>>>>> I tried the solution provided in the post below. But it didn't
>>>>>>>>>> work:
>>>>>>>>>>
>>>>>>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> AP
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>>>>>
>>>>>>>>>
>>>
>>> --
>>> Cordialement.
>>> Philippe Mouawad.
>>>
>>>
>>>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>
>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by A P <ab...@gmail.com>.
@Philippe
I tried different values like 200000, 2000000 for
"jmeter.reportgenerator.statistic_window"
But it doesn't work.

On Sat, Mar 6, 2021 at 2:26 AM Philippe Mouawad <ph...@gmail.com>
wrote:

> -1 does not work
> Did you try increasing it ?
>
> Thanks
>
> On Friday, March 5, 2021, A P <ab...@gmail.com> wrote:
>
>> Yes, I tried " jmeter.reportgenerator.statistic_window = -1"
>> But it does not work.
>>
>> On Fri, Mar 5, 2021 at 11:46 PM Philippe Mouawad <
>> philippe.mouawad@gmail.com> wrote:
>>
>>> Hello,
>>> Have you tried increasing this value in user.properties which defaults
>>> to 20000
>>>
>>>
>>>    - jmeter.reportgenerator.statistic_window
>>>
>>>
>>> See:
>>>
>>>
>>>    -
>>>    https://jmeter.apache.org/usermanual/properties_reference.html#reporting
>>>
>>>
>>> Thanks
>>>
>>> Regards
>>>
>>> Philippe M.
>>>
>>> https://leanpub.com/master-jmeter-from-load-test-to-devops
>>>
>>> On Fri, Mar 5, 2021 at 7:12 PM A P <ab...@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I'm eagerly waiting for solution to this issue. If anyone has a fix
>>>> please reply back.
>>>>
>>>> AP
>>>>
>>>> On Sun, Feb 28, 2021 at 11:59 PM A P <ab...@gmail.com> wrote:
>>>>
>>>>> @ Mariusz
>>>>> Yes.
>>>>>
>>>>> On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <ma...@gmail.com> wrote:
>>>>>
>>>>>> @Abhitosh
>>>>>> To be sure: So you loaded manually JTL file in Aggregate Report
>>>>>> listener ( without CMDRunner ) and compared the results with html report?
>>>>>>
>>>>>> Mariusz
>>>>>>
>>>>>> On Sun, 28 Feb 2021 at 12:27, A P <ab...@gmail.com> wrote:
>>>>>>
>>>>>>> *@Mariusz*
>>>>>>> As per below post, HTML report has wrong values:
>>>>>>>
>>>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>>>
>>>>>>> I automated aggregate report creation using JMeter plugin:
>>>>>>> *CMDRunner*
>>>>>>>
>>>>>>> Also I tried creating HTML and Aggregate reports manually using a
>>>>>>> JTL file. But the results are showing the same difference.
>>>>>>>
>>>>>>> My suggestion to reproduce this issue is:
>>>>>>> Execute a long time JMeter load test with a considerable number of
>>>>>>> samples. Then compare HTML and aggregate reports for the test.
>>>>>>> You can see the difference between them.
>>>>>>>
>>>>>>> *@Felix*
>>>>>>> I saw the post: https://github.com/apache/jmeter/pull/380
>>>>>>> But not getting a clear idea how to implement this.
>>>>>>>
>>>>>>> On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <
>>>>>>> felix.schumacher@internetallee.de> wrote:
>>>>>>>
>>>>>>>> Hi Abhitosh,
>>>>>>>>
>>>>>>>> it would have been a good idea to share sample data, that shows
>>>>>>>> that behaviour.
>>>>>>>>
>>>>>>>> I think you stumbled upon
>>>>>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=61130
>>>>>>>>
>>>>>>>> One problem is, that JMeter uses two different algorithms to
>>>>>>>> calculate a percentile. One for the report and one for the dashboard. They
>>>>>>>> differ in the point, which they use to display the percentile.
>>>>>>>>
>>>>>>>> We did some work to use the same algorithm for both components in
>>>>>>>> https://github.com/apache/jmeter/pull/380 which got closed when
>>>>>>>> converting our repo from subversion to git.
>>>>>>>>
>>>>>>>> In my eyes, that big discrepancy in your values shows, that you are
>>>>>>>> having bad data in the first place. But of course, it would be nice, if
>>>>>>>> JMeter would show consistent values, or even better an indicator of how
>>>>>>>> close the value is to its neighbours.
>>>>>>>>
>>>>>>>> Felix
>>>>>>>> Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
>>>>>>>>
>>>>>>>> @Kieran Lynch
>>>>>>>> I'm using JMeter 5.4
>>>>>>>> I already tried "jmeter.reportgenerator.statistic_window = -1". But
>>>>>>>> it doesn't work.
>>>>>>>>
>>>>>>>> @Mariusz W
>>>>>>>> In my case it's a very huge difference (> 40 seconds) which is not
>>>>>>>> acceptable at all as there are client facing reports.
>>>>>>>>
>>>>>>>> @All
>>>>>>>> I had a discussion with my colleague. As per him there is
>>>>>>>> difference in formula how Aggregate report and HTMl report calculates 90th
>>>>>>>> percentile.
>>>>>>>> Aggregate report taking 5th value out of 6 as 90 th percentile. But
>>>>>>>> HTML report is picking 6th value (which is the highest one) as 90 th
>>>>>>>> percentile.
>>>>>>>> I need to figure it out ASAP now.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes, I have seen this small discrepancies also in my tests but I
>>>>>>>>> don't adhere to exact values and this reports are just first step during
>>>>>>>>> analysis to find an anomaly or something else to focus on , e.g. Is it
>>>>>>>>> worth to look at the mean when data is not symmetrically distributed. If it
>>>>>>>>> is not symmetrical should I care about standard deviation at all. If my
>>>>>>>>> histogram is strange - what was wrong.
>>>>>>>>> But surely the meaning of the report depends on the client's need:)
>>>>>>>>>
>>>>>>>>> Mariusz
>>>>>>>>>
>>>>>>>>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Just chiming in with a higher concurrency test example - not as
>>>>>>>>>> severe but still a difference.
>>>>>>>>>>
>>>>>>>>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to
>>>>>>>>>> me, however. I can give that a go and see if it improves things, but these
>>>>>>>>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>>>>>>>>> and are a bit of a nuisance as they essentially limit me to the output
>>>>>>>>>> format I choose to share with clients...
>>>>>>>>>>
>>>>>>>>>> HTML Output:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Aggregate Report:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Kieran Lynch
>>>>>>>>>>
>>>>>>>>>> WM Reply
>>>>>>>>>> Nova South
>>>>>>>>>> 160 Victoria Street, Westminster
>>>>>>>>>> London SW1E 5LB - UK
>>>>>>>>>> phone: +44 (0)20 7730 6000
>>>>>>>>>> k.lynch@reply.com
>>>>>>>>>> www.reply.com
>>>>>>>>>>
>>>>>>>>>> [image: WM Reply]
>>>>>>>>>> ------------------------------
>>>>>>>>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>>>>>>>>> *Sent:* 26 February 2021 14:28
>>>>>>>>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>>>>>>>>> *Subject:* Re: Incorrect 90th percentile values in JMeter
>>>>>>>>>> dashboard HTML report
>>>>>>>>>>
>>>>>>>>>> This is just an example. Actual test was executed with thousands
>>>>>>>>>> of samples with different transactions.
>>>>>>>>>> It's a transaction controller.
>>>>>>>>>> Also I'm using "Result1.csv" format only.
>>>>>>>>>> I have automated Aggregate report and HTML report creation.
>>>>>>>>>> Manual calculations are not allowed.
>>>>>>>>>>
>>>>>>>>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> Yes, indeed results are strange... You have only 6 samples and
>>>>>>>>>> very wide distribution. Did You try to get more samples? I think that there
>>>>>>>>>> is too small amount of samples to do any descriptive statistics and draw
>>>>>>>>>> conclusions about test results and system. What is this sampler in report?
>>>>>>>>>> Is it http sampler or Transaction Controller? You can always make your own
>>>>>>>>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>>>>>>>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>>>>>>>>> however.
>>>>>>>>>>
>>>>>>>>>> Mariusz
>>>>>>>>>>
>>>>>>>>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <
>>>>>>>>>> abhitosh.qa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I'm generating HTML dashboard report and aggregate report during
>>>>>>>>>> a load test.
>>>>>>>>>> There is a huge difference in 90th percentile values in aggregate
>>>>>>>>>> report and HTML report.
>>>>>>>>>> It seems that HTML dashboard report is showing incorrect values.
>>>>>>>>>> Can someone help me on this?
>>>>>>>>>>
>>>>>>>>>> Screenshots:
>>>>>>>>>> Aggregate report (90th percentile is 123):
>>>>>>>>>> [image: image.png]
>>>>>>>>>>
>>>>>>>>>> HTML report (90th percentile is 44266):
>>>>>>>>>> [image: image.png]
>>>>>>>>>>
>>>>>>>>>> I tried the solution provided in the post below. But it didn't
>>>>>>>>>> work:
>>>>>>>>>>
>>>>>>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> AP
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>>>>>
>>>>>>>>>
>>>
>>> --
>>> Cordialement.
>>> Philippe Mouawad.
>>>
>>>
>>>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>
>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by A P <ab...@gmail.com>.
I'll try it.

On Sat, Mar 6, 2021 at 2:26 AM Philippe Mouawad <ph...@gmail.com>
wrote:

> -1 does not work
> Did you try increasing it ?
>
> Thanks
>
> On Friday, March 5, 2021, A P <ab...@gmail.com> wrote:
>
>> Yes, I tried " jmeter.reportgenerator.statistic_window = -1"
>> But it does not work.
>>
>> On Fri, Mar 5, 2021 at 11:46 PM Philippe Mouawad <
>> philippe.mouawad@gmail.com> wrote:
>>
>>> Hello,
>>> Have you tried increasing this value in user.properties which defaults
>>> to 20000
>>>
>>>
>>>    - jmeter.reportgenerator.statistic_window
>>>
>>>
>>> See:
>>>
>>>
>>>    -
>>>    https://jmeter.apache.org/usermanual/properties_reference.html#reporting
>>>
>>>
>>> Thanks
>>>
>>> Regards
>>>
>>> Philippe M.
>>>
>>> https://leanpub.com/master-jmeter-from-load-test-to-devops
>>>
>>> On Fri, Mar 5, 2021 at 7:12 PM A P <ab...@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I'm eagerly waiting for solution to this issue. If anyone has a fix
>>>> please reply back.
>>>>
>>>> AP
>>>>
>>>> On Sun, Feb 28, 2021 at 11:59 PM A P <ab...@gmail.com> wrote:
>>>>
>>>>> @ Mariusz
>>>>> Yes.
>>>>>
>>>>> On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <ma...@gmail.com> wrote:
>>>>>
>>>>>> @Abhitosh
>>>>>> To be sure: So you loaded manually JTL file in Aggregate Report
>>>>>> listener ( without CMDRunner ) and compared the results with html report?
>>>>>>
>>>>>> Mariusz
>>>>>>
>>>>>> On Sun, 28 Feb 2021 at 12:27, A P <ab...@gmail.com> wrote:
>>>>>>
>>>>>>> *@Mariusz*
>>>>>>> As per below post, HTML report has wrong values:
>>>>>>>
>>>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>>>
>>>>>>> I automated aggregate report creation using JMeter plugin:
>>>>>>> *CMDRunner*
>>>>>>>
>>>>>>> Also I tried creating HTML and Aggregate reports manually using a
>>>>>>> JTL file. But the results are showing the same difference.
>>>>>>>
>>>>>>> My suggestion to reproduce this issue is:
>>>>>>> Execute a long time JMeter load test with a considerable number of
>>>>>>> samples. Then compare HTML and aggregate reports for the test.
>>>>>>> You can see the difference between them.
>>>>>>>
>>>>>>> *@Felix*
>>>>>>> I saw the post: https://github.com/apache/jmeter/pull/380
>>>>>>> But not getting a clear idea how to implement this.
>>>>>>>
>>>>>>> On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <
>>>>>>> felix.schumacher@internetallee.de> wrote:
>>>>>>>
>>>>>>>> Hi Abhitosh,
>>>>>>>>
>>>>>>>> it would have been a good idea to share sample data, that shows
>>>>>>>> that behaviour.
>>>>>>>>
>>>>>>>> I think you stumbled upon
>>>>>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=61130
>>>>>>>>
>>>>>>>> One problem is, that JMeter uses two different algorithms to
>>>>>>>> calculate a percentile. One for the report and one for the dashboard. They
>>>>>>>> differ in the point, which they use to display the percentile.
>>>>>>>>
>>>>>>>> We did some work to use the same algorithm for both components in
>>>>>>>> https://github.com/apache/jmeter/pull/380 which got closed when
>>>>>>>> converting our repo from subversion to git.
>>>>>>>>
>>>>>>>> In my eyes, that big discrepancy in your values shows, that you are
>>>>>>>> having bad data in the first place. But of course, it would be nice, if
>>>>>>>> JMeter would show consistent values, or even better an indicator of how
>>>>>>>> close the value is to its neighbours.
>>>>>>>>
>>>>>>>> Felix
>>>>>>>> Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
>>>>>>>>
>>>>>>>> @Kieran Lynch
>>>>>>>> I'm using JMeter 5.4
>>>>>>>> I already tried "jmeter.reportgenerator.statistic_window = -1". But
>>>>>>>> it doesn't work.
>>>>>>>>
>>>>>>>> @Mariusz W
>>>>>>>> In my case it's a very huge difference (> 40 seconds) which is not
>>>>>>>> acceptable at all as there are client facing reports.
>>>>>>>>
>>>>>>>> @All
>>>>>>>> I had a discussion with my colleague. As per him there is
>>>>>>>> difference in formula how Aggregate report and HTMl report calculates 90th
>>>>>>>> percentile.
>>>>>>>> Aggregate report taking 5th value out of 6 as 90 th percentile. But
>>>>>>>> HTML report is picking 6th value (which is the highest one) as 90 th
>>>>>>>> percentile.
>>>>>>>> I need to figure it out ASAP now.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yes, I have seen this small discrepancies also in my tests but I
>>>>>>>>> don't adhere to exact values and this reports are just first step during
>>>>>>>>> analysis to find an anomaly or something else to focus on , e.g. Is it
>>>>>>>>> worth to look at the mean when data is not symmetrically distributed. If it
>>>>>>>>> is not symmetrical should I care about standard deviation at all. If my
>>>>>>>>> histogram is strange - what was wrong.
>>>>>>>>> But surely the meaning of the report depends on the client's need:)
>>>>>>>>>
>>>>>>>>> Mariusz
>>>>>>>>>
>>>>>>>>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Just chiming in with a higher concurrency test example - not as
>>>>>>>>>> severe but still a difference.
>>>>>>>>>>
>>>>>>>>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to
>>>>>>>>>> me, however. I can give that a go and see if it improves things, but these
>>>>>>>>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>>>>>>>>> and are a bit of a nuisance as they essentially limit me to the output
>>>>>>>>>> format I choose to share with clients...
>>>>>>>>>>
>>>>>>>>>> HTML Output:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Aggregate Report:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Kieran Lynch
>>>>>>>>>>
>>>>>>>>>> WM Reply
>>>>>>>>>> Nova South
>>>>>>>>>> 160 Victoria Street, Westminster
>>>>>>>>>> London SW1E 5LB - UK
>>>>>>>>>> phone: +44 (0)20 7730 6000
>>>>>>>>>> k.lynch@reply.com
>>>>>>>>>> www.reply.com
>>>>>>>>>>
>>>>>>>>>> [image: WM Reply]
>>>>>>>>>> ------------------------------
>>>>>>>>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>>>>>>>>> *Sent:* 26 February 2021 14:28
>>>>>>>>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>>>>>>>>> *Subject:* Re: Incorrect 90th percentile values in JMeter
>>>>>>>>>> dashboard HTML report
>>>>>>>>>>
>>>>>>>>>> This is just an example. Actual test was executed with thousands
>>>>>>>>>> of samples with different transactions.
>>>>>>>>>> It's a transaction controller.
>>>>>>>>>> Also I'm using "Result1.csv" format only.
>>>>>>>>>> I have automated Aggregate report and HTML report creation.
>>>>>>>>>> Manual calculations are not allowed.
>>>>>>>>>>
>>>>>>>>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> Yes, indeed results are strange... You have only 6 samples and
>>>>>>>>>> very wide distribution. Did You try to get more samples? I think that there
>>>>>>>>>> is too small amount of samples to do any descriptive statistics and draw
>>>>>>>>>> conclusions about test results and system. What is this sampler in report?
>>>>>>>>>> Is it http sampler or Transaction Controller? You can always make your own
>>>>>>>>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>>>>>>>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>>>>>>>>> however.
>>>>>>>>>>
>>>>>>>>>> Mariusz
>>>>>>>>>>
>>>>>>>>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <
>>>>>>>>>> abhitosh.qa@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I'm generating HTML dashboard report and aggregate report during
>>>>>>>>>> a load test.
>>>>>>>>>> There is a huge difference in 90th percentile values in aggregate
>>>>>>>>>> report and HTML report.
>>>>>>>>>> It seems that HTML dashboard report is showing incorrect values.
>>>>>>>>>> Can someone help me on this?
>>>>>>>>>>
>>>>>>>>>> Screenshots:
>>>>>>>>>> Aggregate report (90th percentile is 123):
>>>>>>>>>> [image: image.png]
>>>>>>>>>>
>>>>>>>>>> HTML report (90th percentile is 44266):
>>>>>>>>>> [image: image.png]
>>>>>>>>>>
>>>>>>>>>> I tried the solution provided in the post below. But it didn't
>>>>>>>>>> work:
>>>>>>>>>>
>>>>>>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> AP
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>>>>>
>>>>>>>>>
>>>
>>> --
>>> Cordialement.
>>> Philippe Mouawad.
>>>
>>>
>>>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>
>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by Philippe Mouawad <ph...@gmail.com>.
-1 does not work
Did you try increasing it ?

Thanks

On Friday, March 5, 2021, A P <ab...@gmail.com> wrote:

> Yes, I tried " jmeter.reportgenerator.statistic_window = -1"
> But it does not work.
>
> On Fri, Mar 5, 2021 at 11:46 PM Philippe Mouawad <
> philippe.mouawad@gmail.com> wrote:
>
>> Hello,
>> Have you tried increasing this value in user.properties which defaults to
>> 20000
>>
>>
>>    - jmeter.reportgenerator.statistic_window
>>
>>
>> See:
>>
>>
>>    - https://jmeter.apache.org/usermanual/properties_
>>    reference.html#reporting
>>
>>
>> Thanks
>>
>> Regards
>>
>> Philippe M.
>>
>> https://leanpub.com/master-jmeter-from-load-test-to-devops
>>
>> On Fri, Mar 5, 2021 at 7:12 PM A P <ab...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I'm eagerly waiting for solution to this issue. If anyone has a fix
>>> please reply back.
>>>
>>> AP
>>>
>>> On Sun, Feb 28, 2021 at 11:59 PM A P <ab...@gmail.com> wrote:
>>>
>>>> @ Mariusz
>>>> Yes.
>>>>
>>>> On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <ma...@gmail.com> wrote:
>>>>
>>>>> @Abhitosh
>>>>> To be sure: So you loaded manually JTL file in Aggregate Report
>>>>> listener ( without CMDRunner ) and compared the results with html report?
>>>>>
>>>>> Mariusz
>>>>>
>>>>> On Sun, 28 Feb 2021 at 12:27, A P <ab...@gmail.com> wrote:
>>>>>
>>>>>> *@Mariusz*
>>>>>> As per below post, HTML report has wrong values:
>>>>>> https://dzone.com/articles/how-to-achieve-better-
>>>>>> accuracy-in-latency-percenti
>>>>>>
>>>>>> I automated aggregate report creation using JMeter plugin:
>>>>>> *CMDRunner*
>>>>>>
>>>>>> Also I tried creating HTML and Aggregate reports manually using a JTL
>>>>>> file. But the results are showing the same difference.
>>>>>>
>>>>>> My suggestion to reproduce this issue is:
>>>>>> Execute a long time JMeter load test with a considerable number of
>>>>>> samples. Then compare HTML and aggregate reports for the test.
>>>>>> You can see the difference between them.
>>>>>>
>>>>>> *@Felix*
>>>>>> I saw the post: https://github.com/apache/jmeter/pull/380
>>>>>> But not getting a clear idea how to implement this.
>>>>>>
>>>>>> On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <felix.schumacher@
>>>>>> internetallee.de> wrote:
>>>>>>
>>>>>>> Hi Abhitosh,
>>>>>>>
>>>>>>> it would have been a good idea to share sample data, that shows that
>>>>>>> behaviour.
>>>>>>>
>>>>>>> I think you stumbled upon https://bz.apache.org/
>>>>>>> bugzilla/show_bug.cgi?id=61130
>>>>>>>
>>>>>>> One problem is, that JMeter uses two different algorithms to
>>>>>>> calculate a percentile. One for the report and one for the dashboard. They
>>>>>>> differ in the point, which they use to display the percentile.
>>>>>>>
>>>>>>> We did some work to use the same algorithm for both components in
>>>>>>> https://github.com/apache/jmeter/pull/380 which got closed when
>>>>>>> converting our repo from subversion to git.
>>>>>>>
>>>>>>> In my eyes, that big discrepancy in your values shows, that you are
>>>>>>> having bad data in the first place. But of course, it would be nice, if
>>>>>>> JMeter would show consistent values, or even better an indicator of how
>>>>>>> close the value is to its neighbours.
>>>>>>>
>>>>>>> Felix
>>>>>>> Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
>>>>>>>
>>>>>>> @Kieran Lynch
>>>>>>> I'm using JMeter 5.4
>>>>>>> I already tried "jmeter.reportgenerator.statistic_window = -1". But
>>>>>>> it doesn't work.
>>>>>>>
>>>>>>> @Mariusz W
>>>>>>> In my case it's a very huge difference (> 40 seconds) which is not
>>>>>>> acceptable at all as there are client facing reports.
>>>>>>>
>>>>>>> @All
>>>>>>> I had a discussion with my colleague. As per him there is difference
>>>>>>> in formula how Aggregate report and HTMl report calculates 90th percentile.
>>>>>>> Aggregate report taking 5th value out of 6 as 90 th percentile. But
>>>>>>> HTML report is picking 6th value (which is the highest one) as 90 th
>>>>>>> percentile.
>>>>>>> I need to figure it out ASAP now.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yes, I have seen this small discrepancies also in my tests but I
>>>>>>>> don't adhere to exact values and this reports are just first step during
>>>>>>>> analysis to find an anomaly or something else to focus on , e.g. Is it
>>>>>>>> worth to look at the mean when data is not symmetrically distributed. If it
>>>>>>>> is not symmetrical should I care about standard deviation at all. If my
>>>>>>>> histogram is strange - what was wrong.
>>>>>>>> But surely the meaning of the report depends on the client's need:)
>>>>>>>>
>>>>>>>> Mariusz
>>>>>>>>
>>>>>>>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Just chiming in with a higher concurrency test example - not as
>>>>>>>>> severe but still a difference.
>>>>>>>>>
>>>>>>>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to
>>>>>>>>> me, however. I can give that a go and see if it improves things, but these
>>>>>>>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>>>>>>>> and are a bit of a nuisance as they essentially limit me to the output
>>>>>>>>> format I choose to share with clients...
>>>>>>>>>
>>>>>>>>> HTML Output:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Aggregate Report:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Kieran Lynch
>>>>>>>>>
>>>>>>>>> WM Reply
>>>>>>>>> Nova South
>>>>>>>>> 160 Victoria Street, Westminster
>>>>>>>>> London SW1E 5LB - UK
>>>>>>>>> phone: +44 (0)20 7730 6000
>>>>>>>>> k.lynch@reply.com
>>>>>>>>> www.reply.com
>>>>>>>>>
>>>>>>>>> [image: WM Reply]
>>>>>>>>> ------------------------------
>>>>>>>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>>>>>>>> *Sent:* 26 February 2021 14:28
>>>>>>>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>>>>>>>> *Subject:* Re: Incorrect 90th percentile values in JMeter
>>>>>>>>> dashboard HTML report
>>>>>>>>>
>>>>>>>>> This is just an example. Actual test was executed with thousands
>>>>>>>>> of samples with different transactions.
>>>>>>>>> It's a transaction controller.
>>>>>>>>> Also I'm using "Result1.csv" format only.
>>>>>>>>> I have automated Aggregate report and HTML report creation. Manual
>>>>>>>>> calculations are not allowed.
>>>>>>>>>
>>>>>>>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> Yes, indeed results are strange... You have only 6 samples and
>>>>>>>>> very wide distribution. Did You try to get more samples? I think that there
>>>>>>>>> is too small amount of samples to do any descriptive statistics and draw
>>>>>>>>> conclusions about test results and system. What is this sampler in report?
>>>>>>>>> Is it http sampler or Transaction Controller? You can always make your own
>>>>>>>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>>>>>>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>>>>>>>> however.
>>>>>>>>>
>>>>>>>>> Mariusz
>>>>>>>>>
>>>>>>>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <
>>>>>>>>> abhitosh.qa@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm generating HTML dashboard report and aggregate report during a
>>>>>>>>> load test.
>>>>>>>>> There is a huge difference in 90th percentile values in aggregate
>>>>>>>>> report and HTML report.
>>>>>>>>> It seems that HTML dashboard report is showing incorrect values.
>>>>>>>>> Can someone help me on this?
>>>>>>>>>
>>>>>>>>> Screenshots:
>>>>>>>>> Aggregate report (90th percentile is 123):
>>>>>>>>> [image: image.png]
>>>>>>>>>
>>>>>>>>> HTML report (90th percentile is 44266):
>>>>>>>>> [image: image.png]
>>>>>>>>>
>>>>>>>>> I tried the solution provided in the post below. But it didn't
>>>>>>>>> work:
>>>>>>>>> https://dzone.com/articles/how-to-achieve-better-
>>>>>>>>> accuracy-in-latency-percenti
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> AP
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ------------------------------------------------------------
>>>>>>>>> ---------
>>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>>>>
>>>>>>>>
>>
>> --
>> Cordialement.
>> Philippe Mouawad.
>>
>>
>>

-- 
Cordialement.
Philippe Mouawad.

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by A P <ab...@gmail.com>.
Yes, I tried " jmeter.reportgenerator.statistic_window = -1"
But it does not work.

On Fri, Mar 5, 2021 at 11:46 PM Philippe Mouawad <ph...@gmail.com>
wrote:

> Hello,
> Have you tried increasing this value in user.properties which defaults to
> 20000
>
>
>    - jmeter.reportgenerator.statistic_window
>
>
> See:
>
>
>    -
>    https://jmeter.apache.org/usermanual/properties_reference.html#reporting
>
>
> Thanks
>
> Regards
>
> Philippe M.
>
> https://leanpub.com/master-jmeter-from-load-test-to-devops
>
> On Fri, Mar 5, 2021 at 7:12 PM A P <ab...@gmail.com> wrote:
>
>> Hi All,
>>
>> I'm eagerly waiting for solution to this issue. If anyone has a fix
>> please reply back.
>>
>> AP
>>
>> On Sun, Feb 28, 2021 at 11:59 PM A P <ab...@gmail.com> wrote:
>>
>>> @ Mariusz
>>> Yes.
>>>
>>> On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <ma...@gmail.com> wrote:
>>>
>>>> @Abhitosh
>>>> To be sure: So you loaded manually JTL file in Aggregate Report
>>>> listener ( without CMDRunner ) and compared the results with html report?
>>>>
>>>> Mariusz
>>>>
>>>> On Sun, 28 Feb 2021 at 12:27, A P <ab...@gmail.com> wrote:
>>>>
>>>>> *@Mariusz*
>>>>> As per below post, HTML report has wrong values:
>>>>>
>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>
>>>>> I automated aggregate report creation using JMeter plugin: *CMDRunner*
>>>>>
>>>>> Also I tried creating HTML and Aggregate reports manually using a JTL
>>>>> file. But the results are showing the same difference.
>>>>>
>>>>> My suggestion to reproduce this issue is:
>>>>> Execute a long time JMeter load test with a considerable number of
>>>>> samples. Then compare HTML and aggregate reports for the test.
>>>>> You can see the difference between them.
>>>>>
>>>>> *@Felix*
>>>>> I saw the post: https://github.com/apache/jmeter/pull/380
>>>>> But not getting a clear idea how to implement this.
>>>>>
>>>>> On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <
>>>>> felix.schumacher@internetallee.de> wrote:
>>>>>
>>>>>> Hi Abhitosh,
>>>>>>
>>>>>> it would have been a good idea to share sample data, that shows that
>>>>>> behaviour.
>>>>>>
>>>>>> I think you stumbled upon
>>>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=61130
>>>>>>
>>>>>> One problem is, that JMeter uses two different algorithms to
>>>>>> calculate a percentile. One for the report and one for the dashboard. They
>>>>>> differ in the point, which they use to display the percentile.
>>>>>>
>>>>>> We did some work to use the same algorithm for both components in
>>>>>> https://github.com/apache/jmeter/pull/380 which got closed when
>>>>>> converting our repo from subversion to git.
>>>>>>
>>>>>> In my eyes, that big discrepancy in your values shows, that you are
>>>>>> having bad data in the first place. But of course, it would be nice, if
>>>>>> JMeter would show consistent values, or even better an indicator of how
>>>>>> close the value is to its neighbours.
>>>>>>
>>>>>> Felix
>>>>>> Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
>>>>>>
>>>>>> @Kieran Lynch
>>>>>> I'm using JMeter 5.4
>>>>>> I already tried "jmeter.reportgenerator.statistic_window = -1". But
>>>>>> it doesn't work.
>>>>>>
>>>>>> @Mariusz W
>>>>>> In my case it's a very huge difference (> 40 seconds) which is not
>>>>>> acceptable at all as there are client facing reports.
>>>>>>
>>>>>> @All
>>>>>> I had a discussion with my colleague. As per him there is difference
>>>>>> in formula how Aggregate report and HTMl report calculates 90th percentile.
>>>>>> Aggregate report taking 5th value out of 6 as 90 th percentile. But
>>>>>> HTML report is picking 6th value (which is the highest one) as 90 th
>>>>>> percentile.
>>>>>> I need to figure it out ASAP now.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com> wrote:
>>>>>>
>>>>>>> Yes, I have seen this small discrepancies also in my tests but I
>>>>>>> don't adhere to exact values and this reports are just first step during
>>>>>>> analysis to find an anomaly or something else to focus on , e.g. Is it
>>>>>>> worth to look at the mean when data is not symmetrically distributed. If it
>>>>>>> is not symmetrical should I care about standard deviation at all. If my
>>>>>>> histogram is strange - what was wrong.
>>>>>>> But surely the meaning of the report depends on the client's need:)
>>>>>>>
>>>>>>> Mariusz
>>>>>>>
>>>>>>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Just chiming in with a higher concurrency test example - not as
>>>>>>>> severe but still a difference.
>>>>>>>>
>>>>>>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to me,
>>>>>>>> however. I can give that a go and see if it improves things, but these
>>>>>>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>>>>>>> and are a bit of a nuisance as they essentially limit me to the output
>>>>>>>> format I choose to share with clients...
>>>>>>>>
>>>>>>>> HTML Output:
>>>>>>>>
>>>>>>>>
>>>>>>>> Aggregate Report:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Kieran Lynch
>>>>>>>>
>>>>>>>> WM Reply
>>>>>>>> Nova South
>>>>>>>> 160 Victoria Street, Westminster
>>>>>>>> London SW1E 5LB - UK
>>>>>>>> phone: +44 (0)20 7730 6000
>>>>>>>> k.lynch@reply.com
>>>>>>>> www.reply.com
>>>>>>>>
>>>>>>>> [image: WM Reply]
>>>>>>>> ------------------------------
>>>>>>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>>>>>>> *Sent:* 26 February 2021 14:28
>>>>>>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>>>>>>> *Subject:* Re: Incorrect 90th percentile values in JMeter
>>>>>>>> dashboard HTML report
>>>>>>>>
>>>>>>>> This is just an example. Actual test was executed with thousands of
>>>>>>>> samples with different transactions.
>>>>>>>> It's a transaction controller.
>>>>>>>> Also I'm using "Result1.csv" format only.
>>>>>>>> I have automated Aggregate report and HTML report creation. Manual
>>>>>>>> calculations are not allowed.
>>>>>>>>
>>>>>>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> Yes, indeed results are strange... You have only 6 samples and very
>>>>>>>> wide distribution. Did You try to get more samples? I think that there is
>>>>>>>> too small amount of samples to do any descriptive statistics and draw
>>>>>>>> conclusions about test results and system. What is this sampler in report?
>>>>>>>> Is it http sampler or Transaction Controller? You can always make your own
>>>>>>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>>>>>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>>>>>>> however.
>>>>>>>>
>>>>>>>> Mariusz
>>>>>>>>
>>>>>>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm generating HTML dashboard report and aggregate report during a
>>>>>>>> load test.
>>>>>>>> There is a huge difference in 90th percentile values in aggregate
>>>>>>>> report and HTML report.
>>>>>>>> It seems that HTML dashboard report is showing incorrect values.
>>>>>>>> Can someone help me on this?
>>>>>>>>
>>>>>>>> Screenshots:
>>>>>>>> Aggregate report (90th percentile is 123):
>>>>>>>> [image: image.png]
>>>>>>>>
>>>>>>>> HTML report (90th percentile is 44266):
>>>>>>>> [image: image.png]
>>>>>>>>
>>>>>>>> I tried the solution provided in the post below. But it didn't work:
>>>>>>>>
>>>>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> AP
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>>>
>>>>>>>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by Philippe Mouawad <ph...@gmail.com>.
Hello,
Have you tried increasing this value in user.properties which defaults to
20000


   - jmeter.reportgenerator.statistic_window


See:


   -
   https://jmeter.apache.org/usermanual/properties_reference.html#reporting


Thanks

Regards

Philippe M.

https://leanpub.com/master-jmeter-from-load-test-to-devops

On Fri, Mar 5, 2021 at 7:12 PM A P <ab...@gmail.com> wrote:

> Hi All,
>
> I'm eagerly waiting for solution to this issue. If anyone has a fix please
> reply back.
>
> AP
>
> On Sun, Feb 28, 2021 at 11:59 PM A P <ab...@gmail.com> wrote:
>
>> @ Mariusz
>> Yes.
>>
>> On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <ma...@gmail.com> wrote:
>>
>>> @Abhitosh
>>> To be sure: So you loaded manually JTL file in Aggregate Report listener
>>> ( without CMDRunner ) and compared the results with html report?
>>>
>>> Mariusz
>>>
>>> On Sun, 28 Feb 2021 at 12:27, A P <ab...@gmail.com> wrote:
>>>
>>>> *@Mariusz*
>>>> As per below post, HTML report has wrong values:
>>>>
>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>
>>>> I automated aggregate report creation using JMeter plugin: *CMDRunner*
>>>>
>>>> Also I tried creating HTML and Aggregate reports manually using a JTL
>>>> file. But the results are showing the same difference.
>>>>
>>>> My suggestion to reproduce this issue is:
>>>> Execute a long time JMeter load test with a considerable number of
>>>> samples. Then compare HTML and aggregate reports for the test.
>>>> You can see the difference between them.
>>>>
>>>> *@Felix*
>>>> I saw the post: https://github.com/apache/jmeter/pull/380
>>>> But not getting a clear idea how to implement this.
>>>>
>>>> On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <
>>>> felix.schumacher@internetallee.de> wrote:
>>>>
>>>>> Hi Abhitosh,
>>>>>
>>>>> it would have been a good idea to share sample data, that shows that
>>>>> behaviour.
>>>>>
>>>>> I think you stumbled upon
>>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=61130
>>>>>
>>>>> One problem is, that JMeter uses two different algorithms to calculate
>>>>> a percentile. One for the report and one for the dashboard. They differ in
>>>>> the point, which they use to display the percentile.
>>>>>
>>>>> We did some work to use the same algorithm for both components in
>>>>> https://github.com/apache/jmeter/pull/380 which got closed when
>>>>> converting our repo from subversion to git.
>>>>>
>>>>> In my eyes, that big discrepancy in your values shows, that you are
>>>>> having bad data in the first place. But of course, it would be nice, if
>>>>> JMeter would show consistent values, or even better an indicator of how
>>>>> close the value is to its neighbours.
>>>>>
>>>>> Felix
>>>>> Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
>>>>>
>>>>> @Kieran Lynch
>>>>> I'm using JMeter 5.4
>>>>> I already tried "jmeter.reportgenerator.statistic_window = -1". But it
>>>>> doesn't work.
>>>>>
>>>>> @Mariusz W
>>>>> In my case it's a very huge difference (> 40 seconds) which is not
>>>>> acceptable at all as there are client facing reports.
>>>>>
>>>>> @All
>>>>> I had a discussion with my colleague. As per him there is difference
>>>>> in formula how Aggregate report and HTMl report calculates 90th percentile.
>>>>> Aggregate report taking 5th value out of 6 as 90 th percentile. But
>>>>> HTML report is picking 6th value (which is the highest one) as 90 th
>>>>> percentile.
>>>>> I need to figure it out ASAP now.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com> wrote:
>>>>>
>>>>>> Yes, I have seen this small discrepancies also in my tests but I
>>>>>> don't adhere to exact values and this reports are just first step during
>>>>>> analysis to find an anomaly or something else to focus on , e.g. Is it
>>>>>> worth to look at the mean when data is not symmetrically distributed. If it
>>>>>> is not symmetrical should I care about standard deviation at all. If my
>>>>>> histogram is strange - what was wrong.
>>>>>> But surely the meaning of the report depends on the client's need:)
>>>>>>
>>>>>> Mariusz
>>>>>>
>>>>>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com> wrote:
>>>>>>
>>>>>>> Just chiming in with a higher concurrency test example - not as
>>>>>>> severe but still a difference.
>>>>>>>
>>>>>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to me,
>>>>>>> however. I can give that a go and see if it improves things, but these
>>>>>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>>>>>> and are a bit of a nuisance as they essentially limit me to the output
>>>>>>> format I choose to share with clients...
>>>>>>>
>>>>>>> HTML Output:
>>>>>>>
>>>>>>>
>>>>>>> Aggregate Report:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Kieran Lynch
>>>>>>>
>>>>>>> WM Reply
>>>>>>> Nova South
>>>>>>> 160 Victoria Street, Westminster
>>>>>>> London SW1E 5LB - UK
>>>>>>> phone: +44 (0)20 7730 6000
>>>>>>> k.lynch@reply.com
>>>>>>> www.reply.com
>>>>>>>
>>>>>>> [image: WM Reply]
>>>>>>> ------------------------------
>>>>>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>>>>>> *Sent:* 26 February 2021 14:28
>>>>>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>>>>>> *Subject:* Re: Incorrect 90th percentile values in JMeter dashboard
>>>>>>> HTML report
>>>>>>>
>>>>>>> This is just an example. Actual test was executed with thousands of
>>>>>>> samples with different transactions.
>>>>>>> It's a transaction controller.
>>>>>>> Also I'm using "Result1.csv" format only.
>>>>>>> I have automated Aggregate report and HTML report creation. Manual
>>>>>>> calculations are not allowed.
>>>>>>>
>>>>>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> Yes, indeed results are strange... You have only 6 samples and very
>>>>>>> wide distribution. Did You try to get more samples? I think that there is
>>>>>>> too small amount of samples to do any descriptive statistics and draw
>>>>>>> conclusions about test results and system. What is this sampler in report?
>>>>>>> Is it http sampler or Transaction Controller? You can always make your own
>>>>>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>>>>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>>>>>> however.
>>>>>>>
>>>>>>> Mariusz
>>>>>>>
>>>>>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm generating HTML dashboard report and aggregate report during a
>>>>>>> load test.
>>>>>>> There is a huge difference in 90th percentile values in aggregate
>>>>>>> report and HTML report.
>>>>>>> It seems that HTML dashboard report is showing incorrect values.
>>>>>>> Can someone help me on this?
>>>>>>>
>>>>>>> Screenshots:
>>>>>>> Aggregate report (90th percentile is 123):
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>> HTML report (90th percentile is 44266):
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>> I tried the solution provided in the post below. But it didn't work:
>>>>>>>
>>>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>> AP
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>>
>>>>>>

-- 
Cordialement.
Philippe Mouawad.

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by A P <ab...@gmail.com>.
Hi All,

I'm eagerly waiting for solution to this issue. If anyone has a fix please
reply back.

AP

On Sun, Feb 28, 2021 at 11:59 PM A P <ab...@gmail.com> wrote:

> @ Mariusz
> Yes.
>
> On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <ma...@gmail.com> wrote:
>
>> @Abhitosh
>> To be sure: So you loaded manually JTL file in Aggregate Report listener
>> ( without CMDRunner ) and compared the results with html report?
>>
>> Mariusz
>>
>> On Sun, 28 Feb 2021 at 12:27, A P <ab...@gmail.com> wrote:
>>
>>> *@Mariusz*
>>> As per below post, HTML report has wrong values:
>>>
>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>
>>> I automated aggregate report creation using JMeter plugin: *CMDRunner*
>>>
>>> Also I tried creating HTML and Aggregate reports manually using a JTL
>>> file. But the results are showing the same difference.
>>>
>>> My suggestion to reproduce this issue is:
>>> Execute a long time JMeter load test with a considerable number of
>>> samples. Then compare HTML and aggregate reports for the test.
>>> You can see the difference between them.
>>>
>>> *@Felix*
>>> I saw the post: https://github.com/apache/jmeter/pull/380
>>> But not getting a clear idea how to implement this.
>>>
>>> On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <
>>> felix.schumacher@internetallee.de> wrote:
>>>
>>>> Hi Abhitosh,
>>>>
>>>> it would have been a good idea to share sample data, that shows that
>>>> behaviour.
>>>>
>>>> I think you stumbled upon
>>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=61130
>>>>
>>>> One problem is, that JMeter uses two different algorithms to calculate
>>>> a percentile. One for the report and one for the dashboard. They differ in
>>>> the point, which they use to display the percentile.
>>>>
>>>> We did some work to use the same algorithm for both components in
>>>> https://github.com/apache/jmeter/pull/380 which got closed when
>>>> converting our repo from subversion to git.
>>>>
>>>> In my eyes, that big discrepancy in your values shows, that you are
>>>> having bad data in the first place. But of course, it would be nice, if
>>>> JMeter would show consistent values, or even better an indicator of how
>>>> close the value is to its neighbours.
>>>>
>>>> Felix
>>>> Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
>>>>
>>>> @Kieran Lynch
>>>> I'm using JMeter 5.4
>>>> I already tried "jmeter.reportgenerator.statistic_window = -1". But it
>>>> doesn't work.
>>>>
>>>> @Mariusz W
>>>> In my case it's a very huge difference (> 40 seconds) which is not
>>>> acceptable at all as there are client facing reports.
>>>>
>>>> @All
>>>> I had a discussion with my colleague. As per him there is difference in
>>>> formula how Aggregate report and HTMl report calculates 90th percentile.
>>>> Aggregate report taking 5th value out of 6 as 90 th percentile. But
>>>> HTML report is picking 6th value (which is the highest one) as 90 th
>>>> percentile.
>>>> I need to figure it out ASAP now.
>>>>
>>>>
>>>>
>>>> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com> wrote:
>>>>
>>>>> Yes, I have seen this small discrepancies also in my tests but I don't
>>>>> adhere to exact values and this reports are just first step during analysis
>>>>> to find an anomaly or something else to focus on , e.g. Is it worth to look
>>>>> at the mean when data is not symmetrically distributed. If it is not
>>>>> symmetrical should I care about standard deviation at all. If my histogram
>>>>> is strange - what was wrong.
>>>>> But surely the meaning of the report depends on the client's need:)
>>>>>
>>>>> Mariusz
>>>>>
>>>>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com> wrote:
>>>>>
>>>>>> Just chiming in with a higher concurrency test example - not as
>>>>>> severe but still a difference.
>>>>>>
>>>>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to me,
>>>>>> however. I can give that a go and see if it improves things, but these
>>>>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>>>>> and are a bit of a nuisance as they essentially limit me to the output
>>>>>> format I choose to share with clients...
>>>>>>
>>>>>> HTML Output:
>>>>>>
>>>>>>
>>>>>> Aggregate Report:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Kieran Lynch
>>>>>>
>>>>>> WM Reply
>>>>>> Nova South
>>>>>> 160 Victoria Street, Westminster
>>>>>> London SW1E 5LB - UK
>>>>>> phone: +44 (0)20 7730 6000
>>>>>> k.lynch@reply.com
>>>>>> www.reply.com
>>>>>>
>>>>>> [image: WM Reply]
>>>>>> ------------------------------
>>>>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>>>>> *Sent:* 26 February 2021 14:28
>>>>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>>>>> *Subject:* Re: Incorrect 90th percentile values in JMeter dashboard
>>>>>> HTML report
>>>>>>
>>>>>> This is just an example. Actual test was executed with thousands of
>>>>>> samples with different transactions.
>>>>>> It's a transaction controller.
>>>>>> Also I'm using "Result1.csv" format only.
>>>>>> I have automated Aggregate report and HTML report creation. Manual
>>>>>> calculations are not allowed.
>>>>>>
>>>>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> Yes, indeed results are strange... You have only 6 samples and very
>>>>>> wide distribution. Did You try to get more samples? I think that there is
>>>>>> too small amount of samples to do any descriptive statistics and draw
>>>>>> conclusions about test results and system. What is this sampler in report?
>>>>>> Is it http sampler or Transaction Controller? You can always make your own
>>>>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>>>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>>>>> however.
>>>>>>
>>>>>> Mariusz
>>>>>>
>>>>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm generating HTML dashboard report and aggregate report during a
>>>>>> load test.
>>>>>> There is a huge difference in 90th percentile values in aggregate
>>>>>> report and HTML report.
>>>>>> It seems that HTML dashboard report is showing incorrect values.
>>>>>> Can someone help me on this?
>>>>>>
>>>>>> Screenshots:
>>>>>> Aggregate report (90th percentile is 123):
>>>>>> [image: image.png]
>>>>>>
>>>>>> HTML report (90th percentile is 44266):
>>>>>> [image: image.png]
>>>>>>
>>>>>> I tried the solution provided in the post below. But it didn't work:
>>>>>>
>>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> AP
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>>
>>>>>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by A P <ab...@gmail.com>.
@ Mariusz
Yes.

On Sun, Feb 28, 2021 at 8:47 PM Mariusz W <ma...@gmail.com> wrote:

> @Abhitosh
> To be sure: So you loaded manually JTL file in Aggregate Report listener (
> without CMDRunner ) and compared the results with html report?
>
> Mariusz
>
> On Sun, 28 Feb 2021 at 12:27, A P <ab...@gmail.com> wrote:
>
>> *@Mariusz*
>> As per below post, HTML report has wrong values:
>>
>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>
>> I automated aggregate report creation using JMeter plugin: *CMDRunner*
>>
>> Also I tried creating HTML and Aggregate reports manually using a JTL
>> file. But the results are showing the same difference.
>>
>> My suggestion to reproduce this issue is:
>> Execute a long time JMeter load test with a considerable number of
>> samples. Then compare HTML and aggregate reports for the test.
>> You can see the difference between them.
>>
>> *@Felix*
>> I saw the post: https://github.com/apache/jmeter/pull/380
>> But not getting a clear idea how to implement this.
>>
>> On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <
>> felix.schumacher@internetallee.de> wrote:
>>
>>> Hi Abhitosh,
>>>
>>> it would have been a good idea to share sample data, that shows that
>>> behaviour.
>>>
>>> I think you stumbled upon
>>> https://bz.apache.org/bugzilla/show_bug.cgi?id=61130
>>>
>>> One problem is, that JMeter uses two different algorithms to calculate a
>>> percentile. One for the report and one for the dashboard. They differ in
>>> the point, which they use to display the percentile.
>>>
>>> We did some work to use the same algorithm for both components in
>>> https://github.com/apache/jmeter/pull/380 which got closed when
>>> converting our repo from subversion to git.
>>>
>>> In my eyes, that big discrepancy in your values shows, that you are
>>> having bad data in the first place. But of course, it would be nice, if
>>> JMeter would show consistent values, or even better an indicator of how
>>> close the value is to its neighbours.
>>>
>>> Felix
>>> Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
>>>
>>> @Kieran Lynch
>>> I'm using JMeter 5.4
>>> I already tried "jmeter.reportgenerator.statistic_window = -1". But it
>>> doesn't work.
>>>
>>> @Mariusz W
>>> In my case it's a very huge difference (> 40 seconds) which is not
>>> acceptable at all as there are client facing reports.
>>>
>>> @All
>>> I had a discussion with my colleague. As per him there is difference in
>>> formula how Aggregate report and HTMl report calculates 90th percentile.
>>> Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML
>>> report is picking 6th value (which is the highest one) as 90 th percentile.
>>> I need to figure it out ASAP now.
>>>
>>>
>>>
>>> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com> wrote:
>>>
>>>> Yes, I have seen this small discrepancies also in my tests but I don't
>>>> adhere to exact values and this reports are just first step during analysis
>>>> to find an anomaly or something else to focus on , e.g. Is it worth to look
>>>> at the mean when data is not symmetrically distributed. If it is not
>>>> symmetrical should I care about standard deviation at all. If my histogram
>>>> is strange - what was wrong.
>>>> But surely the meaning of the report depends on the client's need:)
>>>>
>>>> Mariusz
>>>>
>>>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com> wrote:
>>>>
>>>>> Just chiming in with a higher concurrency test example - not as severe
>>>>> but still a difference.
>>>>>
>>>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to me,
>>>>> however. I can give that a go and see if it improves things, but these
>>>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>>>> and are a bit of a nuisance as they essentially limit me to the output
>>>>> format I choose to share with clients...
>>>>>
>>>>> HTML Output:
>>>>>
>>>>>
>>>>> Aggregate Report:
>>>>>
>>>>>
>>>>>
>>>>> Kieran Lynch
>>>>>
>>>>> WM Reply
>>>>> Nova South
>>>>> 160 Victoria Street, Westminster
>>>>> London SW1E 5LB - UK
>>>>> phone: +44 (0)20 7730 6000
>>>>> k.lynch@reply.com
>>>>> www.reply.com
>>>>>
>>>>> [image: WM Reply]
>>>>> ------------------------------
>>>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>>>> *Sent:* 26 February 2021 14:28
>>>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>>>> *Subject:* Re: Incorrect 90th percentile values in JMeter dashboard
>>>>> HTML report
>>>>>
>>>>> This is just an example. Actual test was executed with thousands of
>>>>> samples with different transactions.
>>>>> It's a transaction controller.
>>>>> Also I'm using "Result1.csv" format only.
>>>>> I have automated Aggregate report and HTML report creation. Manual
>>>>> calculations are not allowed.
>>>>>
>>>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>> Yes, indeed results are strange... You have only 6 samples and very
>>>>> wide distribution. Did You try to get more samples? I think that there is
>>>>> too small amount of samples to do any descriptive statistics and draw
>>>>> conclusions about test results and system. What is this sampler in report?
>>>>> Is it http sampler or Transaction Controller? You can always make your own
>>>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>>>> however.
>>>>>
>>>>> Mariusz
>>>>>
>>>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I'm generating HTML dashboard report and aggregate report during a
>>>>> load test.
>>>>> There is a huge difference in 90th percentile values in aggregate
>>>>> report and HTML report.
>>>>> It seems that HTML dashboard report is showing incorrect values.
>>>>> Can someone help me on this?
>>>>>
>>>>> Screenshots:
>>>>> Aggregate report (90th percentile is 123):
>>>>> [image: image.png]
>>>>>
>>>>> HTML report (90th percentile is 44266):
>>>>> [image: image.png]
>>>>>
>>>>> I tried the solution provided in the post below. But it didn't work:
>>>>>
>>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>>
>>>>>
>>>>> Regards,
>>>>> AP
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>>
>>>>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by Mariusz W <ma...@gmail.com>.
@Abhitosh
To be sure: So you loaded manually JTL file in Aggregate Report listener (
without CMDRunner ) and compared the results with html report?

Mariusz

On Sun, 28 Feb 2021 at 12:27, A P <ab...@gmail.com> wrote:

> *@Mariusz*
> As per below post, HTML report has wrong values:
>
> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>
> I automated aggregate report creation using JMeter plugin: *CMDRunner*
>
> Also I tried creating HTML and Aggregate reports manually using a JTL
> file. But the results are showing the same difference.
>
> My suggestion to reproduce this issue is:
> Execute a long time JMeter load test with a considerable number of
> samples. Then compare HTML and aggregate reports for the test.
> You can see the difference between them.
>
> *@Felix*
> I saw the post: https://github.com/apache/jmeter/pull/380
> But not getting a clear idea how to implement this.
>
> On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <
> felix.schumacher@internetallee.de> wrote:
>
>> Hi Abhitosh,
>>
>> it would have been a good idea to share sample data, that shows that
>> behaviour.
>>
>> I think you stumbled upon
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=61130
>>
>> One problem is, that JMeter uses two different algorithms to calculate a
>> percentile. One for the report and one for the dashboard. They differ in
>> the point, which they use to display the percentile.
>>
>> We did some work to use the same algorithm for both components in
>> https://github.com/apache/jmeter/pull/380 which got closed when
>> converting our repo from subversion to git.
>>
>> In my eyes, that big discrepancy in your values shows, that you are
>> having bad data in the first place. But of course, it would be nice, if
>> JMeter would show consistent values, or even better an indicator of how
>> close the value is to its neighbours.
>>
>> Felix
>> Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
>>
>> @Kieran Lynch
>> I'm using JMeter 5.4
>> I already tried "jmeter.reportgenerator.statistic_window = -1". But it
>> doesn't work.
>>
>> @Mariusz W
>> In my case it's a very huge difference (> 40 seconds) which is not
>> acceptable at all as there are client facing reports.
>>
>> @All
>> I had a discussion with my colleague. As per him there is difference in
>> formula how Aggregate report and HTMl report calculates 90th percentile.
>> Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML
>> report is picking 6th value (which is the highest one) as 90 th percentile.
>> I need to figure it out ASAP now.
>>
>>
>>
>> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com> wrote:
>>
>>> Yes, I have seen this small discrepancies also in my tests but I don't
>>> adhere to exact values and this reports are just first step during analysis
>>> to find an anomaly or something else to focus on , e.g. Is it worth to look
>>> at the mean when data is not symmetrically distributed. If it is not
>>> symmetrical should I care about standard deviation at all. If my histogram
>>> is strange - what was wrong.
>>> But surely the meaning of the report depends on the client's need:)
>>>
>>> Mariusz
>>>
>>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com> wrote:
>>>
>>>> Just chiming in with a higher concurrency test example - not as severe
>>>> but still a difference.
>>>>
>>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to me,
>>>> however. I can give that a go and see if it improves things, but these
>>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>>> and are a bit of a nuisance as they essentially limit me to the output
>>>> format I choose to share with clients...
>>>>
>>>> HTML Output:
>>>>
>>>>
>>>> Aggregate Report:
>>>>
>>>>
>>>>
>>>> Kieran Lynch
>>>>
>>>> WM Reply
>>>> Nova South
>>>> 160 Victoria Street, Westminster
>>>> London SW1E 5LB - UK
>>>> phone: +44 (0)20 7730 6000
>>>> k.lynch@reply.com
>>>> www.reply.com
>>>>
>>>> [image: WM Reply]
>>>> ------------------------------
>>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>>> *Sent:* 26 February 2021 14:28
>>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>>> *Subject:* Re: Incorrect 90th percentile values in JMeter dashboard
>>>> HTML report
>>>>
>>>> This is just an example. Actual test was executed with thousands of
>>>> samples with different transactions.
>>>> It's a transaction controller.
>>>> Also I'm using "Result1.csv" format only.
>>>> I have automated Aggregate report and HTML report creation. Manual
>>>> calculations are not allowed.
>>>>
>>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com> wrote:
>>>>
>>>> Hi,
>>>> Yes, indeed results are strange... You have only 6 samples and very
>>>> wide distribution. Did You try to get more samples? I think that there is
>>>> too small amount of samples to do any descriptive statistics and draw
>>>> conclusions about test results and system. What is this sampler in report?
>>>> Is it http sampler or Transaction Controller? You can always make your own
>>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>>> however.
>>>>
>>>> Mariusz
>>>>
>>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm generating HTML dashboard report and aggregate report during a load
>>>> test.
>>>> There is a huge difference in 90th percentile values in aggregate
>>>> report and HTML report.
>>>> It seems that HTML dashboard report is showing incorrect values.
>>>> Can someone help me on this?
>>>>
>>>> Screenshots:
>>>> Aggregate report (90th percentile is 123):
>>>> [image: image.png]
>>>>
>>>> HTML report (90th percentile is 44266):
>>>> [image: image.png]
>>>>
>>>> I tried the solution provided in the post below. But it didn't work:
>>>>
>>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>>
>>>>
>>>> Regards,
>>>> AP
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>>
>>>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by A P <ab...@gmail.com>.
*@Mariusz*
As per below post, HTML report has wrong values:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti

I automated aggregate report creation using JMeter plugin: *CMDRunner*

Also I tried creating HTML and Aggregate reports manually using a JTL file.
But the results are showing the same difference.

My suggestion to reproduce this issue is:
Execute a long time JMeter load test with a considerable number of samples.
Then compare HTML and aggregate reports for the test.
You can see the difference between them.

*@Felix*
I saw the post: https://github.com/apache/jmeter/pull/380
But not getting a clear idea how to implement this.

On Sun, Feb 28, 2021 at 4:47 PM Felix Schumacher <
felix.schumacher@internetallee.de> wrote:

> Hi Abhitosh,
>
> it would have been a good idea to share sample data, that shows that
> behaviour.
>
> I think you stumbled upon
> https://bz.apache.org/bugzilla/show_bug.cgi?id=61130
>
> One problem is, that JMeter uses two different algorithms to calculate a
> percentile. One for the report and one for the dashboard. They differ in
> the point, which they use to display the percentile.
>
> We did some work to use the same algorithm for both components in
> https://github.com/apache/jmeter/pull/380 which got closed when
> converting our repo from subversion to git.
>
> In my eyes, that big discrepancy in your values shows, that you are having
> bad data in the first place. But of course, it would be nice, if JMeter
> would show consistent values, or even better an indicator of how close the
> value is to its neighbours.
>
> Felix
> Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
>
> @Kieran Lynch
> I'm using JMeter 5.4
> I already tried "jmeter.reportgenerator.statistic_window = -1". But it
> doesn't work.
>
> @Mariusz W
> In my case it's a very huge difference (> 40 seconds) which is not
> acceptable at all as there are client facing reports.
>
> @All
> I had a discussion with my colleague. As per him there is difference in
> formula how Aggregate report and HTMl report calculates 90th percentile.
> Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML
> report is picking 6th value (which is the highest one) as 90 th percentile.
> I need to figure it out ASAP now.
>
>
>
> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com> wrote:
>
>> Yes, I have seen this small discrepancies also in my tests but I don't
>> adhere to exact values and this reports are just first step during analysis
>> to find an anomaly or something else to focus on , e.g. Is it worth to look
>> at the mean when data is not symmetrically distributed. If it is not
>> symmetrical should I care about standard deviation at all. If my histogram
>> is strange - what was wrong.
>> But surely the meaning of the report depends on the client's need:)
>>
>> Mariusz
>>
>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com> wrote:
>>
>>> Just chiming in with a higher concurrency test example - not as severe
>>> but still a difference.
>>>
>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to me,
>>> however. I can give that a go and see if it improves things, but these
>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>> and are a bit of a nuisance as they essentially limit me to the output
>>> format I choose to share with clients...
>>>
>>> HTML Output:
>>>
>>>
>>> Aggregate Report:
>>>
>>>
>>>
>>> Kieran Lynch
>>>
>>> WM Reply
>>> Nova South
>>> 160 Victoria Street, Westminster
>>> London SW1E 5LB - UK
>>> phone: +44 (0)20 7730 6000
>>> k.lynch@reply.com
>>> www.reply.com
>>>
>>> [image: WM Reply]
>>> ------------------------------
>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>> *Sent:* 26 February 2021 14:28
>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>> *Subject:* Re: Incorrect 90th percentile values in JMeter dashboard
>>> HTML report
>>>
>>> This is just an example. Actual test was executed with thousands of
>>> samples with different transactions.
>>> It's a transaction controller.
>>> Also I'm using "Result1.csv" format only.
>>> I have automated Aggregate report and HTML report creation. Manual
>>> calculations are not allowed.
>>>
>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com> wrote:
>>>
>>> Hi,
>>> Yes, indeed results are strange... You have only 6 samples and very wide
>>> distribution. Did You try to get more samples? I think that there is too
>>> small amount of samples to do any descriptive statistics and draw
>>> conclusions about test results and system. What is this sampler in report?
>>> Is it http sampler or Transaction Controller? You can always make your own
>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>> however.
>>>
>>> Mariusz
>>>
>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I'm generating HTML dashboard report and aggregate report during a load
>>> test.
>>> There is a huge difference in 90th percentile values in aggregate report
>>> and HTML report.
>>> It seems that HTML dashboard report is showing incorrect values.
>>> Can someone help me on this?
>>>
>>> Screenshots:
>>> Aggregate report (90th percentile is 123):
>>> [image: image.png]
>>>
>>> HTML report (90th percentile is 44266):
>>> [image: image.png]
>>>
>>> I tried the solution provided in the post below. But it didn't work:
>>>
>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>
>>>
>>> Regards,
>>> AP
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by Felix Schumacher <fe...@internetallee.de>.
Hi Abhitosh,

it would have been a good idea to share sample data, that shows that
behaviour.

I think you stumbled upon
https://bz.apache.org/bugzilla/show_bug.cgi?id=61130

One problem is, that JMeter uses two different algorithms to calculate a
percentile. One for the report and one for the dashboard. They differ in
the point, which they use to display the percentile.

We did some work to use the same algorithm for both components in
https://github.com/apache/jmeter/pull/380 which got closed when
converting our repo from subversion to git.

In my eyes, that big discrepancy in your values shows, that you are
having bad data in the first place. But of course, it would be nice, if
JMeter would show consistent values, or even better an indicator of how
close the value is to its neighbours.

Felix

Am 26.02.21 um 19:38 schrieb Abhitosh Patil:
> @Kieran Lynch
> I'm using JMeter 5.4
> I already tried "jmeter.reportgenerator.statistic_window = -1". But it
> doesn't work.
>
> @Mariusz W
> In my case it's a very huge difference (> 40 seconds) which is not
> acceptable at all as there are client facing reports.
>
> @All
> I had a discussion with my colleague. As per him there is difference
> in formula how Aggregate report and HTMl report calculates 90th
> percentile.
> Aggregate report taking 5th value out of 6 as 90 th percentile. But
> HTML report is picking 6th value (which is the highest one) as 90 th
> percentile.
> I need to figure it out ASAP now.
>
>
>
> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <mawasak@gmail.com
> <ma...@gmail.com>> wrote:
>
>     Yes, I have seen this small discrepancies also in my tests but I
>     don't adhere to exact values and this reports are just first step
>     during analysis to find an anomaly or something else to focus on ,
>     e.g. Is it worth to look at the mean when data is not
>     symmetrically distributed. If it is not symmetrical should I care
>     about standard deviation at all. If my histogram is strange - what
>     was wrong.
>     But surely the meaning of the report depends on the client's need:)
>
>     Mariusz
>
>     On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k.lynch@reply.com
>     <ma...@reply.com>> wrote:
>
>         Just chiming in with a higher concurrency test example - not
>         as severe but still a difference.
>
>         Setting jmeter.reportgenerator.statistic_window = -1 is news
>         to me, however. I can give that a go and see if it improves
>         things, but these discrepancies have been present as long as
>         I've been using jmeter (~v3.2) and are a bit of a nuisance as
>         they essentially limit me to the output format I choose to
>         share with clients...
>
>         HTML Output:
>
>
>         Aggregate Report:
>
>
>
>         Kieran Lynch
>
>         WM Reply
>         Nova South
>         160 Victoria Street, Westminster
>         London SW1E 5LB - UK
>         phone: +44 (0)20 7730 6000
>         k.lynch@reply.com <ma...@reply.com>
>         www.reply.com <http://www.reply.com>
>
>         WM Reply
>         ------------------------------------------------------------------------
>         *From:* Abhitosh Patil <abhitosh.qa@gmail.com
>         <ma...@gmail.com>>
>         *Sent:* 26 February 2021 14:28
>         *To:* JMeter Users List <user@jmeter.apache.org
>         <ma...@jmeter.apache.org>>
>         *Subject:* Re: Incorrect 90th percentile values in JMeter
>         dashboard HTML report
>          
>         This is just an example. Actual test was executed with
>         thousands of samples with different transactions.
>         It's a transaction controller.
>         Also I'm using "Result1.csv" format only.
>         I have automated Aggregate report and HTML report creation.
>         Manual calculations are not allowed.
>
>         On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <mawasak@gmail.com
>         <ma...@gmail.com>> wrote:
>
>             Hi,
>             Yes, indeed results are strange... You have only 6 samples
>             and very wide distribution. Did You try to get more
>             samples? I think that there is too small amount of samples
>             to do any descriptive statistics and draw conclusions
>             about test results and system. What is this sampler in
>             report? Is it http sampler or Transaction Controller? You
>             can always make your own  calculations in Excel (use -l
>             result1.csv during test and load csv file in Excel). In
>             excel you cannot use PERCENTILE.EXC on small number of
>             samples however.
>
>             Mariusz
>
>             On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil
>             <abhitosh.qa@gmail.com <ma...@gmail.com>> wrote:
>
>                 Hi,
>
>                 I'm generating HTML dashboard report and aggregate
>                 report during a load test.
>                 There is a huge difference in 90th percentile values
>                 in aggregate report and HTML report.
>                 It seems that HTML dashboard report is showing
>                 incorrect values.
>                 Can someone help me on this?
>
>                 Screenshots:
>                 Aggregate report (90th percentile is 123):
>                 image.png
>
>                 HTML report (90th percentile is 44266):
>                 image.png
>
>                 I tried the solution provided in the post below. But
>                 it didn't work:
>                 https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>                 <https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti>
>
>
>
>                 Regards,
>                 AP
>
>
>         ---------------------------------------------------------------------
>         To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>         <ma...@jmeter.apache.org>
>         For additional commands, e-mail: user-help@jmeter.apache.org
>         <ma...@jmeter.apache.org>
>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by Mariusz W <ma...@gmail.com>.
I have to admint that I have never seen such big discrepancies in JMeter
reports. So in which report data is wrong in your opinion? When you talk
about "Aggregate report" do you think about JMeter report generated by
Aggregate Report listener? I ask because in your images You have data in
Excel and standard deviation is present but (as far as I know it is not
present in this listener). Standard deviation is generated in Summary
Report listener. So you calculated it yourself or merged reports? We also
see only some examples (only 6 samples report) so can you send any more
realistic data (more samples)? My hypothesis (based on information I see so
far, or my understanding of this information:) ) is that You have some
error in automatic report generation (You said: "I have automated Aggregate
report and HTML report creation" ). But as I have said,  it is based on my
understanding of data you provided:)

Mariusz

On Fri, 26 Feb 2021 at 19:39, Abhitosh Patil <ab...@gmail.com> wrote:

> @Kieran Lynch
> I'm using JMeter 5.4
> I already tried "jmeter.reportgenerator.statistic_window = -1". But it
> doesn't work.
>
> @Mariusz W
> In my case it's a very huge difference (> 40 seconds) which is not
> acceptable at all as there are client facing reports.
>
> @All
> I had a discussion with my colleague. As per him there is difference in
> formula how Aggregate report and HTMl report calculates 90th percentile.
> Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML
> report is picking 6th value (which is the highest one) as 90 th percentile.
> I need to figure it out ASAP now.
>
>
>
> On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com> wrote:
>
>> Yes, I have seen this small discrepancies also in my tests but I don't
>> adhere to exact values and this reports are just first step during analysis
>> to find an anomaly or something else to focus on , e.g. Is it worth to look
>> at the mean when data is not symmetrically distributed. If it is not
>> symmetrical should I care about standard deviation at all. If my histogram
>> is strange - what was wrong.
>> But surely the meaning of the report depends on the client's need:)
>>
>> Mariusz
>>
>> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com> wrote:
>>
>>> Just chiming in with a higher concurrency test example - not as severe
>>> but still a difference.
>>>
>>> Setting jmeter.reportgenerator.statistic_window = -1 is news to me,
>>> however. I can give that a go and see if it improves things, but these
>>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>>> and are a bit of a nuisance as they essentially limit me to the output
>>> format I choose to share with clients...
>>>
>>> HTML Output:
>>>
>>>
>>> Aggregate Report:
>>>
>>>
>>>
>>> Kieran Lynch
>>>
>>> WM Reply
>>> Nova South
>>> 160 Victoria Street, Westminster
>>> London SW1E 5LB - UK
>>> phone: +44 (0)20 7730 6000
>>> k.lynch@reply.com
>>> www.reply.com
>>>
>>> [image: WM Reply]
>>> ------------------------------
>>> *From:* Abhitosh Patil <ab...@gmail.com>
>>> *Sent:* 26 February 2021 14:28
>>> *To:* JMeter Users List <us...@jmeter.apache.org>
>>> *Subject:* Re: Incorrect 90th percentile values in JMeter dashboard
>>> HTML report
>>>
>>> This is just an example. Actual test was executed with thousands of
>>> samples with different transactions.
>>> It's a transaction controller.
>>> Also I'm using "Result1.csv" format only.
>>> I have automated Aggregate report and HTML report creation. Manual
>>> calculations are not allowed.
>>>
>>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com> wrote:
>>>
>>> Hi,
>>> Yes, indeed results are strange... You have only 6 samples and very wide
>>> distribution. Did You try to get more samples? I think that there is too
>>> small amount of samples to do any descriptive statistics and draw
>>> conclusions about test results and system. What is this sampler in report?
>>> Is it http sampler or Transaction Controller? You can always make your own
>>> calculations in Excel (use -l result1.csv during test and load csv file in
>>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>>> however.
>>>
>>> Mariusz
>>>
>>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I'm generating HTML dashboard report and aggregate report during a load
>>> test.
>>> There is a huge difference in 90th percentile values in aggregate report
>>> and HTML report.
>>> It seems that HTML dashboard report is showing incorrect values.
>>> Can someone help me on this?
>>>
>>> Screenshots:
>>> Aggregate report (90th percentile is 123):
>>> [image: image.png]
>>>
>>> HTML report (90th percentile is 44266):
>>> [image: image.png]
>>>
>>> I tried the solution provided in the post below. But it didn't work:
>>>
>>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>>
>>>
>>> Regards,
>>> AP
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>>> For additional commands, e-mail: user-help@jmeter.apache.org
>>
>>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by Abhitosh Patil <ab...@gmail.com>.
@Kieran Lynch
I'm using JMeter 5.4
I already tried "jmeter.reportgenerator.statistic_window = -1". But it
doesn't work.

@Mariusz W
In my case it's a very huge difference (> 40 seconds) which is not
acceptable at all as there are client facing reports.

@All
I had a discussion with my colleague. As per him there is difference in
formula how Aggregate report and HTMl report calculates 90th percentile.
Aggregate report taking 5th value out of 6 as 90 th percentile. But HTML
report is picking 6th value (which is the highest one) as 90 th percentile.
I need to figure it out ASAP now.



On Fri, Feb 26, 2021 at 10:50 PM Mariusz W <ma...@gmail.com> wrote:

> Yes, I have seen this small discrepancies also in my tests but I don't
> adhere to exact values and this reports are just first step during analysis
> to find an anomaly or something else to focus on , e.g. Is it worth to look
> at the mean when data is not symmetrically distributed. If it is not
> symmetrical should I care about standard deviation at all. If my histogram
> is strange - what was wrong.
> But surely the meaning of the report depends on the client's need:)
>
> Mariusz
>
> On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com> wrote:
>
>> Just chiming in with a higher concurrency test example - not as severe
>> but still a difference.
>>
>> Setting jmeter.reportgenerator.statistic_window = -1 is news to me,
>> however. I can give that a go and see if it improves things, but these
>> discrepancies have been present as long as I've been using jmeter (~v3.2)
>> and are a bit of a nuisance as they essentially limit me to the output
>> format I choose to share with clients...
>>
>> HTML Output:
>>
>>
>> Aggregate Report:
>>
>>
>>
>> Kieran Lynch
>>
>> WM Reply
>> Nova South
>> 160 Victoria Street, Westminster
>> London SW1E 5LB - UK
>> phone: +44 (0)20 7730 6000
>> k.lynch@reply.com
>> www.reply.com
>>
>> [image: WM Reply]
>> ------------------------------
>> *From:* Abhitosh Patil <ab...@gmail.com>
>> *Sent:* 26 February 2021 14:28
>> *To:* JMeter Users List <us...@jmeter.apache.org>
>> *Subject:* Re: Incorrect 90th percentile values in JMeter dashboard HTML
>> report
>>
>> This is just an example. Actual test was executed with thousands of
>> samples with different transactions.
>> It's a transaction controller.
>> Also I'm using "Result1.csv" format only.
>> I have automated Aggregate report and HTML report creation. Manual
>> calculations are not allowed.
>>
>> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com> wrote:
>>
>> Hi,
>> Yes, indeed results are strange... You have only 6 samples and very wide
>> distribution. Did You try to get more samples? I think that there is too
>> small amount of samples to do any descriptive statistics and draw
>> conclusions about test results and system. What is this sampler in report?
>> Is it http sampler or Transaction Controller? You can always make your own
>> calculations in Excel (use -l result1.csv during test and load csv file in
>> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
>> however.
>>
>> Mariusz
>>
>> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com>
>> wrote:
>>
>> Hi,
>>
>> I'm generating HTML dashboard report and aggregate report during a load
>> test.
>> There is a huge difference in 90th percentile values in aggregate report
>> and HTML report.
>> It seems that HTML dashboard report is showing incorrect values.
>> Can someone help me on this?
>>
>> Screenshots:
>> Aggregate report (90th percentile is 123):
>> [image: image.png]
>>
>> HTML report (90th percentile is 44266):
>> [image: image.png]
>>
>> I tried the solution provided in the post below. But it didn't work:
>>
>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>
>>
>> Regards,
>> AP
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> For additional commands, e-mail: user-help@jmeter.apache.org
>
>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by Mariusz W <ma...@gmail.com>.
Yes, I have seen this small discrepancies also in my tests but I don't
adhere to exact values and this reports are just first step during analysis
to find an anomaly or something else to focus on , e.g. Is it worth to look
at the mean when data is not symmetrically distributed. If it is not
symmetrical should I care about standard deviation at all. If my histogram
is strange - what was wrong.
But surely the meaning of the report depends on the client's need:)

Mariusz

On Fri, 26 Feb 2021 at 15:52, Lynch Kieran <k....@reply.com> wrote:

> Just chiming in with a higher concurrency test example - not as severe but
> still a difference.
>
> Setting jmeter.reportgenerator.statistic_window = -1 is news to me,
> however. I can give that a go and see if it improves things, but these
> discrepancies have been present as long as I've been using jmeter (~v3.2)
> and are a bit of a nuisance as they essentially limit me to the output
> format I choose to share with clients...
>
> HTML Output:
>
>
> Aggregate Report:
>
>
>
> Kieran Lynch
>
> WM Reply
> Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.lynch@reply.com
> www.reply.com
>
> [image: WM Reply]
> ------------------------------
> *From:* Abhitosh Patil <ab...@gmail.com>
> *Sent:* 26 February 2021 14:28
> *To:* JMeter Users List <us...@jmeter.apache.org>
> *Subject:* Re: Incorrect 90th percentile values in JMeter dashboard HTML
> report
>
> This is just an example. Actual test was executed with thousands of
> samples with different transactions.
> It's a transaction controller.
> Also I'm using "Result1.csv" format only.
> I have automated Aggregate report and HTML report creation. Manual
> calculations are not allowed.
>
> On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com> wrote:
>
> Hi,
> Yes, indeed results are strange... You have only 6 samples and very wide
> distribution. Did You try to get more samples? I think that there is too
> small amount of samples to do any descriptive statistics and draw
> conclusions about test results and system. What is this sampler in report?
> Is it http sampler or Transaction Controller? You can always make your own
> calculations in Excel (use -l result1.csv during test and load csv file in
> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
> however.
>
> Mariusz
>
> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com>
> wrote:
>
> Hi,
>
> I'm generating HTML dashboard report and aggregate report during a load
> test.
> There is a huge difference in 90th percentile values in aggregate report
> and HTML report.
> It seems that HTML dashboard report is showing incorrect values.
> Can someone help me on this?
>
> Screenshots:
> Aggregate report (90th percentile is 123):
> [image: image.png]
>
> HTML report (90th percentile is 44266):
> [image: image.png]
>
> I tried the solution provided in the post below. But it didn't work:
>
> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>
>
> Regards,
> AP
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by Lynch Kieran <k....@reply.com>.
Just chiming in with a higher concurrency test example - not as severe but still a difference.

Setting jmeter.reportgenerator.statistic_window = -1 is news to me, however. I can give that a go and see if it improves things, but these discrepancies have been present as long as I've been using jmeter (~v3.2) and are a bit of a nuisance as they essentially limit me to the output format I choose to share with clients...

HTML Output:
[cid:821d3dd6-91b3-4c5c-8be7-6b30aa5054e9]

Aggregate Report:
[cid:0227a6a3-ced7-4ee7-b82f-11c90be69696]


Kieran Lynch

WM Reply
Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK
phone: +44 (0)20 7730 6000
k.lynch@reply.com<ma...@reply.com>
www.reply.com<http://www.reply.com> <http://>

[WM Reply]
________________________________
From: Abhitosh Patil <ab...@gmail.com>
Sent: 26 February 2021 14:28
To: JMeter Users List <us...@jmeter.apache.org>
Subject: Re: Incorrect 90th percentile values in JMeter dashboard HTML report

This is just an example. Actual test was executed with thousands of samples with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com>> wrote:
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide distribution. Did You try to get more samples? I think that there is too small amount of samples to do any descriptive statistics and draw conclusions about test results and system. What is this sampler in report? Is it http sampler or Transaction Controller? You can always make your own  calculations in Excel (use -l result1.csv during test and load csv file in Excel). In excel you cannot use PERCENTILE.EXC on small number of samples however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com>> wrote:
Hi,

I'm generating HTML dashboard report and aggregate report during a load test.
There is a huge difference in 90th percentile values in aggregate report and HTML report.
It seems that HTML dashboard report is showing incorrect values.
Can someone help me on this?

Screenshots:
Aggregate report (90th percentile is 123):
[image.png]

HTML report (90th percentile is 44266):
[image.png]

I tried the solution provided in the post below. But it didn't work:
https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti


Regards,
AP

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by Abhitosh Patil <ab...@gmail.com>.
This is just an example. Actual test was executed with thousands of samples
with different transactions.
It's a transaction controller.
Also I'm using "Result1.csv" format only.
I have automated Aggregate report and HTML report creation. Manual
calculations are not allowed.

On Fri, Feb 26, 2021 at 7:48 PM Mariusz W <ma...@gmail.com> wrote:

> Hi,
> Yes, indeed results are strange... You have only 6 samples and very wide
> distribution. Did You try to get more samples? I think that there is too
> small amount of samples to do any descriptive statistics and draw
> conclusions about test results and system. What is this sampler in report?
> Is it http sampler or Transaction Controller? You can always make your own
> calculations in Excel (use -l result1.csv during test and load csv file in
> Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
> however.
>
> Mariusz
>
> On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm generating HTML dashboard report and aggregate report during a load
>> test.
>> There is a huge difference in 90th percentile values in aggregate report
>> and HTML report.
>> It seems that HTML dashboard report is showing incorrect values.
>> Can someone help me on this?
>>
>> Screenshots:
>> Aggregate report (90th percentile is 123):
>> [image: image.png]
>>
>> HTML report (90th percentile is 44266):
>> [image: image.png]
>>
>> I tried the solution provided in the post below. But it didn't work:
>>
>> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>>
>>
>> Regards,
>> AP
>>
>

Re: Incorrect 90th percentile values in JMeter dashboard HTML report

Posted by Mariusz W <ma...@gmail.com>.
Hi,
Yes, indeed results are strange... You have only 6 samples and very wide
distribution. Did You try to get more samples? I think that there is too
small amount of samples to do any descriptive statistics and draw
conclusions about test results and system. What is this sampler in report?
Is it http sampler or Transaction Controller? You can always make your own
calculations in Excel (use -l result1.csv during test and load csv file in
Excel). In excel you cannot use PERCENTILE.EXC on small number of samples
however.

Mariusz

On Fri, 26 Feb 2021 at 14:15, Abhitosh Patil <ab...@gmail.com> wrote:

> Hi,
>
> I'm generating HTML dashboard report and aggregate report during a load
> test.
> There is a huge difference in 90th percentile values in aggregate report
> and HTML report.
> It seems that HTML dashboard report is showing incorrect values.
> Can someone help me on this?
>
> Screenshots:
> Aggregate report (90th percentile is 123):
> [image: image.png]
>
> HTML report (90th percentile is 44266):
> [image: image.png]
>
> I tried the solution provided in the post below. But it didn't work:
>
> https://dzone.com/articles/how-to-achieve-better-accuracy-in-latency-percenti
>
>
> Regards,
> AP
>