You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Deepak Chaudhari <de...@gmail.com> on 2021/06/02 13:28:23 UTC
Wrong 90th and 95th percentile values in JMeter HTML report
Hi,
I collected JTL file during a test run and using that I'm generating HTML
report and aggregate report. When I compare aggregate and HTML reports I
found that 90th and 95th percentile values are wrong in HTML report.
I tried to change "jmeter.reportgenerator.statistic_window" value in
user.properties but still getting wrong values.
*Aggregate report:*
[image: image.png]
*HTML report:*
[image: image.png]
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by Deepak Chaudhari <de...@gmail.com>.
Indeed. :)
On Fri, Jun 4, 2021 at 11:52 PM Philippe Mouawad <
p.mouawad@ubik-ingenierie.com> wrote:
> You rock Felix !
>
> On Fri, Jun 4, 2021 at 5:39 PM Deepak Chaudhari <de...@gmail.com>
> wrote:
>
>> I downloaded JMeter 5.5 and compared aggregate and HTML reports.
>> Now the results are exactly the same
>>
>> Thank you very much Felix for fixing it on such short notice. :)
>>
>> [image: image.png]
>>
>> [image: image.png]
>> .
>>
>> On Thu, Jun 3, 2021 at 11:14 PM Felix Schumacher <
>> felix.schumacher@internetallee.de> wrote:
>>
>>> You might find https://bz.apache.org/bugzilla/show_bug.cgi?id=65353
>>> interesting.
>>>
>>> With the next nightly build or build from trunk, you should be able to use the new property "backend_metrics_percentile_estimator" with a value of "R_3" to lessen the difference between both reports. But keep in mind, that the HTML Report use a sliding window, while the Aggregate Report does not (and will eventually OOM).
>>>
>>> Felix
>>>
>>>
>>> Am 03.06.21 um 17:46 schrieb Deepak Chaudhari:
>>>
>>> I need to send both the reports to the client and surely the question
>>> will come "Why is there so much difference?"
>>> If there is any way with which we can generate almost similar aggregate
>>> and HTML reports?
>>>
>>> On Thu, Jun 3, 2021 at 9:00 PM Felix Schumacher <
>>> felix.schumacher@internetallee.de> wrote:
>>>
>>>> Without the actual data it is impossible to say, if the reports are
>>>> wrong, or if they follow different ways to calculate a percentile.
>>>>
>>>> The problem here is, that the two reports are using different
>>>> algorithms to calculate the percentiles. We are working with discrete
>>>> numbers and here a percentile is more like a range than a single (correct
>>>> or wrong) number. Say, you have the values [1, 2, 4, 8, 16, 32, 64, 128,
>>>> 256, 512] want to calculate a 90% percentile. For this you take any number
>>>> that splits the (sorted) list into two lists, where one list has all
>>>> elements, that are smaller (or equal) to the number and one list contains
>>>> all elements that are bigger than the number. For this example, any number
>>>> between 256 and 511.99 would be a valid 90% percentile. (that is of course
>>>> a broad description only)
>>>>
>>>> The Aggregate report currently uses a memory intensive way and stores a
>>>> list with all values and picks the smallest number, that fits the above
>>>> description. (This memory hogging behaviour should be fixed and would lead
>>>> to less accurate data and less OOMs or slow JMeter instances)
>>>>
>>>> The HTML report gives you a value that would probably be a good 90%
>>>> percentile, if you would have more data like the one, you already have by
>>>> picking some value between the lower bound and the upper bound of the
>>>> described range (in our example something between 256 and 511.999).
>>>>
>>>> As your report has very few samples (21 for the biggest difference in
>>>> the numbers), the skew between the reported percentiles can look big, but
>>>> are not necessarily wrong.
>>>>
>>>> Note, that if the numbers that both reports tell you are far apart,
>>>> that is probably a sign of having either too few samples, or the samples
>>>> (durations of the samples) are distributed sparsely near the percentile
>>>> ranges.
>>>>
>>>> We have discussed in the past to use the same algorithm for both
>>>> components, so that you get consistent values, but there were always other
>>>> issues, that seemed to be more important.
>>>>
>>>> Felix
>>>> Am 02.06.21 um 15:28 schrieb Deepak Chaudhari:
>>>>
>>>> Hi,
>>>>
>>>> I collected JTL file during a test run and using that I'm generating
>>>> HTML report and aggregate report. When I compare aggregate and HTML reports
>>>> I found that 90th and 95th percentile values are wrong in HTML report.
>>>> I tried to change "jmeter.reportgenerator.statistic_window" value in
>>>> user.properties but still getting wrong values.
>>>>
>>>> *Aggregate report:*
>>>> [image: image.png]
>>>>
>>>> *HTML report:*
>>>> [image: image.png]
>>>>
>>>>
>
> --
> Cordialement
> Philippe M.
> Ubik-Ingenierie
>
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by Philippe Mouawad <p....@ubik-ingenierie.com>.
You rock Felix !
On Fri, Jun 4, 2021 at 5:39 PM Deepak Chaudhari <de...@gmail.com>
wrote:
> I downloaded JMeter 5.5 and compared aggregate and HTML reports.
> Now the results are exactly the same
>
> Thank you very much Felix for fixing it on such short notice. :)
>
> [image: image.png]
>
> [image: image.png]
> .
>
> On Thu, Jun 3, 2021 at 11:14 PM Felix Schumacher <
> felix.schumacher@internetallee.de> wrote:
>
>> You might find https://bz.apache.org/bugzilla/show_bug.cgi?id=65353
>> interesting.
>>
>> With the next nightly build or build from trunk, you should be able to use the new property "backend_metrics_percentile_estimator" with a value of "R_3" to lessen the difference between both reports. But keep in mind, that the HTML Report use a sliding window, while the Aggregate Report does not (and will eventually OOM).
>>
>> Felix
>>
>>
>> Am 03.06.21 um 17:46 schrieb Deepak Chaudhari:
>>
>> I need to send both the reports to the client and surely the question
>> will come "Why is there so much difference?"
>> If there is any way with which we can generate almost similar aggregate
>> and HTML reports?
>>
>> On Thu, Jun 3, 2021 at 9:00 PM Felix Schumacher <
>> felix.schumacher@internetallee.de> wrote:
>>
>>> Without the actual data it is impossible to say, if the reports are
>>> wrong, or if they follow different ways to calculate a percentile.
>>>
>>> The problem here is, that the two reports are using different algorithms
>>> to calculate the percentiles. We are working with discrete numbers and here
>>> a percentile is more like a range than a single (correct or wrong) number.
>>> Say, you have the values [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] want to
>>> calculate a 90% percentile. For this you take any number that splits the
>>> (sorted) list into two lists, where one list has all elements, that are
>>> smaller (or equal) to the number and one list contains all elements that
>>> are bigger than the number. For this example, any number between 256 and
>>> 511.99 would be a valid 90% percentile. (that is of course a broad
>>> description only)
>>>
>>> The Aggregate report currently uses a memory intensive way and stores a
>>> list with all values and picks the smallest number, that fits the above
>>> description. (This memory hogging behaviour should be fixed and would lead
>>> to less accurate data and less OOMs or slow JMeter instances)
>>>
>>> The HTML report gives you a value that would probably be a good 90%
>>> percentile, if you would have more data like the one, you already have by
>>> picking some value between the lower bound and the upper bound of the
>>> described range (in our example something between 256 and 511.999).
>>>
>>> As your report has very few samples (21 for the biggest difference in
>>> the numbers), the skew between the reported percentiles can look big, but
>>> are not necessarily wrong.
>>>
>>> Note, that if the numbers that both reports tell you are far apart, that
>>> is probably a sign of having either too few samples, or the samples
>>> (durations of the samples) are distributed sparsely near the percentile
>>> ranges.
>>>
>>> We have discussed in the past to use the same algorithm for both
>>> components, so that you get consistent values, but there were always other
>>> issues, that seemed to be more important.
>>>
>>> Felix
>>> Am 02.06.21 um 15:28 schrieb Deepak Chaudhari:
>>>
>>> Hi,
>>>
>>> I collected JTL file during a test run and using that I'm generating
>>> HTML report and aggregate report. When I compare aggregate and HTML reports
>>> I found that 90th and 95th percentile values are wrong in HTML report.
>>> I tried to change "jmeter.reportgenerator.statistic_window" value in
>>> user.properties but still getting wrong values.
>>>
>>> *Aggregate report:*
>>> [image: image.png]
>>>
>>> *HTML report:*
>>> [image: image.png]
>>>
>>>
--
Cordialement
Philippe M.
Ubik-Ingenierie
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by Deepak Chaudhari <de...@gmail.com>.
I downloaded JMeter 5.5 and compared aggregate and HTML reports.
Now the results are exactly the same
Thank you very much Felix for fixing it on such short notice. :)
[image: image.png]
[image: image.png]
.
On Thu, Jun 3, 2021 at 11:14 PM Felix Schumacher <
felix.schumacher@internetallee.de> wrote:
> You might find https://bz.apache.org/bugzilla/show_bug.cgi?id=65353
> interesting.
>
> With the next nightly build or build from trunk, you should be able to use the new property "backend_metrics_percentile_estimator" with a value of "R_3" to lessen the difference between both reports. But keep in mind, that the HTML Report use a sliding window, while the Aggregate Report does not (and will eventually OOM).
>
> Felix
>
>
> Am 03.06.21 um 17:46 schrieb Deepak Chaudhari:
>
> I need to send both the reports to the client and surely the question will
> come "Why is there so much difference?"
> If there is any way with which we can generate almost similar aggregate
> and HTML reports?
>
> On Thu, Jun 3, 2021 at 9:00 PM Felix Schumacher <
> felix.schumacher@internetallee.de> wrote:
>
>> Without the actual data it is impossible to say, if the reports are
>> wrong, or if they follow different ways to calculate a percentile.
>>
>> The problem here is, that the two reports are using different algorithms
>> to calculate the percentiles. We are working with discrete numbers and here
>> a percentile is more like a range than a single (correct or wrong) number.
>> Say, you have the values [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] want to
>> calculate a 90% percentile. For this you take any number that splits the
>> (sorted) list into two lists, where one list has all elements, that are
>> smaller (or equal) to the number and one list contains all elements that
>> are bigger than the number. For this example, any number between 256 and
>> 511.99 would be a valid 90% percentile. (that is of course a broad
>> description only)
>>
>> The Aggregate report currently uses a memory intensive way and stores a
>> list with all values and picks the smallest number, that fits the above
>> description. (This memory hogging behaviour should be fixed and would lead
>> to less accurate data and less OOMs or slow JMeter instances)
>>
>> The HTML report gives you a value that would probably be a good 90%
>> percentile, if you would have more data like the one, you already have by
>> picking some value between the lower bound and the upper bound of the
>> described range (in our example something between 256 and 511.999).
>>
>> As your report has very few samples (21 for the biggest difference in the
>> numbers), the skew between the reported percentiles can look big, but are
>> not necessarily wrong.
>>
>> Note, that if the numbers that both reports tell you are far apart, that
>> is probably a sign of having either too few samples, or the samples
>> (durations of the samples) are distributed sparsely near the percentile
>> ranges.
>>
>> We have discussed in the past to use the same algorithm for both
>> components, so that you get consistent values, but there were always other
>> issues, that seemed to be more important.
>>
>> Felix
>> Am 02.06.21 um 15:28 schrieb Deepak Chaudhari:
>>
>> Hi,
>>
>> I collected JTL file during a test run and using that I'm generating HTML
>> report and aggregate report. When I compare aggregate and HTML reports I
>> found that 90th and 95th percentile values are wrong in HTML report.
>> I tried to change "jmeter.reportgenerator.statistic_window" value in
>> user.properties but still getting wrong values.
>>
>> *Aggregate report:*
>> [image: image.png]
>>
>> *HTML report:*
>> [image: image.png]
>>
>>
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by Deepak Chaudhari <de...@gmail.com>.
Thanks for informing Prateek.
I'll download from here.
On Fri, Jun 4, 2021 at 5:16 PM Prateek Dua <pr...@go-mmt.com.invalid>
wrote:
> Hi Deepak,
>
> You need to download from here
> https://ci-builds.apache.org/job/JMeter/job/JMeter-trunk/ .
>
> JMeter 5.4.1 is the current latest version which I guess don't have the
> changes you're looking for.
>
>
> Thanks,
> Prateek
> ------------------------------
> *From:* Deepak Chaudhari <de...@gmail.com>
> *Sent:* 04 June 2021 16:15
> *To:* JMeter Users List <us...@jmeter.apache.org>
> *Subject:* Re: Wrong 90th and 95th percentile values in JMeter HTML report
>
> Is it implemented or not yet?
> I downloaded JMeter 5.4.1 and added property
> "backend_metrics_percentile_estimator = R_3" in user.properties
> Still same results.
>
> [image: image.png]
>
> [image: image.png]
>
> On Thu, Jun 3, 2021 at 11:43 PM Felix Schumacher <
> felix.schumacher@internetallee.de> wrote:
>
>
> Am 03.06.21 um 20:10 schrieb Deepak Chaudhari:
>
> Thank you very much for your reply.
> Will it be in user.properties?
> Do we need to install the latest JMeter version to reflect the changes?
>
> A build from trunk can be found at
> https://ci-builds.apache.org/job/JMeter/job/JMeter-trunk/
>
> Yes, the property can be set via user.properties.
>
> Felix
>
>
> On Thu, Jun 3, 2021 at 11:14 PM Felix Schumacher <
> felix.schumacher@internetallee.de> wrote:
>
> You might find https://bz.apache.org/bugzilla/show_bug.cgi?id=65353
> interesting.
>
> With the next nightly build or build from trunk, you should be able to use the new property "backend_metrics_percentile_estimator" with a value of "R_3" to lessen the difference between both reports. But keep in mind, that the HTML Report use a sliding window, while the Aggregate Report does not (and will eventually OOM).
>
> Felix
>
>
> Am 03.06.21 um 17:46 schrieb Deepak Chaudhari:
>
> I need to send both the reports to the client and surely the question will
> come "Why is there so much difference?"
> If there is any way with which we can generate almost similar aggregate
> and HTML reports?
>
> On Thu, Jun 3, 2021 at 9:00 PM Felix Schumacher <
> felix.schumacher@internetallee.de> wrote:
>
> Without the actual data it is impossible to say, if the reports are wrong,
> or if they follow different ways to calculate a percentile.
>
> The problem here is, that the two reports are using different algorithms
> to calculate the percentiles. We are working with discrete numbers and here
> a percentile is more like a range than a single (correct or wrong) number.
> Say, you have the values [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] want to
> calculate a 90% percentile. For this you take any number that splits the
> (sorted) list into two lists, where one list has all elements, that are
> smaller (or equal) to the number and one list contains all elements that
> are bigger than the number. For this example, any number between 256 and
> 511.99 would be a valid 90% percentile. (that is of course a broad
> description only)
>
> The Aggregate report currently uses a memory intensive way and stores a
> list with all values and picks the smallest number, that fits the above
> description. (This memory hogging behaviour should be fixed and would lead
> to less accurate data and less OOMs or slow JMeter instances)
>
> The HTML report gives you a value that would probably be a good 90%
> percentile, if you would have more data like the one, you already have by
> picking some value between the lower bound and the upper bound of the
> described range (in our example something between 256 and 511.999).
>
> As your report has very few samples (21 for the biggest difference in the
> numbers), the skew between the reported percentiles can look big, but are
> not necessarily wrong.
>
> Note, that if the numbers that both reports tell you are far apart, that
> is probably a sign of having either too few samples, or the samples
> (durations of the samples) are distributed sparsely near the percentile
> ranges.
>
> We have discussed in the past to use the same algorithm for both
> components, so that you get consistent values, but there were always other
> issues, that seemed to be more important.
>
> Felix
> Am 02.06.21 um 15:28 schrieb Deepak Chaudhari:
>
> Hi,
>
> I collected JTL file during a test run and using that I'm generating HTML
> report and aggregate report. When I compare aggregate and HTML reports I
> found that 90th and 95th percentile values are wrong in HTML report.
> I tried to change "jmeter.reportgenerator.statistic_window" value in
> user.properties but still getting wrong values.
>
> *Aggregate report:*
> [image: image.png]
>
> *HTML report:*
> [image: image.png]
>
>
> The contents of this email, including the attachments, are *privileged
> and confidential* to the intended recipient at the email address to which
> it has been addressed. If you receive it in error, please notify the sender
> immediately by return email and then permanently delete it from your
> system. The unauthorized use, distribution, copying or alteration of this
> email, including the attachments, is strictly forbidden. Please note that
> neither the sender nor the company accepts any responsibility for viruses
> and it is your responsibility to scan the email and attachments (if any).
>
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by Prateek Dua <pr...@go-mmt.com.INVALID>.
Hi Deepak,
You need to download from here https://ci-builds.apache.org/job/JMeter/job/JMeter-trunk/ .
JMeter 5.4.1 is the current latest version which I guess don't have the changes you're looking for.
Thanks,
Prateek
________________________________
From: Deepak Chaudhari <de...@gmail.com>
Sent: 04 June 2021 16:15
To: JMeter Users List <us...@jmeter.apache.org>
Subject: Re: Wrong 90th and 95th percentile values in JMeter HTML report
Is it implemented or not yet?
I downloaded JMeter 5.4.1 and added property "backend_metrics_percentile_estimator = R_3" in user.properties
Still same results.
[image.png]
[image.png]
On Thu, Jun 3, 2021 at 11:43 PM Felix Schumacher <fe...@internetallee.de>> wrote:
Am 03.06.21 um 20:10 schrieb Deepak Chaudhari:
Thank you very much for your reply.
Will it be in user.properties?
Do we need to install the latest JMeter version to reflect the changes?
A build from trunk can be found at https://ci-builds.apache.org/job/JMeter/job/JMeter-trunk/
Yes, the property can be set via user.properties.
Felix
On Thu, Jun 3, 2021 at 11:14 PM Felix Schumacher <fe...@internetallee.de>> wrote:
You might find
https://bz.apache.org/bugzilla/show_bug.cgi?id=65353
interesting.
With the next nightly build or build from trunk, you should be able to use the new property "backend_metrics_percentile_estimator" with a value of "R_3" to lessen the difference between both reports. But keep in mind, that the HTML Report use a sliding window, while the Aggregate Report does not (and will eventually OOM).
Felix
Am 03.06.21 um 17:46 schrieb Deepak Chaudhari:
I need to send both the reports to the client and surely the question will come "Why is there so much difference?"
If there is any way with which we can generate almost similar aggregate and HTML reports?
On Thu, Jun 3, 2021 at 9:00 PM Felix Schumacher <fe...@internetallee.de>> wrote:
Without the actual data it is impossible to say, if the reports are wrong, or if they follow different ways to calculate a percentile.
The problem here is, that the two reports are using different algorithms to calculate the percentiles. We are working with discrete numbers and here a percentile is more like a range than a single (correct or wrong) number. Say, you have the values [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] want to calculate a 90% percentile. For this you take any number that splits the (sorted) list into two lists, where one list has all elements, that are smaller (or equal) to the number and one list contains all elements that are bigger than the number. For this example, any number between 256 and 511.99 would be a valid 90% percentile. (that is of course a broad description only)
The Aggregate report currently uses a memory intensive way and stores a list with all values and picks the smallest number, that fits the above description. (This memory hogging behaviour should be fixed and would lead to less accurate data and less OOMs or slow JMeter instances)
The HTML report gives you a value that would probably be a good 90% percentile, if you would have more data like the one, you already have by picking some value between the lower bound and the upper bound of the described range (in our example something between 256 and 511.999).
As your report has very few samples (21 for the biggest difference in the numbers), the skew between the reported percentiles can look big, but are not necessarily wrong.
Note, that if the numbers that both reports tell you are far apart, that is probably a sign of having either too few samples, or the samples (durations of the samples) are distributed sparsely near the percentile ranges.
We have discussed in the past to use the same algorithm for both components, so that you get consistent values, but there were always other issues, that seemed to be more important.
Felix
Am 02.06.21 um 15:28 schrieb Deepak Chaudhari:
Hi,
I collected JTL file during a test run and using that I'm generating HTML report and aggregate report. When I compare aggregate and HTML reports I found that 90th and 95th percentile values are wrong in HTML report.
I tried to change "jmeter.reportgenerator.statistic_window" value in user.properties but still getting wrong values.
Aggregate report:
[image.png]
HTML report:
[image.png]
The contents of this email, including the attachments, are privileged and confidential to the intended recipient at the email address to which it has been addressed. If you receive it in error, please notify the sender immediately by return email and then permanently delete it from your system. The unauthorized use, distribution, copying or alteration of this email, including the attachments, is strictly forbidden. Please note that neither the sender nor the company accepts any responsibility for viruses and it is your responsibility to scan the email and attachments (if any).
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by Deepak Chaudhari <de...@gmail.com>.
Is it implemented or not yet?
I downloaded JMeter 5.4.1 and added property
"backend_metrics_percentile_estimator = R_3" in user.properties
Still same results.
[image: image.png]
[image: image.png]
On Thu, Jun 3, 2021 at 11:43 PM Felix Schumacher <
felix.schumacher@internetallee.de> wrote:
>
> Am 03.06.21 um 20:10 schrieb Deepak Chaudhari:
>
> Thank you very much for your reply.
> Will it be in user.properties?
> Do we need to install the latest JMeter version to reflect the changes?
>
> A build from trunk can be found at
> https://ci-builds.apache.org/job/JMeter/job/JMeter-trunk/
>
> Yes, the property can be set via user.properties.
>
> Felix
>
>
> On Thu, Jun 3, 2021 at 11:14 PM Felix Schumacher <
> felix.schumacher@internetallee.de> wrote:
>
>> You might find https://bz.apache.org/bugzilla/show_bug.cgi?id=65353
>> interesting.
>>
>> With the next nightly build or build from trunk, you should be able to use the new property "backend_metrics_percentile_estimator" with a value of "R_3" to lessen the difference between both reports. But keep in mind, that the HTML Report use a sliding window, while the Aggregate Report does not (and will eventually OOM).
>>
>> Felix
>>
>>
>> Am 03.06.21 um 17:46 schrieb Deepak Chaudhari:
>>
>> I need to send both the reports to the client and surely the question
>> will come "Why is there so much difference?"
>> If there is any way with which we can generate almost similar aggregate
>> and HTML reports?
>>
>> On Thu, Jun 3, 2021 at 9:00 PM Felix Schumacher <
>> felix.schumacher@internetallee.de> wrote:
>>
>>> Without the actual data it is impossible to say, if the reports are
>>> wrong, or if they follow different ways to calculate a percentile.
>>>
>>> The problem here is, that the two reports are using different algorithms
>>> to calculate the percentiles. We are working with discrete numbers and here
>>> a percentile is more like a range than a single (correct or wrong) number.
>>> Say, you have the values [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] want to
>>> calculate a 90% percentile. For this you take any number that splits the
>>> (sorted) list into two lists, where one list has all elements, that are
>>> smaller (or equal) to the number and one list contains all elements that
>>> are bigger than the number. For this example, any number between 256 and
>>> 511.99 would be a valid 90% percentile. (that is of course a broad
>>> description only)
>>>
>>> The Aggregate report currently uses a memory intensive way and stores a
>>> list with all values and picks the smallest number, that fits the above
>>> description. (This memory hogging behaviour should be fixed and would lead
>>> to less accurate data and less OOMs or slow JMeter instances)
>>>
>>> The HTML report gives you a value that would probably be a good 90%
>>> percentile, if you would have more data like the one, you already have by
>>> picking some value between the lower bound and the upper bound of the
>>> described range (in our example something between 256 and 511.999).
>>>
>>> As your report has very few samples (21 for the biggest difference in
>>> the numbers), the skew between the reported percentiles can look big, but
>>> are not necessarily wrong.
>>>
>>> Note, that if the numbers that both reports tell you are far apart, that
>>> is probably a sign of having either too few samples, or the samples
>>> (durations of the samples) are distributed sparsely near the percentile
>>> ranges.
>>>
>>> We have discussed in the past to use the same algorithm for both
>>> components, so that you get consistent values, but there were always other
>>> issues, that seemed to be more important.
>>>
>>> Felix
>>> Am 02.06.21 um 15:28 schrieb Deepak Chaudhari:
>>>
>>> Hi,
>>>
>>> I collected JTL file during a test run and using that I'm generating
>>> HTML report and aggregate report. When I compare aggregate and HTML reports
>>> I found that 90th and 95th percentile values are wrong in HTML report.
>>> I tried to change "jmeter.reportgenerator.statistic_window" value in
>>> user.properties but still getting wrong values.
>>>
>>> *Aggregate report:*
>>> [image: image.png]
>>>
>>> *HTML report:*
>>> [image: image.png]
>>>
>>>
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by Felix Schumacher <fe...@internetallee.de>.
Am 03.06.21 um 20:10 schrieb Deepak Chaudhari:
> Thank you very much for your reply.
> Will it be in user.properties?
> Do we need to install the latest JMeter version to reflect the changes?
A build from trunk can be found at
https://ci-builds.apache.org/job/JMeter/job/JMeter-trunk/
Yes, the property can be set via user.properties.
Felix
>
> On Thu, Jun 3, 2021 at 11:14 PM Felix Schumacher
> <felix.schumacher@internetallee.de
> <ma...@internetallee.de>> wrote:
>
> You might find
> https://bz.apache.org/bugzilla/show_bug.cgi?id=65353 <https://bz.apache.org/bugzilla/show_bug.cgi?id=65353>
> interesting.
>
> With the next nightly build or build from trunk, you should be able to use the new property "backend_metrics_percentile_estimator" with a value of "R_3" to lessen the difference between both reports. But keep in mind, that the HTML Report use a sliding window, while the Aggregate Report does not (and will eventually OOM).
>
> Felix
>
> Am 03.06.21 um 17:46 schrieb Deepak Chaudhari:
>> I need to send both the reports to the client and surely the
>> question will come "Why is there so much difference?"
>> If there is any way with which we can generate almost similar
>> aggregate and HTML reports?
>>
>> On Thu, Jun 3, 2021 at 9:00 PM Felix Schumacher
>> <felix.schumacher@internetallee.de
>> <ma...@internetallee.de>> wrote:
>>
>> Without the actual data it is impossible to say, if the
>> reports are wrong, or if they follow different ways to
>> calculate a percentile.
>>
>> The problem here is, that the two reports are using different
>> algorithms to calculate the percentiles. We are working with
>> discrete numbers and here a percentile is more like a range
>> than a single (correct or wrong) number. Say, you have the
>> values [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] want to
>> calculate a 90% percentile. For this you take any number that
>> splits the (sorted) list into two lists, where one list has
>> all elements, that are smaller (or equal) to the number and
>> one list contains all elements that are bigger than the
>> number. For this example, any number between 256 and 511.99
>> would be a valid 90% percentile. (that is of course a broad
>> description only)
>>
>> The Aggregate report currently uses a memory intensive way
>> and stores a list with all values and picks the smallest
>> number, that fits the above description. (This memory hogging
>> behaviour should be fixed and would lead to less accurate
>> data and less OOMs or slow JMeter instances)
>>
>> The HTML report gives you a value that would probably be a
>> good 90% percentile, if you would have more data like the
>> one, you already have by picking some value between the lower
>> bound and the upper bound of the described range (in our
>> example something between 256 and 511.999).
>>
>> As your report has very few samples (21 for the biggest
>> difference in the numbers), the skew between the reported
>> percentiles can look big, but are not necessarily wrong.
>>
>> Note, that if the numbers that both reports tell you are far
>> apart, that is probably a sign of having either too few
>> samples, or the samples (durations of the samples) are
>> distributed sparsely near the percentile ranges.
>>
>> We have discussed in the past to use the same algorithm for
>> both components, so that you get consistent values, but there
>> were always other issues, that seemed to be more important.
>>
>> Felix
>>
>> Am 02.06.21 um 15:28 schrieb Deepak Chaudhari:
>>> Hi,
>>>
>>> I collected JTL file during a test run and using that I'm
>>> generating HTML report and aggregate report. When I compare
>>> aggregate and HTML reports I found that 90th and 95th
>>> percentile values are wrong in HTML report.
>>> I tried to change "jmeter.reportgenerator.statistic_window"
>>> value in user.properties but still getting wrong values.
>>>
>>> *Aggregate report:*
>>> image.png
>>>
>>> *HTML report:*
>>> image.png
>>
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by Deepak Chaudhari <de...@gmail.com>.
Thank you very much for your reply.
Will it be in user.properties?
Do we need to install the latest JMeter version to reflect the changes?
On Thu, Jun 3, 2021 at 11:14 PM Felix Schumacher <
felix.schumacher@internetallee.de> wrote:
> You might find https://bz.apache.org/bugzilla/show_bug.cgi?id=65353
> interesting.
>
> With the next nightly build or build from trunk, you should be able to use the new property "backend_metrics_percentile_estimator" with a value of "R_3" to lessen the difference between both reports. But keep in mind, that the HTML Report use a sliding window, while the Aggregate Report does not (and will eventually OOM).
>
> Felix
>
>
> Am 03.06.21 um 17:46 schrieb Deepak Chaudhari:
>
> I need to send both the reports to the client and surely the question will
> come "Why is there so much difference?"
> If there is any way with which we can generate almost similar aggregate
> and HTML reports?
>
> On Thu, Jun 3, 2021 at 9:00 PM Felix Schumacher <
> felix.schumacher@internetallee.de> wrote:
>
>> Without the actual data it is impossible to say, if the reports are
>> wrong, or if they follow different ways to calculate a percentile.
>>
>> The problem here is, that the two reports are using different algorithms
>> to calculate the percentiles. We are working with discrete numbers and here
>> a percentile is more like a range than a single (correct or wrong) number.
>> Say, you have the values [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] want to
>> calculate a 90% percentile. For this you take any number that splits the
>> (sorted) list into two lists, where one list has all elements, that are
>> smaller (or equal) to the number and one list contains all elements that
>> are bigger than the number. For this example, any number between 256 and
>> 511.99 would be a valid 90% percentile. (that is of course a broad
>> description only)
>>
>> The Aggregate report currently uses a memory intensive way and stores a
>> list with all values and picks the smallest number, that fits the above
>> description. (This memory hogging behaviour should be fixed and would lead
>> to less accurate data and less OOMs or slow JMeter instances)
>>
>> The HTML report gives you a value that would probably be a good 90%
>> percentile, if you would have more data like the one, you already have by
>> picking some value between the lower bound and the upper bound of the
>> described range (in our example something between 256 and 511.999).
>>
>> As your report has very few samples (21 for the biggest difference in the
>> numbers), the skew between the reported percentiles can look big, but are
>> not necessarily wrong.
>>
>> Note, that if the numbers that both reports tell you are far apart, that
>> is probably a sign of having either too few samples, or the samples
>> (durations of the samples) are distributed sparsely near the percentile
>> ranges.
>>
>> We have discussed in the past to use the same algorithm for both
>> components, so that you get consistent values, but there were always other
>> issues, that seemed to be more important.
>>
>> Felix
>> Am 02.06.21 um 15:28 schrieb Deepak Chaudhari:
>>
>> Hi,
>>
>> I collected JTL file during a test run and using that I'm generating HTML
>> report and aggregate report. When I compare aggregate and HTML reports I
>> found that 90th and 95th percentile values are wrong in HTML report.
>> I tried to change "jmeter.reportgenerator.statistic_window" value in
>> user.properties but still getting wrong values.
>>
>> *Aggregate report:*
>> [image: image.png]
>>
>> *HTML report:*
>> [image: image.png]
>>
>>
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by Felix Schumacher <fe...@internetallee.de>.
You might find
https://bz.apache.org/bugzilla/show_bug.cgi?id=65353
interesting.
With the next nightly build or build from trunk, you should be able to use the new property "backend_metrics_percentile_estimator" with a value of "R_3" to lessen the difference between both reports. But keep in mind, that the HTML Report use a sliding window, while the Aggregate Report does not (and will eventually OOM).
Felix
Am 03.06.21 um 17:46 schrieb Deepak Chaudhari:
> I need to send both the reports to the client and surely the question
> will come "Why is there so much difference?"
> If there is any way with which we can generate almost similar
> aggregate and HTML reports?
>
> On Thu, Jun 3, 2021 at 9:00 PM Felix Schumacher
> <felix.schumacher@internetallee.de
> <ma...@internetallee.de>> wrote:
>
> Without the actual data it is impossible to say, if the reports
> are wrong, or if they follow different ways to calculate a percentile.
>
> The problem here is, that the two reports are using different
> algorithms to calculate the percentiles. We are working with
> discrete numbers and here a percentile is more like a range than a
> single (correct or wrong) number. Say, you have the values [1, 2,
> 4, 8, 16, 32, 64, 128, 256, 512] want to calculate a 90%
> percentile. For this you take any number that splits the (sorted)
> list into two lists, where one list has all elements, that are
> smaller (or equal) to the number and one list contains all
> elements that are bigger than the number. For this example, any
> number between 256 and 511.99 would be a valid 90% percentile.
> (that is of course a broad description only)
>
> The Aggregate report currently uses a memory intensive way and
> stores a list with all values and picks the smallest number, that
> fits the above description. (This memory hogging behaviour should
> be fixed and would lead to less accurate data and less OOMs or
> slow JMeter instances)
>
> The HTML report gives you a value that would probably be a good
> 90% percentile, if you would have more data like the one, you
> already have by picking some value between the lower bound and the
> upper bound of the described range (in our example something
> between 256 and 511.999).
>
> As your report has very few samples (21 for the biggest difference
> in the numbers), the skew between the reported percentiles can
> look big, but are not necessarily wrong.
>
> Note, that if the numbers that both reports tell you are far
> apart, that is probably a sign of having either too few samples,
> or the samples (durations of the samples) are distributed sparsely
> near the percentile ranges.
>
> We have discussed in the past to use the same algorithm for both
> components, so that you get consistent values, but there were
> always other issues, that seemed to be more important.
>
> Felix
>
> Am 02.06.21 um 15:28 schrieb Deepak Chaudhari:
>> Hi,
>>
>> I collected JTL file during a test run and using that I'm
>> generating HTML report and aggregate report. When I compare
>> aggregate and HTML reports I found that 90th and 95th percentile
>> values are wrong in HTML report.
>> I tried to change "jmeter.reportgenerator.statistic_window" value
>> in user.properties but still getting wrong values.
>>
>> *Aggregate report:*
>> image.png
>>
>> *HTML report:*
>> image.png
>
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by Deepak Chaudhari <de...@gmail.com>.
I need to send both the reports to the client and surely the question will
come "Why is there so much difference?"
If there is any way with which we can generate almost similar aggregate and
HTML reports?
On Thu, Jun 3, 2021 at 9:00 PM Felix Schumacher <
felix.schumacher@internetallee.de> wrote:
> Without the actual data it is impossible to say, if the reports are wrong,
> or if they follow different ways to calculate a percentile.
>
> The problem here is, that the two reports are using different algorithms
> to calculate the percentiles. We are working with discrete numbers and here
> a percentile is more like a range than a single (correct or wrong) number.
> Say, you have the values [1, 2, 4, 8, 16, 32, 64, 128, 256, 512] want to
> calculate a 90% percentile. For this you take any number that splits the
> (sorted) list into two lists, where one list has all elements, that are
> smaller (or equal) to the number and one list contains all elements that
> are bigger than the number. For this example, any number between 256 and
> 511.99 would be a valid 90% percentile. (that is of course a broad
> description only)
>
> The Aggregate report currently uses a memory intensive way and stores a
> list with all values and picks the smallest number, that fits the above
> description. (This memory hogging behaviour should be fixed and would lead
> to less accurate data and less OOMs or slow JMeter instances)
>
> The HTML report gives you a value that would probably be a good 90%
> percentile, if you would have more data like the one, you already have by
> picking some value between the lower bound and the upper bound of the
> described range (in our example something between 256 and 511.999).
>
> As your report has very few samples (21 for the biggest difference in the
> numbers), the skew between the reported percentiles can look big, but are
> not necessarily wrong.
>
> Note, that if the numbers that both reports tell you are far apart, that
> is probably a sign of having either too few samples, or the samples
> (durations of the samples) are distributed sparsely near the percentile
> ranges.
>
> We have discussed in the past to use the same algorithm for both
> components, so that you get consistent values, but there were always other
> issues, that seemed to be more important.
>
> Felix
> Am 02.06.21 um 15:28 schrieb Deepak Chaudhari:
>
> Hi,
>
> I collected JTL file during a test run and using that I'm generating HTML
> report and aggregate report. When I compare aggregate and HTML reports I
> found that 90th and 95th percentile values are wrong in HTML report.
> I tried to change "jmeter.reportgenerator.statistic_window" value in
> user.properties but still getting wrong values.
>
> *Aggregate report:*
> [image: image.png]
>
> *HTML report:*
> [image: image.png]
>
>
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by Felix Schumacher <fe...@internetallee.de>.
Without the actual data it is impossible to say, if the reports are
wrong, or if they follow different ways to calculate a percentile.
The problem here is, that the two reports are using different algorithms
to calculate the percentiles. We are working with discrete numbers and
here a percentile is more like a range than a single (correct or wrong)
number. Say, you have the values [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
want to calculate a 90% percentile. For this you take any number that
splits the (sorted) list into two lists, where one list has all
elements, that are smaller (or equal) to the number and one list
contains all elements that are bigger than the number. For this example,
any number between 256 and 511.99 would be a valid 90% percentile. (that
is of course a broad description only)
The Aggregate report currently uses a memory intensive way and stores a
list with all values and picks the smallest number, that fits the above
description. (This memory hogging behaviour should be fixed and would
lead to less accurate data and less OOMs or slow JMeter instances)
The HTML report gives you a value that would probably be a good 90%
percentile, if you would have more data like the one, you already have
by picking some value between the lower bound and the upper bound of the
described range (in our example something between 256 and 511.999).
As your report has very few samples (21 for the biggest difference in
the numbers), the skew between the reported percentiles can look big,
but are not necessarily wrong.
Note, that if the numbers that both reports tell you are far apart, that
is probably a sign of having either too few samples, or the samples
(durations of the samples) are distributed sparsely near the percentile
ranges.
We have discussed in the past to use the same algorithm for both
components, so that you get consistent values, but there were always
other issues, that seemed to be more important.
Felix
Am 02.06.21 um 15:28 schrieb Deepak Chaudhari:
> Hi,
>
> I collected JTL file during a test run and using that I'm generating
> HTML report and aggregate report. When I compare aggregate and HTML
> reports I found that 90th and 95th percentile values are wrong in HTML
> report.
> I tried to change "jmeter.reportgenerator.statistic_window" value in
> user.properties but still getting wrong values.
>
> *Aggregate report:*
> image.png
>
> *HTML report:*
> image.png
Re: Wrong 90th and 95th percentile values in JMeter HTML report
Posted by A P <ab...@gmail.com>.
I am also facing the same issue:
http://mail-archives.apache.org/mod_mbox/jmeter-user/202102.mbox/browser
Tried different values for "jmeter.reportgenerator.statistic_window". But
nothing worked. Seems like it's a bug in JMeter.
On Wed, Jun 2, 2021 at 6:57 PM Deepak Chaudhari <de...@gmail.com>
wrote:
> Hi,
>
> I collected JTL file during a test run and using that I'm generating HTML
> report and aggregate report. When I compare aggregate and HTML reports I
> found that 90th and 95th percentile values are wrong in HTML report.
> I tried to change "jmeter.reportgenerator.statistic_window" value in
> user.properties but still getting wrong values.
>
> *Aggregate report:*
> [image: image.png]
>
> *HTML report:*
> [image: image.png]
>