[Bug 65446] New: Report / Dashboard: Percentile over time values depend on the order of the data even with large window size

[bu...@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=65446

            Bug ID: 65446
           Summary: Report / Dashboard: Percentile over time values depend
                    on the order of the data even with large window size
           Product: JMeter
           Version: 5.4.1
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Main
          Assignee: issues@jmeter.apache.org
          Reporter: rithvikp98@gmail.com
  Target Milestone: JMETER_5.5

We have a use case where we are merging CSV files from two different,
independent JMeter instances and using the combined CSV file to generate the
dashboard. While merging two files, there is obviously a question about the
order of the data points in the CSV and how it impacts the final dashboard. To
test it out, I created two CSV files with the same data but one where the data
is in increasing order of timestamps and the other is in decreasing order and
compared the Response Time Percentiles Over Time (successful responses) graphs. 

From what I understand from Kth select algorithm used for percentile
calculation, if the input data set contains of the same values, irrespective of
the order, the output should be the same every time. In JMeter, the input data
set is controlled by the jmeter.reportgenerator.statistic_window property.

So, my hypothesis is, if the statistic_window is large enough to hold all the
samples for a given time-granularity window, the output graphs should be 100%
identical but that doesn't seem to be the case. I generated graphs with a
statistic_window of 100 million which is far more than the number of samples in
the entire CSV file and still, the graphs are different from one another, even
if it is by a small percentage. 

I like to understand if my assumptions are wrong or if there is a bug in JMeter
that I need to search for.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 65446] Report / Dashboard: statistic_window property is not being considered while calculating percentiles.

[bu...@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=65446

--- Comment #2 from Rithvik Patibandla <ri...@gmail.com> ---
Any update of this?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 65446] Report / Dashboard: statistic_window property is not being considered while calculating percentiles.

[bu...@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=65446

Rithvik Patibandla <ri...@gmail.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|Report / Dashboard:         |Report / Dashboard:
                   |Percentile over time values |statistic_window property
                   |depend on the order of the  |is not being considered
                   |data even with large window |while calculating
                   |size                        |percentiles.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 65446] Report / Dashboard: Percentile over time values depend on the order of the data even with large window size

[bu...@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=65446

--- Comment #1 from Rithvik Patibandla <ri...@gmail.com> ---
I did a little more digging and I think the issue is with the following line:

private static final int SLIDING_WINDOW_SIZE = JMeterUtils.getPropDefault(
            ReportGeneratorConfiguration.REPORT_GENERATOR_KEY_PREFIX
                    + ReportGeneratorConfiguration.KEY_DELIMITER
                    + "statistic_window", 20000);


getPropDefault() calls appProperties.getProperty(). But in the appProperties we
are only loading jmeter.properties, user.properties, system.properties etc and
_not_ reportgenerator.properties where the value for statistic_window is
supposed to be set. So the default value of 20,000 is being picked irrespective
of what is set in the properties file. I confirmed this by moving the property
to jmeter.properties file and then my custom value got picked as expected. 

If the maintainers can confirm this as an issue, I can raise a pull request
with the fix which uses ReportGeneratorConfiguration instead. This issue was
raised in a separate thread as well:
https://bz.apache.org/bugzilla/show_bug.cgi?id=58959

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 65446] Report / Dashboard: statistic_window property is not being considered while calculating percentiles.

[bu...@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=65446

Felix Schumacher <fe...@internetallee.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #3 from Felix Schumacher <fe...@internetallee.de> ---
Has been included in current master branch.

Thanks for the work.

commit 228ce437fb8686c4057409ba36c6fe4ce3b721fd
AuthorDate: Fri Jul 30 14:46:00 2021 +0530

    Move definition of jmeter.reportgenerator.statistic_window to correct
source

    JMeter will not look in reportgenerator.properties, but in user.properties
    for the value of this property.

    Bugzilla Id: 65446
    Closes #671
---
 bin/reportgenerator.properties | 4 ----
 bin/user.properties            | 4 ++++
 xdocs/changes.xml              | 4 ++++
 3 files changed, 8 insertions(+), 4 deletions(-)

-- 
You are receiving this mail because:
You are the assignee for the bug.