[Bug 65399] New: Big difference in performance between non-gui and gui executions

[bu...@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=65399

            Bug ID: 65399
           Summary: Big difference in performance between non-gui and gui
                    executions
           Product: JMeter
           Version: 5.4.1
          Hardware: PC
            Status: NEW
          Severity: normal
          Priority: P2
         Component: Main
          Assignee: issues@jmeter.apache.org
          Reporter: valentina.pinto@akzio.cl
  Target Milestone: JMETER_5.5

Hi,
I've been comparing results of different load test executions and i've been
getting a big difference in performance.
Executing in command line gives a longer response time (about 100% or 200%
more).
The thread group i use is "Concurrency Thread Group". I run the test with 100
users and hold it for 10 seconds.
I've tested public apis (dog api, cat api, etc), i get this results with
https://random.dog too. I've tried httpbin (https://httpbin.org/) and i get a
similar difference.
I don't know if i'm doing something wrong. Any help or comment please?
Thanks in advance.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 65399] Big difference in performance between non-gui and gui executions

[bu...@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=65399

--- Comment #2 from Valentina <va...@akzio.cl> ---
Created attachment 37921
  --> https://bz.apache.org/bugzilla/attachment.cgi?id=37921&action=edit
Test plan

This is a simple test plan.
I've compared the response times using the "Aggregate report", using the
"Report Dashboard" and generating .jtl files and then calculating percentiles
by my own.
I can reproduce it without the plugin, the attachment file is a simple example.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 65399] Big difference in performance between non-gui and gui executions

[bu...@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=65399

--- Comment #4 from Valentina <va...@akzio.cl> ---
Thank you so much for your answer. Then i think it might be the JVM because i
keep getting the difference.

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 65399] Big difference in performance between non-gui and gui executions

[bu...@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=65399

Felix Schumacher <fe...@internetallee.de> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |NEEDINFO
                 OS|                            |All

--- Comment #1 from Felix Schumacher <fe...@internetallee.de> ---
Can you give a minimal test plan, that shows the problem?

How are you comparing the response times?

Can you reproduce the behaviour without the third party plugin?

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 65399] Big difference in performance between non-gui and gui executions

[bu...@apache.org]

https://bz.apache.org/bugzilla/show_bug.cgi?id=65399

--- Comment #3 from Felix Schumacher <fe...@internetallee.de> ---
Thanks for the test plan.

In my short tests, the first two times the GUI tests run faster, but the
percentage of faster was nowhere near 100 or even 200 %. It was more 1 or 2 %.
(I have done two tries, which is not really statistically relevant and a last
test done after me writing this showed the non-GUI run faster 106 TX/s vs. 99
TX/s in the GUI)

I even expanded the test duration (I used a duration of 100 seconds) and the
GUI is (in that one run I did a bit) slower than running it without the GUI. (I
have set up a local test server in order to avoid DOSing some random service)

The results (from dashboard) were:
GUI: 1204597 Requests total, 12051.15 TX/s, 97 max, 7 Median, 12 95%, 22 99%
Non-GUI: 1235490 Requests total, 12358.61 TX/s, 83 max, 7 Median, 12 95%, 24
99% 

The reason for the (possibly true) advantage of the GUI for short test duration
could be, that a JVM needs a bit to warm up and that we give it the time when
loading a test plan and trying to find the right button to start it. In the
non-GUI mode, that warm up phase is missing and it might hurt the performance
of short tests (a wild guess).

If you are wondering about differences between the stats when viewed in the
dashboard, the aggregate report or excel, you might like to have a look at bug
65353.

-- 
You are receiving this mail because:
You are the assignee for the bug.