You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@jmeter.apache.org by bu...@apache.org on 2017/09/21 11:15:56 UTC
[Bug 61556] New: Performance regression with IfController and JS,
Groovy evaluations
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
Bug ID: 61556
Summary: Performance regression with IfController and JS,
Groovy evaluations
Product: JMeter
Version: 3.2
Hardware: PC
OS: Linux
Status: NEW
Severity: regression
Priority: P2
Component: Main
Assignee: issues@jmeter.apache.org
Reporter: be_strew@yahoo.co.uk
Target Milestone: ---
Created attachment 35346
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35346&action=edit
Groovy condition evaluation
There appears to be a regression in performance with the move from 3.1 to 3.2
in the evaluation of IfController in JS. This appears to be a slow memory leak
and only was noticeable in running tests with 500 tps over a period of 4 hours.
Jexl evaluation appears to be the same as before. Groovy evaluation can also
lead to a OOM in metaspace, although this is not limited by default in Jmeter!
Which it should be.
I attached some examples tests, which execute a debug controller with a delay,
then evaluates a condition before executing another debug controller with the
same delay. Metaspace limited to 256mb. The condition is of the form:
"${SOME_VALUE}" != "DEFAULT"
Summary of results (test files attached)
Jmeter 3.2
Groovy evaluation = failed after 3 mins with out of memory in metaspace.
JS evaluation = 1808 transactions/s
Jexl evaluation = 1980 transactions/s
Jmeter 3.1
Groovy evaluation = 223 transactions/s
JS evaluation = 1963 transactions/s
Jexl evaluation = 1982 transactions/s
I have attached the results as well as screen shots of the metaspace and cpu
through jvisualvm.
So here are two issues really:
1. Performance degradation (possible slow metaspace leak) on JS evaluation in
IfController.
2. Documentation needs to be clearer on not using variable replacement in
Groovy scripts either in JSR233 components or in Groovy functions such as in
the IfController.
${__groovy( "${SOME_VAR}" != "DEFAULT" )}
is going to be much slower than
${__groovy(vars.get("SOME_VAR") != "NULL" )}
I assume the later is cached and re-used while the former involve more classes
being created each time and eventually leads to a leak. What I don't
understand is why the space is not freed up in the metaspace, I would expect it
to be slower but not to cause a OOM and that is the behaviour that differs from
3.1 to 3.2 but I've not run the tests for longer than 5 mins.
We first noticed the issue on longer tests (4 hr) were the throughput was
trending downward towards the end of the test and noticed that the metaspace
was growing throughout the test execution. This test makes use of quite a few
IfController evaluations in the form:
"${SOME_VAR}" != "DEFAULT"
These are with Interpret Condition as Variable expression = false.
On reverting back to Jmeter 3.1 we did not see the degradation in throughput as
the test continued.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #17 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
(In reply to Justin McCartney from comment #16)
> Hi Philippe,
>
> Groovy
>
> I verified again and the (3.2 default included) groovy jar is definitely
> removed. The only Groovy jar on the class path is groovy-all-2.4.7.jar. I
> re-executed the test with 3.2 and received the same result, OOM metaspace.
>
> I see that your Jmeter docs you linked to do actually mention not to be use
> variable replacement for Groovy scripts which is good. However, I think
> this should be added to the IfController docs as it should be clearer there
> that using variable replacement will have a negative impact on performance.
> Also some examples could be provided to give clarity on the correct syntax
> to use.
>
> The function help for __groovy could also do with this information.
>
> There seem to be other metaspace bugs against later versions of Groovy,
> e.g.: https://issues.apache.org/jira/browse/GROOVY-8189
>
> Javascript
>
> I currently haven't got a simple test that shows the problems we encountered
> with our long running load tests in 3.2 vs earlier Jmeter. I will try and
> re-execute the tests, but I might not get a slot to execute the tests until
> early next week. I would ignore anything related to JS until then.
Hello Justin,
Would you be ready to contribute a PR for documentation improvements ?
- http://jmeter.apache.org/building.html
I'll be happy to merge it.
Thanks
Regards
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |p.mouawad@ubik-ingenierie.c
| |om
--- Comment #11 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Hello,
Thanks for report.
Could you try JMeter 3.2 with groovy version used in 3.1 ?
Thanks
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #12 from Justin McCartney <be...@yahoo.co.uk> ---
Hi Philippe,
I tested again with Groovy evaluations using the 3.2 Jmeter with the Groovy lib
from 3.1 (labelled groovy-all-2.4.7.jar).
I still see the OOM metaspace at ~3mins.
So there didn't seem to be much difference there.
Justin
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
Justin McCartney <be...@yahoo.co.uk> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #35346|Groovy condition evaluation |Groovy condition evaluation
description| |jmx
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Clarify in documentation performance impacts of ${} var usage in IfController and groovy
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #21 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Author: pmouawad
Date: Thu Oct 5 20:28:05 2017
New Revision: 1811257
URL: http://svn.apache.org/viewvc?rev=1811257&view=rev
Log:
Bug 61556 - Clarify in documentation performance impacts of ${} var usage in
IfController and groovy
Contributed by Justin McCartney
This closes #314
Bugzilla Id: 61556
Modified:
jmeter/trunk/xdocs/changes.xml
jmeter/trunk/xdocs/usermanual/component_reference.xml
jmeter/trunk/xdocs/usermanual/functions.xml
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Clarify in documentation performance impacts of ${} var usage in IfController and groovy
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Hardware|PC |All
Summary|Performance regression with |Clarify in documentation
|IfController and JS, Groovy |performance impacts of ${}
|evaluations |var usage in IfController
| |and groovy
Severity|regression |enhancement
OS|Linux |All
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
Justin McCartney <be...@yahoo.co.uk> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #35347|Jexl condition evaluation |Jexl condition evaluation
description| |jmx
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Clarify in documentation performance impacts of ${} var usage in IfController and groovy
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |RESOLVED
Resolution|--- |FIXED
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #10 from Justin McCartney <be...@yahoo.co.uk> ---
Created attachment 35356
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35356&action=edit
Mem Jmeter 3.2 JS evaluation
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #3 from Justin McCartney <be...@yahoo.co.uk> ---
Created attachment 35349
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35349&action=edit
Jmeter 3.1 summary results
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #18 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Hi Justin,
Any feedback on this issue ?
Thanks
Regards
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #19 from Justin McCartney <be...@yahoo.co.uk> ---
(In reply to Philippe Mouawad from comment #18)
> Hi Justin,
> Any feedback on this issue ?
>
> Thanks
> Regards
Hi Philippe,
sorry I have been busy on other tasks but I am currently adding to the docs and
should hopefully have a pull request for the doc update today.
I still need to get time to re-execute the long duration tests in relation to
the JS change from 3.1 to 3.2, but I might not get a slot for that until late
next week.
Thanks,
Justin
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #4 from Justin McCartney <be...@yahoo.co.uk> ---
Created attachment 35350
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35350&action=edit
Jmeter 3.2 summary results
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #1 from Justin McCartney <be...@yahoo.co.uk> ---
Created attachment 35347
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35347&action=edit
Jexl condition evaluation
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #13 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Hello,
Are you running JMeter 3.1 and 3.2 with same Java version ?
Thanks
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #16 from Justin McCartney <be...@yahoo.co.uk> ---
Hi Philippe,
Groovy
I verified again and the (3.2 default included) groovy jar is definitely
removed. The only Groovy jar on the class path is groovy-all-2.4.7.jar. I
re-executed the test with 3.2 and received the same result, OOM metaspace.
I see that your Jmeter docs you linked to do actually mention not to be use
variable replacement for Groovy scripts which is good. However, I think this
should be added to the IfController docs as it should be clearer there that
using variable replacement will have a negative impact on performance. Also
some examples could be provided to give clarity on the correct syntax to use.
The function help for __groovy could also do with this information.
There seem to be other metaspace bugs against later versions of Groovy, e.g.:
https://issues.apache.org/jira/browse/GROOVY-8189
Javascript
I currently haven't got a simple test that shows the problems we encountered
with our long running load tests in 3.2 vs earlier Jmeter. I will try and
re-execute the tests, but I might not get a slot to execute the tests until
early next week. I would ignore anything related to JS until then.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #5 from Justin McCartney <be...@yahoo.co.uk> ---
Created attachment 35351
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35351&action=edit
Mem Jmeter 3.1 Groovy evaluation
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #6 from Justin McCartney <be...@yahoo.co.uk> ---
Created attachment 35352
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35352&action=edit
Mem Jmeter 3.1 Jexl evaluation
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
Philippe Mouawad <p....@ubik-ingenierie.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #15 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Hello,
Thanks for your feedback.
Few notes:
- 3.2 switched from Rhino to Nashorn. If you diagnose a leak, it would be a
Java leak IMO
- Regarding Groovy, the memory leak you see is due to Groovy, it disappears if
you add this -Dgroovy.use.classvalue=true. See
http://jmeter.apache.org/usermanual/component_reference.html#JSR223_Sampler. I
don't reproduce OOM with this setting but throughput is negatively affected.
- I agree with your note on Metaspace, we should indeed add
-XX:MaxMetaspaceSize=256m
Now what I don't understand is the difference in behaviour between 3.1 and 3.2
when using same groovy version. Are you sure you removed the new groovy jar in
3.2 when testing old one ?
The second strange thing, is performance regression in JS between 3.1 and 3.2,
in the tests I made, Nashorn was outperforming Rhino.
Would it be possible to attach your log file and ensure you're using fresh
versions of 3.1 and 3.2 without any plugin and using same exact settings if not
already done ?
Thank you
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Clarify in documentation performance impacts of ${} var usage in IfController and groovy
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #22 from Philippe Mouawad <p....@ubik-ingenierie.com> ---
Thanks a lot for your PR.
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #20 from Justin McCartney <be...@yahoo.co.uk> ---
Added pull request for documentation changes:
https://github.com/apache/jmeter/pull/314
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #8 from Justin McCartney <be...@yahoo.co.uk> ---
Created attachment 35354
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35354&action=edit
Mem Jmeter 3.2 Groovy evaluation
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #7 from Justin McCartney <be...@yahoo.co.uk> ---
Created attachment 35353
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35353&action=edit
Mem Jmeter 3.1 JS evaluation
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #2 from Justin McCartney <be...@yahoo.co.uk> ---
Created attachment 35348
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35348&action=edit
JS condition evaluation jmx
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #9 from Justin McCartney <be...@yahoo.co.uk> ---
Created attachment 35355
--> https://bz.apache.org/bugzilla/attachment.cgi?id=35355&action=edit
Mem Jmeter 3.2 Jexl evaluation
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 61556] Performance regression with IfController and JS, Groovy
evaluations
Posted by bu...@apache.org.
https://bz.apache.org/bugzilla/show_bug.cgi?id=61556
--- Comment #14 from Justin McCartney <be...@yahoo.co.uk> ---
Hi,
yes all tests were executed with the same Java version no matter what version
of Jmeter.
> java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)
--
You are receiving this mail because:
You are the assignee for the bug.