You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@jmeter.apache.org by Tuukka Mustonen <tu...@gmail.com> on 2017/02/01 07:21:01 UTC

High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Hi,

After upgrading to JMeter 3.0 or 3.1 I experience high CPU usage that
renders JMeter unusable. CPU usage ramps quickly up and after having
~100-200 concurrent users or so (in ~30 seconds), it hits 100% on my
i7-4600. Shortly after, whole JMeter freezes and I have to kill it.

On 2.13 everything goes smoothly (with default settings). CPU usage stays
at ~30% even with 500 users.

The experience is similar on both Windows 10 and Linux (Ubuntu 14.04 LTS
guest in Virtualbox).

The problems I saw in JMeter 3:

- java.net.SocketException: Permission denied: connect
- After a while it runs out of heap space

For "Permission denied: connect" I tried to set java.net.preferIPv4Stack=true
(in a few places to make sure it would really be in use) and tried to run
JMeter as administrator (in Win). No effect.

I also tried increasing heap size, but that didn't help either. And I think
memory size is not the root cause here - rather the result.

I am normally connected to VPN but I also tried disabling it. No effect.

I tried disabling with and without IPv6 enabled in Windows network
connection settings. No effect.

I am running a few extra JMeter plugins. Tried cutting my test scenario to
minimum. No effect.

I am running anti-virus software that I cannot disable, due to this being
forced by IT department. However, it's not a problem with 2.13 so I assume
it shouldn't be a problem here either.

FINALLY, once I set HTTP request implementation from Apache HttpClient 4 to
HttpClient 3.1 or JAVA, the problem vanishes - CPU stays where it did with
JMeter 2.13.

I'm running Oracle JVM 8 (update 121 on Win, update 111 on Linux).

Why does (the recommended) HttpClient 4 cause problems here? Is this a know
bug/issue? JMeter 3.x is running newer version compared to 2.13, but what
change in HttpClient 4 might cause the problems described?

Any ideas?

Regards,
Tuukka

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

Could you provide the Heap Dump on drive ?
Thanks

On Wed, Feb 1, 2017 at 10:07 AM, Tuukka Mustonen <tu...@gmail.com>
wrote:

> Oh, and I did try nightly build. It did not have effect.
>
> The dumps were created in JMeter 3.1 that I assume should be more stable
> than the nightly.
>
> Tuukka
>
>
> On Wed, Feb 1, 2017 at 10:55 AM, Tuukka Mustonen <
> tuukka.mustonen@gmail.com>
> wrote:
>
> > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
> > slightly different behavior in my Linux guest. I'll write more about that
> > later, but for simplicity I'm testing only on Windows host now:
> >
> > Thread dumps are downloadable at: https://drive.google.com/
> drive/folders/
> > 0B1uBdVuLED5NS1RFbEFuSS1iejA
> >
> > (Took them with 7-8s interval I think, hope it's ok).
> >
> > I took the thread dumps once JMeter started filling in errors, which is
> > somewhat after my CPU usage hit 100% (and music playback started to
> distort
> > etc.).
> >
> > I also created the head dump, note that I took it after running the test
> > and after restarting JMeter, as JMeter froze during the test and I had to
> > kill it. DId you mean I should take it *during* the test? Anyway, before
> > sharing it I want to make sure it can't contain anything confidential?
> > (parts of my test plan, passwords, environment variables, anything...)
> >
> > Tuukka
> >
> >
> >
> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > support@ubikloadpack.com> wrote:
> >
> >> Hello,
> >> To investigate the problem:
> >> 1/ to create  thread dumps, install a JDK of same version as the one
> >> running your JMeter.
> >>
> >> Run until you get the problem, then call using same user as the one
> >> running
> >> JMeter 3 times at 5s interval:
> >>  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> >>
> >> It will also help if you provide the structure of your Test Plan.
> >>
> >> 2/ It would also help if you can provide a Heap Dump. With JMeter GUI,
> use
> >> menu Help > Create a Heap Dump
> >> This will create a Heap Dump (rather big file) , it would help a lot if
> >> you
> >> can make it available.
> >>
> >> Regards
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> >> tuukka.mustonen@gmail.com>
> >> wrote:
> >>
> >> > @Philippe:
> >> >
> >> > Yeah, I am normally running in non-GUI mode, but to
> >> > study/inspect/understand this problem I am running in GUI mode. JMeter
> >> 2.13
> >> > works just fine in GUI mode for these tests so JMeter 3.x should too.
> >> >
> >> > I'll give the nightly build a try.
> >> >
> >> > Tuukka
> >> >
> >> >
> >> > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> >> > philippe.mouawad@gmail.com
> >> > > wrote:
> >> >
> >> > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> >> > tuukka.mustonen@gmail.com
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > @Philippe:
> >> > > >
> >> > > > 1. Yeah, I compared them with the same plan, same machine. Should
> be
> >> > > > everything equal.
> >> > > > 2. I tried both the defaults in jmeter.bat (-Xms512m -Xmx512m) and
> >> as
> >> > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them much
> >> > > because I
> >> > > > don't believe the issue lies there.
> >> > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> >> > > > 4. I am not familiar with that but I can try. If you have
> >> instructions
> >> > to
> >> > > > point to, please do.
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > > 5. I am running with GUI.
> >> > >
> >> > >
> >> > > This is a known anti-pattern .
> >> > > switch to non gui mode if you want correct results.
> >> > > http://jmeter.apache.org/usermanual/best-practices.html
> >> > >
> >> > > http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
> >> > >
> >> > >
> >> > > > About file sending:
> >> > > >
> >> > > > 6. Nope. I'm just doing simple GETs.
> >> > >
> >> > >  Could you give nightly build a try ?
> >> > > http://jmeter.apache.org/nightly.html
> >> > >
> >> > >
> >> > > > Tuukka
> >> > > >
> >> > > >
> >> > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> >> > > > philippe.mouawad@gmail.com <javascript:;>
> >> > > > > wrote:
> >> > > >
> >> > > > > Also are you sending files in post body ?
> >> > > > >
> >> > > > > Thank you
> >> > > > >
> >> > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> >> > > > > philippe.mouawad@gmail.com <javascript:;>>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hello,
> >> > > > > > Few questions, please try to answer them all:
> >> > > > > >
> >> > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape machine...
> >> > > > everything
> >> > > > > > equal ?
> >> > > > > > 2/ What are your Xmx settings
> >> > > > > > 3/ does your plan download embedded resources ?
> >> > > > > > 4/ when issue happens can you make 3 thread dumps at 5 seconds
> >> > > interval
> >> > > > > > and send file ? (use jstack)
> >> > > > > > 5/ are you running gui or non gui ?
> >> > > > > >
> >> > > > > > Thanks
> >> > > > > > Regards
> >> > > > > >
> >> > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> >> > > > > tuukka.mustonen@gmail.com <javascript:;>
> >> > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> >> > > <javascript:;>');>>
> >> > > > wrote:
> >> > > > > >
> >> > > > > >> Hi,
> >> > > > > >>
> >> > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high CPU
> >> usage
> >> > > that
> >> > > > > >> renders JMeter unusable. CPU usage ramps quickly up and after
> >> > having
> >> > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it hits
> 100%
> >> on
> >> > my
> >> > > > > >> i7-4600. Shortly after, whole JMeter freezes and I have to
> kill
> >> > it.
> >> > > > > >>
> >> > > > > >> On 2.13 everything goes smoothly (with default settings). CPU
> >> > usage
> >> > > > > stays
> >> > > > > >> at ~30% even with 500 users.
> >> > > > > >>
> >> > > > > >> The experience is similar on both Windows 10 and Linux
> (Ubuntu
> >> > 14.04
> >> > > > LTS
> >> > > > > >> guest in Virtualbox).
> >> > > > > >>
> >> > > > > >> The problems I saw in JMeter 3:
> >> > > > > >>
> >> > > > > >> - java.net.SocketException: Permission denied: connect
> >> > > > > >> - After a while it runs out of heap space
> >> > > > > >>
> >> > > > > >> For "Permission denied: connect" I tried to set
> >> > > > > >> java.net.preferIPv4Stack=true
> >> > > > > >> (in a few places to make sure it would really be in use) and
> >> tried
> >> > > to
> >> > > > > run
> >> > > > > >> JMeter as administrator (in Win). No effect.
> >> > > > > >>
> >> > > > > >> I also tried increasing heap size, but that didn't help
> either.
> >> > And
> >> > > I
> >> > > > > >> think
> >> > > > > >> memory size is not the root cause here - rather the result.
> >> > > > > >>
> >> > > > > >> I am normally connected to VPN but I also tried disabling it.
> >> No
> >> > > > effect.
> >> > > > > >>
> >> > > > > >> I tried disabling with and without IPv6 enabled in Windows
> >> network
> >> > > > > >> connection settings. No effect.
> >> > > > > >>
> >> > > > > >> I am running a few extra JMeter plugins. Tried cutting my
> test
> >> > > > scenario
> >> > > > > to
> >> > > > > >> minimum. No effect.
> >> > > > > >>
> >> > > > > >> I am running anti-virus software that I cannot disable, due
> to
> >> > this
> >> > > > > being
> >> > > > > >> forced by IT department. However, it's not a problem with
> 2.13
> >> so
> >> > I
> >> > > > > assume
> >> > > > > >> it shouldn't be a problem here either.
> >> > > > > >>
> >> > > > > >> FINALLY, once I set HTTP request implementation from Apache
> >> > > > HttpClient 4
> >> > > > > >> to
> >> > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays
> where
> >> it
> >> > > did
> >> > > > > with
> >> > > > > >> JMeter 2.13.
> >> > > > > >>
> >> > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update 111 on
> >> Linux).
> >> > > > > >>
> >> > > > > >> Why does (the recommended) HttpClient 4 cause problems here?
> Is
> >> > > this a
> >> > > > > >> know
> >> > > > > >> bug/issue? JMeter 3.x is running newer version compared to
> >> 2.13,
> >> > but
> >> > > > > what
> >> > > > > >> change in HttpClient 4 might cause the problems described?
> >> > > > > >>
> >> > > > > >> Any ideas?
> >> > > > > >>
> >> > > > > >> Regards,
> >> > > > > >> Tuukka
> >> > > > > >>
> >> > > > > >
> >> > > > > >
> >> > > > > > --
> >> > > > > > Cordialement.
> >> > > > > > Philippe Mouawad.
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Cordialement.
> >> > > > > Philippe Mouawad.
> >> > > > >
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Cordialement.
> >> > > Philippe Mouawad.
> >> > >
> >> >
> >>
> >
> >
>



-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

@UBIK Load Pack support: is there some other email address that I can send
info to? support@ubikloadpack.com sounds a bit too generic :)

Tuukka


On Wed, Feb 1, 2017 at 3:29 PM, Tuukka Mustonen <tu...@gmail.com>
wrote:

> Hi,
>
>
> *> From thread dump I see you have in GUI mode around  500 threads and
> you using a View Resuls Tree and Summary Report ?*
> *> Does view results Tree only catch errors or also successes ?*
>
> Yeah I add 20 users every 3 seconds up until 500. Then I run 2mins (if
> 2.13 at least).
>
> Yeah, I have Summary Report enabled and View Results Tree, but I only show
> Errors in View Results Tree. So nothing gets bloated there until things
> break apart.
>
> *>    -  try today this exact same plan with this number of threads in
> JMeter 2.13 with a View Results Tree configured as with this *
>
> Yes, everything is equal.
>
>
> *>   - was JMeter 2.13 running with same JVM version as this one ?*
>
> Yeah the JVM is the same Oracle Java 8u121.
>
> I did a couple of re-runs as requested by Philippe. I will send you the
> results in private mail, too.
>
> (Sorry for slowish response time. Gmail thread view didn't show your
> message when I was in "Sent" mode. Now back in Inbox and it shows up...)
>
> Tuukka
>
>
> On Wed, Feb 1, 2017 at 2:26 PM, UBIK LOAD PACK Support <
> support@ubikloadpack.com> wrote:
>
>> Hello,
>> From thread dump I see you have in GUI mode around  500 threads and you
>> using a View Resuls Tree and Summary Report ?
>> Does view results Tree only catch errors or also successes ?
>>
>> Did you:
>>
>>    -  try today this exact same plan with this number of threads in JMeter
>>    2.13 with a View Results Tree configured as with this plan ?
>>    - was JMeter 2.13 running with same JVM version as this one ?
>>
>> Thanks
>>
>> On Wed, Feb 1, 2017 at 12:57 PM, Tuukka Mustonen <
>> tuukka.mustonen@gmail.com>
>> wrote:
>>
>> > Ok I re-ran the test.
>> >
>> > Took Heap Dump just before JMeter froze.
>> >
>> > Took Thread Dumps right after JMeter froze.
>> >
>> > https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
>> >
>> > ...
>> >
>> > For the fun of it, I also tried upgrading to the latest HttpClient 4.5.3
>> > version. No effect.
>> >
>> > I also noticed that if I ping (google) during the perf testing, Windows
>> > ping reports "General failure." when JMeter breaks. After killing it,
>> > connectivity restores and there are replies from the server again. It
>> seems
>> > like JMeter/HttpClient kills my whole network connection...
>> >
>> > Tuukka
>> >
>> >
>> > On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
>> > philippe.mouawad@gmail.com> wrote:
>> >
>> > > On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen <
>> > tuukka.mustonen@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
>> > > slightly
>> > > > different behavior in my Linux guest. I'll write more about that
>> later,
>> > > but
>> > > > for simplicity I'm testing only on Windows host now:
>> > > >
>> > > > Thread dumps are downloadable at:
>> > > > https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
>> > > >
>> > > > (Took them with 7-8s interval I think, hope it's ok).
>> > > >
>> > > > I took the thread dumps once JMeter started filling in errors,
>> which is
>> > > > somewhat after my CPU usage hit 100% (and music playback started to
>> > > distort
>> > > > etc.).
>> > > >
>> > > > I also created the head dump, note that I took it after running the
>> > test
>> > > > and after restarting JMeter, as JMeter froze during the test and I
>> had
>> > to
>> > > > kill it. DId you mean I should take it *during* the test?
>> > >
>> > >
>> > > YES.
>> > >
>> > >
>> > > > Anyway, before
>> > > > sharing it I want to make sure it can't contain anything
>> confidential?
>> > > > (parts of my test plan, passwords, environment variables,
>> anything...)
>> > > >
>> > > > Tuukka
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
>> > > > support@ubikloadpack.com> wrote:
>> > > >
>> > > > > Hello,
>> > > > > To investigate the problem:
>> > > > > 1/ to create  thread dumps, install a JDK of same version as the
>> one
>> > > > > running your JMeter.
>> > > > >
>> > > > > Run until you get the problem, then call using same user as the
>> one
>> > > > running
>> > > > > JMeter 3 times at 5s interval:
>> > > > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
>> > > > >
>> > > > > It will also help if you provide the structure of your Test Plan.
>> > > > >
>> > > > > 2/ It would also help if you can provide a Heap Dump. With JMeter
>> > GUI,
>> > > > use
>> > > > > menu Help > Create a Heap Dump
>> > > > > This will create a Heap Dump (rather big file) , it would help a
>> lot
>> > if
>> > > > you
>> > > > > can make it available.
>> > > > >
>> > > > > Regards
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
>> > > > tuukka.mustonen@gmail.com
>> > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > @Philippe:
>> > > > > >
>> > > > > > Yeah, I am normally running in non-GUI mode, but to
>> > > > > > study/inspect/understand this problem I am running in GUI mode.
>> > > JMeter
>> > > > > 2.13
>> > > > > > works just fine in GUI mode for these tests so JMeter 3.x should
>> > too.
>> > > > > >
>> > > > > > I'll give the nightly build a try.
>> > > > > >
>> > > > > > Tuukka
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
>> > > > > > philippe.mouawad@gmail.com
>> > > > > > > wrote:
>> > > > > >
>> > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> > > > > > tuukka.mustonen@gmail.com
>> > > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > @Philippe:
>> > > > > > > >
>> > > > > > > > 1. Yeah, I compared them with the same plan, same machine.
>> > Should
>> > > > be
>> > > > > > > > everything equal.
>> > > > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
>> -Xmx512m)
>> > > and
>> > > > as
>> > > > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them
>> > much
>> > > > > > > because I
>> > > > > > > > don't believe the issue lies there.
>> > > > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
>> > > > > > > > 4. I am not familiar with that but I can try. If you have
>> > > > > instructions
>> > > > > > to
>> > > > > > > > point to, please do.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > > 5. I am running with GUI.
>> > > > > > >
>> > > > > > >
>> > > > > > > This is a known anti-pattern .
>> > > > > > > switch to non gui mode if you want correct results.
>> > > > > > > http://jmeter.apache.org/usermanual/best-practices.html
>> > > > > > >
>> > > > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
>> > > tuning_tips/
>> > > > > > >
>> > > > > > >
>> > > > > > > > About file sending:
>> > > > > > > >
>> > > > > > > > 6. Nope. I'm just doing simple GETs.
>> > > > > > >
>> > > > > > >  Could you give nightly build a try ?
>> > > > > > > http://jmeter.apache.org/nightly.html
>> > > > > > >
>> > > > > > >
>> > > > > > > > Tuukka
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
>> > > > > > > > philippe.mouawad@gmail.com <javascript:;>
>> > > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Also are you sending files in post body ?
>> > > > > > > > >
>> > > > > > > > > Thank you
>> > > > > > > > >
>> > > > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
>> > > > > > > > > philippe.mouawad@gmail.com <javascript:;>>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hello,
>> > > > > > > > > > Few questions, please try to answer them all:
>> > > > > > > > > >
>> > > > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
>> > > machine...
>> > > > > > > > everything
>> > > > > > > > > > equal ?
>> > > > > > > > > > 2/ What are your Xmx settings
>> > > > > > > > > > 3/ does your plan download embedded resources ?
>> > > > > > > > > > 4/ when issue happens can you make 3 thread dumps at 5
>> > > seconds
>> > > > > > > interval
>> > > > > > > > > > and send file ? (use jstack)
>> > > > > > > > > > 5/ are you running gui or non gui ?
>> > > > > > > > > >
>> > > > > > > > > > Thanks
>> > > > > > > > > > Regards
>> > > > > > > > > >
>> > > > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> > > > > > > > > tuukka.mustonen@gmail.com <javascript:;>
>> > > > > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
>> > > > > > > <javascript:;>');>>
>> > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > >> Hi,
>> > > > > > > > > >>
>> > > > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high
>> CPU
>> > > > usage
>> > > > > > > that
>> > > > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up and
>> > > after
>> > > > > > having
>> > > > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it
>> hits
>> > > 100%
>> > > > > on
>> > > > > > my
>> > > > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I
>> have to
>> > > > kill
>> > > > > > it.
>> > > > > > > > > >>
>> > > > > > > > > >> On 2.13 everything goes smoothly (with default
>> settings).
>> > > CPU
>> > > > > > usage
>> > > > > > > > > stays
>> > > > > > > > > >> at ~30% even with 500 users.
>> > > > > > > > > >>
>> > > > > > > > > >> The experience is similar on both Windows 10 and Linux
>> > > (Ubuntu
>> > > > > > 14.04
>> > > > > > > > LTS
>> > > > > > > > > >> guest in Virtualbox).
>> > > > > > > > > >>
>> > > > > > > > > >> The problems I saw in JMeter 3:
>> > > > > > > > > >>
>> > > > > > > > > >> - java.net.SocketException: Permission denied: connect
>> > > > > > > > > >> - After a while it runs out of heap space
>> > > > > > > > > >>
>> > > > > > > > > >> For "Permission denied: connect" I tried to set
>> > > > > > > > > >> java.net.preferIPv4Stack=true
>> > > > > > > > > >> (in a few places to make sure it would really be in
>> use)
>> > and
>> > > > > tried
>> > > > > > > to
>> > > > > > > > > run
>> > > > > > > > > >> JMeter as administrator (in Win). No effect.
>> > > > > > > > > >>
>> > > > > > > > > >> I also tried increasing heap size, but that didn't help
>> > > > either.
>> > > > > > And
>> > > > > > > I
>> > > > > > > > > >> think
>> > > > > > > > > >> memory size is not the root cause here - rather the
>> > result.
>> > > > > > > > > >>
>> > > > > > > > > >> I am normally connected to VPN but I also tried
>> disabling
>> > > it.
>> > > > No
>> > > > > > > > effect.
>> > > > > > > > > >>
>> > > > > > > > > >> I tried disabling with and without IPv6 enabled in
>> Windows
>> > > > > network
>> > > > > > > > > >> connection settings. No effect.
>> > > > > > > > > >>
>> > > > > > > > > >> I am running a few extra JMeter plugins. Tried cutting
>> my
>> > > test
>> > > > > > > > scenario
>> > > > > > > > > to
>> > > > > > > > > >> minimum. No effect.
>> > > > > > > > > >>
>> > > > > > > > > >> I am running anti-virus software that I cannot disable,
>> > due
>> > > to
>> > > > > > this
>> > > > > > > > > being
>> > > > > > > > > >> forced by IT department. However, it's not a problem
>> with
>> > > 2.13
>> > > > > so
>> > > > > > I
>> > > > > > > > > assume
>> > > > > > > > > >> it shouldn't be a problem here either.
>> > > > > > > > > >>
>> > > > > > > > > >> FINALLY, once I set HTTP request implementation from
>> > Apache
>> > > > > > > > HttpClient 4
>> > > > > > > > > >> to
>> > > > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU
>> stays
>> > > where
>> > > > > it
>> > > > > > > did
>> > > > > > > > > with
>> > > > > > > > > >> JMeter 2.13.
>> > > > > > > > > >>
>> > > > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update
>> 111 on
>> > > > > Linux).
>> > > > > > > > > >>
>> > > > > > > > > >> Why does (the recommended) HttpClient 4 cause problems
>> > here?
>> > > > Is
>> > > > > > > this a
>> > > > > > > > > >> know
>> > > > > > > > > >> bug/issue? JMeter 3.x is running newer version
>> compared to
>> > > > 2.13,
>> > > > > > but
>> > > > > > > > > what
>> > > > > > > > > >> change in HttpClient 4 might cause the problems
>> described?
>> > > > > > > > > >>
>> > > > > > > > > >> Any ideas?
>> > > > > > > > > >>
>> > > > > > > > > >> Regards,
>> > > > > > > > > >> Tuukka
>> > > > > > > > > >>
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > --
>> > > > > > > > > > Cordialement.
>> > > > > > > > > > Philippe Mouawad.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > > Cordialement.
>> > > > > > > > > Philippe Mouawad.
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Cordialement.
>> > > > > > > Philippe Mouawad.
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Cordialement.
>> > > Philippe Mouawad.
>> > >
>> >
>>
>>
>>
>> --
>>
>> Regards
>> Ubik Load Pack <http://ubikloadpack.com> Team
>> Follow us on Twitter <http://twitter.com/ubikloadpack>
>>
>>
>> Cordialement
>> L'équipe Ubik Load Pack <http://ubikloadpack.com>
>> Suivez-nous sur Twitter <http://twitter.com/ubikloadpack>
>>
>
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

Hi,


*> From thread dump I see you have in GUI mode around  500 threads and
you using a View Resuls Tree and Summary Report ?*
*> Does view results Tree only catch errors or also successes ?*

Yeah I add 20 users every 3 seconds up until 500. Then I run 2mins (if 2.13
at least).

Yeah, I have Summary Report enabled and View Results Tree, but I only show
Errors in View Results Tree. So nothing gets bloated there until things
break apart.

*>    -  try today this exact same plan with this number of threads in
JMeter 2.13 with a View Results Tree configured as with this *

Yes, everything is equal.


*>   - was JMeter 2.13 running with same JVM version as this one ?*

Yeah the JVM is the same Oracle Java 8u121.

I did a couple of re-runs as requested by Philippe. I will send you the
results in private mail, too.

(Sorry for slowish response time. Gmail thread view didn't show your
message when I was in "Sent" mode. Now back in Inbox and it shows up...)

Tuukka


On Wed, Feb 1, 2017 at 2:26 PM, UBIK LOAD PACK Support <
support@ubikloadpack.com> wrote:

> Hello,
> From thread dump I see you have in GUI mode around  500 threads and you
> using a View Resuls Tree and Summary Report ?
> Does view results Tree only catch errors or also successes ?
>
> Did you:
>
>    -  try today this exact same plan with this number of threads in JMeter
>    2.13 with a View Results Tree configured as with this plan ?
>    - was JMeter 2.13 running with same JVM version as this one ?
>
> Thanks
>
> On Wed, Feb 1, 2017 at 12:57 PM, Tuukka Mustonen <
> tuukka.mustonen@gmail.com>
> wrote:
>
> > Ok I re-ran the test.
> >
> > Took Heap Dump just before JMeter froze.
> >
> > Took Thread Dumps right after JMeter froze.
> >
> > https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
> >
> > ...
> >
> > For the fun of it, I also tried upgrading to the latest HttpClient 4.5.3
> > version. No effect.
> >
> > I also noticed that if I ping (google) during the perf testing, Windows
> > ping reports "General failure." when JMeter breaks. After killing it,
> > connectivity restores and there are replies from the server again. It
> seems
> > like JMeter/HttpClient kills my whole network connection...
> >
> > Tuukka
> >
> >
> > On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > philippe.mouawad@gmail.com> wrote:
> >
> > > On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen <
> > tuukka.mustonen@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
> > > slightly
> > > > different behavior in my Linux guest. I'll write more about that
> later,
> > > but
> > > > for simplicity I'm testing only on Windows host now:
> > > >
> > > > Thread dumps are downloadable at:
> > > > https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> > > >
> > > > (Took them with 7-8s interval I think, hope it's ok).
> > > >
> > > > I took the thread dumps once JMeter started filling in errors, which
> is
> > > > somewhat after my CPU usage hit 100% (and music playback started to
> > > distort
> > > > etc.).
> > > >
> > > > I also created the head dump, note that I took it after running the
> > test
> > > > and after restarting JMeter, as JMeter froze during the test and I
> had
> > to
> > > > kill it. DId you mean I should take it *during* the test?
> > >
> > >
> > > YES.
> > >
> > >
> > > > Anyway, before
> > > > sharing it I want to make sure it can't contain anything
> confidential?
> > > > (parts of my test plan, passwords, environment variables,
> anything...)
> > > >
> > > > Tuukka
> > > >
> > > >
> > > >
> > > > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > > > support@ubikloadpack.com> wrote:
> > > >
> > > > > Hello,
> > > > > To investigate the problem:
> > > > > 1/ to create  thread dumps, install a JDK of same version as the
> one
> > > > > running your JMeter.
> > > > >
> > > > > Run until you get the problem, then call using same user as the one
> > > > running
> > > > > JMeter 3 times at 5s interval:
> > > > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> > > > >
> > > > > It will also help if you provide the structure of your Test Plan.
> > > > >
> > > > > 2/ It would also help if you can provide a Heap Dump. With JMeter
> > GUI,
> > > > use
> > > > > menu Help > Create a Heap Dump
> > > > > This will create a Heap Dump (rather big file) , it would help a
> lot
> > if
> > > > you
> > > > > can make it available.
> > > > >
> > > > > Regards
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > > > tuukka.mustonen@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > @Philippe:
> > > > > >
> > > > > > Yeah, I am normally running in non-GUI mode, but to
> > > > > > study/inspect/understand this problem I am running in GUI mode.
> > > JMeter
> > > > > 2.13
> > > > > > works just fine in GUI mode for these tests so JMeter 3.x should
> > too.
> > > > > >
> > > > > > I'll give the nightly build a try.
> > > > > >
> > > > > > Tuukka
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > > > > > philippe.mouawad@gmail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > tuukka.mustonen@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > @Philippe:
> > > > > > > >
> > > > > > > > 1. Yeah, I compared them with the same plan, same machine.
> > Should
> > > > be
> > > > > > > > everything equal.
> > > > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
> -Xmx512m)
> > > and
> > > > as
> > > > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them
> > much
> > > > > > > because I
> > > > > > > > don't believe the issue lies there.
> > > > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > > > > > > > 4. I am not familiar with that but I can try. If you have
> > > > > instructions
> > > > > > to
> > > > > > > > point to, please do.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > 5. I am running with GUI.
> > > > > > >
> > > > > > >
> > > > > > > This is a known anti-pattern .
> > > > > > > switch to non gui mode if you want correct results.
> > > > > > > http://jmeter.apache.org/usermanual/best-practices.html
> > > > > > >
> > > > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> > > tuning_tips/
> > > > > > >
> > > > > > >
> > > > > > > > About file sending:
> > > > > > > >
> > > > > > > > 6. Nope. I'm just doing simple GETs.
> > > > > > >
> > > > > > >  Could you give nightly build a try ?
> > > > > > > http://jmeter.apache.org/nightly.html
> > > > > > >
> > > > > > >
> > > > > > > > Tuukka
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > > > > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Also are you sending files in post body ?
> > > > > > > > >
> > > > > > > > > Thank you
> > > > > > > > >
> > > > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > > Few questions, please try to answer them all:
> > > > > > > > > >
> > > > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
> > > machine...
> > > > > > > > everything
> > > > > > > > > > equal ?
> > > > > > > > > > 2/ What are your Xmx settings
> > > > > > > > > > 3/ does your plan download embedded resources ?
> > > > > > > > > > 4/ when issue happens can you make 3 thread dumps at 5
> > > seconds
> > > > > > > interval
> > > > > > > > > > and send file ? (use jstack)
> > > > > > > > > > 5/ are you running gui or non gui ?
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Regards
> > > > > > > > > >
> > > > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > > > > > > <javascript:;>');>>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi,
> > > > > > > > > >>
> > > > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high
> CPU
> > > > usage
> > > > > > > that
> > > > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up and
> > > after
> > > > > > having
> > > > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it
> hits
> > > 100%
> > > > > on
> > > > > > my
> > > > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I have
> to
> > > > kill
> > > > > > it.
> > > > > > > > > >>
> > > > > > > > > >> On 2.13 everything goes smoothly (with default
> settings).
> > > CPU
> > > > > > usage
> > > > > > > > > stays
> > > > > > > > > >> at ~30% even with 500 users.
> > > > > > > > > >>
> > > > > > > > > >> The experience is similar on both Windows 10 and Linux
> > > (Ubuntu
> > > > > > 14.04
> > > > > > > > LTS
> > > > > > > > > >> guest in Virtualbox).
> > > > > > > > > >>
> > > > > > > > > >> The problems I saw in JMeter 3:
> > > > > > > > > >>
> > > > > > > > > >> - java.net.SocketException: Permission denied: connect
> > > > > > > > > >> - After a while it runs out of heap space
> > > > > > > > > >>
> > > > > > > > > >> For "Permission denied: connect" I tried to set
> > > > > > > > > >> java.net.preferIPv4Stack=true
> > > > > > > > > >> (in a few places to make sure it would really be in use)
> > and
> > > > > tried
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > >> JMeter as administrator (in Win). No effect.
> > > > > > > > > >>
> > > > > > > > > >> I also tried increasing heap size, but that didn't help
> > > > either.
> > > > > > And
> > > > > > > I
> > > > > > > > > >> think
> > > > > > > > > >> memory size is not the root cause here - rather the
> > result.
> > > > > > > > > >>
> > > > > > > > > >> I am normally connected to VPN but I also tried
> disabling
> > > it.
> > > > No
> > > > > > > > effect.
> > > > > > > > > >>
> > > > > > > > > >> I tried disabling with and without IPv6 enabled in
> Windows
> > > > > network
> > > > > > > > > >> connection settings. No effect.
> > > > > > > > > >>
> > > > > > > > > >> I am running a few extra JMeter plugins. Tried cutting
> my
> > > test
> > > > > > > > scenario
> > > > > > > > > to
> > > > > > > > > >> minimum. No effect.
> > > > > > > > > >>
> > > > > > > > > >> I am running anti-virus software that I cannot disable,
> > due
> > > to
> > > > > > this
> > > > > > > > > being
> > > > > > > > > >> forced by IT department. However, it's not a problem
> with
> > > 2.13
> > > > > so
> > > > > > I
> > > > > > > > > assume
> > > > > > > > > >> it shouldn't be a problem here either.
> > > > > > > > > >>
> > > > > > > > > >> FINALLY, once I set HTTP request implementation from
> > Apache
> > > > > > > > HttpClient 4
> > > > > > > > > >> to
> > > > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays
> > > where
> > > > > it
> > > > > > > did
> > > > > > > > > with
> > > > > > > > > >> JMeter 2.13.
> > > > > > > > > >>
> > > > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update 111
> on
> > > > > Linux).
> > > > > > > > > >>
> > > > > > > > > >> Why does (the recommended) HttpClient 4 cause problems
> > here?
> > > > Is
> > > > > > > this a
> > > > > > > > > >> know
> > > > > > > > > >> bug/issue? JMeter 3.x is running newer version compared
> to
> > > > 2.13,
> > > > > > but
> > > > > > > > > what
> > > > > > > > > >> change in HttpClient 4 might cause the problems
> described?
> > > > > > > > > >>
> > > > > > > > > >> Any ideas?
> > > > > > > > > >>
> > > > > > > > > >> Regards,
> > > > > > > > > >> Tuukka
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Cordialement.
> > > > > > > > > > Philippe Mouawad.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Cordialement.
> > > > > > > > > Philippe Mouawad.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Cordialement.
> > > > > > > Philippe Mouawad.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cordialement.
> > > Philippe Mouawad.
> > >
> >
>
>
>
> --
>
> Regards
> Ubik Load Pack <http://ubikloadpack.com> Team
> Follow us on Twitter <http://twitter.com/ubikloadpack>
>
>
> Cordialement
> L'équipe Ubik Load Pack <http://ubikloadpack.com>
> Suivez-nous sur Twitter <http://twitter.com/ubikloadpack>
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by UBIK LOAD PACK Support <su...@ubikloadpack.com>.

Hello,
From thread dump I see you have in GUI mode around  500 threads and you
using a View Resuls Tree and Summary Report ?
Does view results Tree only catch errors or also successes ?

Did you:

   -  try today this exact same plan with this number of threads in JMeter
   2.13 with a View Results Tree configured as with this plan ?
   - was JMeter 2.13 running with same JVM version as this one ?

Thanks

On Wed, Feb 1, 2017 at 12:57 PM, Tuukka Mustonen <tu...@gmail.com>
wrote:

> Ok I re-ran the test.
>
> Took Heap Dump just before JMeter froze.
>
> Took Thread Dumps right after JMeter froze.
>
> https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
>
> ...
>
> For the fun of it, I also tried upgrading to the latest HttpClient 4.5.3
> version. No effect.
>
> I also noticed that if I ping (google) during the perf testing, Windows
> ping reports "General failure." when JMeter breaks. After killing it,
> connectivity restores and there are replies from the server again. It seems
> like JMeter/HttpClient kills my whole network connection...
>
> Tuukka
>
>
> On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> philippe.mouawad@gmail.com> wrote:
>
> > On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen <
> tuukka.mustonen@gmail.com
> > >
> > wrote:
> >
> > > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
> > slightly
> > > different behavior in my Linux guest. I'll write more about that later,
> > but
> > > for simplicity I'm testing only on Windows host now:
> > >
> > > Thread dumps are downloadable at:
> > > https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> > >
> > > (Took them with 7-8s interval I think, hope it's ok).
> > >
> > > I took the thread dumps once JMeter started filling in errors, which is
> > > somewhat after my CPU usage hit 100% (and music playback started to
> > distort
> > > etc.).
> > >
> > > I also created the head dump, note that I took it after running the
> test
> > > and after restarting JMeter, as JMeter froze during the test and I had
> to
> > > kill it. DId you mean I should take it *during* the test?
> >
> >
> > YES.
> >
> >
> > > Anyway, before
> > > sharing it I want to make sure it can't contain anything confidential?
> > > (parts of my test plan, passwords, environment variables, anything...)
> > >
> > > Tuukka
> > >
> > >
> > >
> > > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > > support@ubikloadpack.com> wrote:
> > >
> > > > Hello,
> > > > To investigate the problem:
> > > > 1/ to create  thread dumps, install a JDK of same version as the one
> > > > running your JMeter.
> > > >
> > > > Run until you get the problem, then call using same user as the one
> > > running
> > > > JMeter 3 times at 5s interval:
> > > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> > > >
> > > > It will also help if you provide the structure of your Test Plan.
> > > >
> > > > 2/ It would also help if you can provide a Heap Dump. With JMeter
> GUI,
> > > use
> > > > menu Help > Create a Heap Dump
> > > > This will create a Heap Dump (rather big file) , it would help a lot
> if
> > > you
> > > > can make it available.
> > > >
> > > > Regards
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > > tuukka.mustonen@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > @Philippe:
> > > > >
> > > > > Yeah, I am normally running in non-GUI mode, but to
> > > > > study/inspect/understand this problem I am running in GUI mode.
> > JMeter
> > > > 2.13
> > > > > works just fine in GUI mode for these tests so JMeter 3.x should
> too.
> > > > >
> > > > > I'll give the nightly build a try.
> > > > >
> > > > > Tuukka
> > > > >
> > > > >
> > > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > > > > philippe.mouawad@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > tuukka.mustonen@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > @Philippe:
> > > > > > >
> > > > > > > 1. Yeah, I compared them with the same plan, same machine.
> Should
> > > be
> > > > > > > everything equal.
> > > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m -Xmx512m)
> > and
> > > as
> > > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them
> much
> > > > > > because I
> > > > > > > don't believe the issue lies there.
> > > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > > > > > > 4. I am not familiar with that but I can try. If you have
> > > > instructions
> > > > > to
> > > > > > > point to, please do.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > 5. I am running with GUI.
> > > > > >
> > > > > >
> > > > > > This is a known anti-pattern .
> > > > > > switch to non gui mode if you want correct results.
> > > > > > http://jmeter.apache.org/usermanual/best-practices.html
> > > > > >
> > > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> > tuning_tips/
> > > > > >
> > > > > >
> > > > > > > About file sending:
> > > > > > >
> > > > > > > 6. Nope. I'm just doing simple GETs.
> > > > > >
> > > > > >  Could you give nightly build a try ?
> > > > > > http://jmeter.apache.org/nightly.html
> > > > > >
> > > > > >
> > > > > > > Tuukka
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > > > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Also are you sending files in post body ?
> > > > > > > >
> > > > > > > > Thank you
> > > > > > > >
> > > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > > Few questions, please try to answer them all:
> > > > > > > > >
> > > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
> > machine...
> > > > > > > everything
> > > > > > > > > equal ?
> > > > > > > > > 2/ What are your Xmx settings
> > > > > > > > > 3/ does your plan download embedded resources ?
> > > > > > > > > 4/ when issue happens can you make 3 thread dumps at 5
> > seconds
> > > > > > interval
> > > > > > > > > and send file ? (use jstack)
> > > > > > > > > 5/ are you running gui or non gui ?
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Regards
> > > > > > > > >
> > > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > > > > > <javascript:;>');>>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi,
> > > > > > > > >>
> > > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high CPU
> > > usage
> > > > > > that
> > > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up and
> > after
> > > > > having
> > > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it hits
> > 100%
> > > > on
> > > > > my
> > > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I have to
> > > kill
> > > > > it.
> > > > > > > > >>
> > > > > > > > >> On 2.13 everything goes smoothly (with default settings).
> > CPU
> > > > > usage
> > > > > > > > stays
> > > > > > > > >> at ~30% even with 500 users.
> > > > > > > > >>
> > > > > > > > >> The experience is similar on both Windows 10 and Linux
> > (Ubuntu
> > > > > 14.04
> > > > > > > LTS
> > > > > > > > >> guest in Virtualbox).
> > > > > > > > >>
> > > > > > > > >> The problems I saw in JMeter 3:
> > > > > > > > >>
> > > > > > > > >> - java.net.SocketException: Permission denied: connect
> > > > > > > > >> - After a while it runs out of heap space
> > > > > > > > >>
> > > > > > > > >> For "Permission denied: connect" I tried to set
> > > > > > > > >> java.net.preferIPv4Stack=true
> > > > > > > > >> (in a few places to make sure it would really be in use)
> and
> > > > tried
> > > > > > to
> > > > > > > > run
> > > > > > > > >> JMeter as administrator (in Win). No effect.
> > > > > > > > >>
> > > > > > > > >> I also tried increasing heap size, but that didn't help
> > > either.
> > > > > And
> > > > > > I
> > > > > > > > >> think
> > > > > > > > >> memory size is not the root cause here - rather the
> result.
> > > > > > > > >>
> > > > > > > > >> I am normally connected to VPN but I also tried disabling
> > it.
> > > No
> > > > > > > effect.
> > > > > > > > >>
> > > > > > > > >> I tried disabling with and without IPv6 enabled in Windows
> > > > network
> > > > > > > > >> connection settings. No effect.
> > > > > > > > >>
> > > > > > > > >> I am running a few extra JMeter plugins. Tried cutting my
> > test
> > > > > > > scenario
> > > > > > > > to
> > > > > > > > >> minimum. No effect.
> > > > > > > > >>
> > > > > > > > >> I am running anti-virus software that I cannot disable,
> due
> > to
> > > > > this
> > > > > > > > being
> > > > > > > > >> forced by IT department. However, it's not a problem with
> > 2.13
> > > > so
> > > > > I
> > > > > > > > assume
> > > > > > > > >> it shouldn't be a problem here either.
> > > > > > > > >>
> > > > > > > > >> FINALLY, once I set HTTP request implementation from
> Apache
> > > > > > > HttpClient 4
> > > > > > > > >> to
> > > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays
> > where
> > > > it
> > > > > > did
> > > > > > > > with
> > > > > > > > >> JMeter 2.13.
> > > > > > > > >>
> > > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update 111 on
> > > > Linux).
> > > > > > > > >>
> > > > > > > > >> Why does (the recommended) HttpClient 4 cause problems
> here?
> > > Is
> > > > > > this a
> > > > > > > > >> know
> > > > > > > > >> bug/issue? JMeter 3.x is running newer version compared to
> > > 2.13,
> > > > > but
> > > > > > > > what
> > > > > > > > >> change in HttpClient 4 might cause the problems described?
> > > > > > > > >>
> > > > > > > > >> Any ideas?
> > > > > > > > >>
> > > > > > > > >> Regards,
> > > > > > > > >> Tuukka
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Cordialement.
> > > > > > > > > Philippe Mouawad.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Cordialement.
> > > > > > > > Philippe Mouawad.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Cordialement.
> > > > > > Philippe Mouawad.
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
>



-- 

Regards
Ubik Load Pack <http://ubikloadpack.com> Team
Follow us on Twitter <http://twitter.com/ubikloadpack>


Cordialement
L'équipe Ubik Load Pack <http://ubikloadpack.com>
Suivez-nous sur Twitter <http://twitter.com/ubikloadpack>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

Weird, Google Drive excludes it for some reason.

Anyway, here's a direct link to just the Heap Dump:
https://drive.google.com/file/d/0B1uBdVuLED5NMWRkY0t3c3ktNTA/view

Tuukka


On Wed, Feb 1, 2017 at 2:28 PM, Philippe Mouawad <philippe.mouawad@gmail.com
> wrote:

> Hi,
> I only see 3 files for the Thread dumps.
>
> Did you make it public ?
> Thanks
>
> On Wed, Feb 1, 2017 at 1:26 PM, Tuukka Mustonen <tuukka.mustonen@gmail.com
> >
> wrote:
>
> > *dump_20170201_124717_708.hprof* is the file that JMeter created. That
> > should be in the linked folder...
> >
> > Tuukka
> >
> > Tuukka
> >
> >
> > On Wed, Feb 1, 2017 at 2:09 PM, Philippe Mouawad <
> > philippe.mouawad@gmail.com
> > > wrote:
> >
> > > Hi,
> > > I don't see any heap dump.
> > > When JMeter creates it it shows the file name.
> > > Regards
> > >
> > > On Wed, Feb 1, 2017 at 12:57 PM, Tuukka Mustonen <
> > > tuukka.mustonen@gmail.com>
> > > wrote:
> > >
> > > > Ok I re-ran the test.
> > > >
> > > > Took Heap Dump just before JMeter froze.
> > > >
> > > > Took Thread Dumps right after JMeter froze.
> > > >
> > > > https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
> > > >
> > > > ...
> > > >
> > > > For the fun of it, I also tried upgrading to the latest HttpClient
> > 4.5.3
> > > > version. No effect.
> > > >
> > > > I also noticed that if I ping (google) during the perf testing,
> Windows
> > > > ping reports "General failure." when JMeter breaks. After killing it,
> > > > connectivity restores and there are replies from the server again. It
> > > seems
> > > > like JMeter/HttpClient kills my whole network connection...
> > > >
> > > > Tuukka
> > > >
> > > >
> > > > On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > > > philippe.mouawad@gmail.com> wrote:
> > > >
> > > > > On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen <
> > > > tuukka.mustonen@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi, I'm running solely on Win 10 now. In re-attempts I think I
> saw
> > > > > slightly
> > > > > > different behavior in my Linux guest. I'll write more about that
> > > later,
> > > > > but
> > > > > > for simplicity I'm testing only on Windows host now:
> > > > > >
> > > > > > Thread dumps are downloadable at:
> > > > > > https://drive.google.com/drive/folders/
> > 0B1uBdVuLED5NS1RFbEFuSS1iejA
> > > > > >
> > > > > > (Took them with 7-8s interval I think, hope it's ok).
> > > > > >
> > > > > > I took the thread dumps once JMeter started filling in errors,
> > which
> > > is
> > > > > > somewhat after my CPU usage hit 100% (and music playback started
> to
> > > > > distort
> > > > > > etc.).
> > > > > >
> > > > > > I also created the head dump, note that I took it after running
> the
> > > > test
> > > > > > and after restarting JMeter, as JMeter froze during the test and
> I
> > > had
> > > > to
> > > > > > kill it. DId you mean I should take it *during* the test?
> > > > >
> > > > >
> > > > > YES.
> > > > >
> > > > >
> > > > > > Anyway, before
> > > > > > sharing it I want to make sure it can't contain anything
> > > confidential?
> > > > > > (parts of my test plan, passwords, environment variables,
> > > anything...)
> > > > > >
> > > > > > Tuukka
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > > > > > support@ubikloadpack.com> wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > > To investigate the problem:
> > > > > > > 1/ to create  thread dumps, install a JDK of same version as
> the
> > > one
> > > > > > > running your JMeter.
> > > > > > >
> > > > > > > Run until you get the problem, then call using same user as the
> > one
> > > > > > running
> > > > > > > JMeter 3 times at 5s interval:
> > > > > > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> > > > > > >
> > > > > > > It will also help if you provide the structure of your Test
> Plan.
> > > > > > >
> > > > > > > 2/ It would also help if you can provide a Heap Dump. With
> JMeter
> > > > GUI,
> > > > > > use
> > > > > > > menu Help > Create a Heap Dump
> > > > > > > This will create a Heap Dump (rather big file) , it would help
> a
> > > lot
> > > > if
> > > > > > you
> > > > > > > can make it available.
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > > > > > tuukka.mustonen@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > @Philippe:
> > > > > > > >
> > > > > > > > Yeah, I am normally running in non-GUI mode, but to
> > > > > > > > study/inspect/understand this problem I am running in GUI
> mode.
> > > > > JMeter
> > > > > > > 2.13
> > > > > > > > works just fine in GUI mode for these tests so JMeter 3.x
> > should
> > > > too.
> > > > > > > >
> > > > > > > > I'll give the nightly build a try.
> > > > > > > >
> > > > > > > > Tuukka
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > > > > > > > philippe.mouawad@gmail.com
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > > > tuukka.mustonen@gmail.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > @Philippe:
> > > > > > > > > >
> > > > > > > > > > 1. Yeah, I compared them with the same plan, same
> machine.
> > > > Should
> > > > > > be
> > > > > > > > > > everything equal.
> > > > > > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
> > > -Xmx512m)
> > > > > and
> > > > > > as
> > > > > > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak
> > them
> > > > much
> > > > > > > > > because I
> > > > > > > > > > don't believe the issue lies there.
> > > > > > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > > > > > > > > > 4. I am not familiar with that but I can try. If you have
> > > > > > > instructions
> > > > > > > > to
> > > > > > > > > > point to, please do.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > 5. I am running with GUI.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This is a known anti-pattern .
> > > > > > > > > switch to non gui mode if you want correct results.
> > > > > > > > > http://jmeter.apache.org/usermanual/best-practices.html
> > > > > > > > >
> > > > > > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> > > > > tuning_tips/
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > About file sending:
> > > > > > > > > >
> > > > > > > > > > 6. Nope. I'm just doing simple GETs.
> > > > > > > > >
> > > > > > > > >  Could you give nightly build a try ?
> > > > > > > > > http://jmeter.apache.org/nightly.html
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > > Tuukka
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > > > > > > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Also are you sending files in post body ?
> > > > > > > > > > >
> > > > > > > > > > > Thank you
> > > > > > > > > > >
> > > > > > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > > > > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hello,
> > > > > > > > > > > > Few questions, please try to answer them all:
> > > > > > > > > > > >
> > > > > > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
> > > > > machine...
> > > > > > > > > > everything
> > > > > > > > > > > > equal ?
> > > > > > > > > > > > 2/ What are your Xmx settings
> > > > > > > > > > > > 3/ does your plan download embedded resources ?
> > > > > > > > > > > > 4/ when issue happens can you make 3 thread dumps at
> 5
> > > > > seconds
> > > > > > > > > interval
> > > > > > > > > > > > and send file ? (use jstack)
> > > > > > > > > > > > 5/ are you running gui or non gui ?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > > Regards
> > > > > > > > > > > >
> > > > > > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > > > > > > > > > <javascript:_e(%7B%7D,'cvml','
> > tuukka.mustonen@gmail.com
> > > > > > > > > <javascript:;>');>>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> Hi,
> > > > > > > > > > > >>
> > > > > > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience
> high
> > > CPU
> > > > > > usage
> > > > > > > > > that
> > > > > > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up
> > and
> > > > > after
> > > > > > > > having
> > > > > > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it
> > > hits
> > > > > 100%
> > > > > > > on
> > > > > > > > my
> > > > > > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I
> > have
> > > to
> > > > > > kill
> > > > > > > > it.
> > > > > > > > > > > >>
> > > > > > > > > > > >> On 2.13 everything goes smoothly (with default
> > > settings).
> > > > > CPU
> > > > > > > > usage
> > > > > > > > > > > stays
> > > > > > > > > > > >> at ~30% even with 500 users.
> > > > > > > > > > > >>
> > > > > > > > > > > >> The experience is similar on both Windows 10 and
> Linux
> > > > > (Ubuntu
> > > > > > > > 14.04
> > > > > > > > > > LTS
> > > > > > > > > > > >> guest in Virtualbox).
> > > > > > > > > > > >>
> > > > > > > > > > > >> The problems I saw in JMeter 3:
> > > > > > > > > > > >>
> > > > > > > > > > > >> - java.net.SocketException: Permission denied:
> connect
> > > > > > > > > > > >> - After a while it runs out of heap space
> > > > > > > > > > > >>
> > > > > > > > > > > >> For "Permission denied: connect" I tried to set
> > > > > > > > > > > >> java.net.preferIPv4Stack=true
> > > > > > > > > > > >> (in a few places to make sure it would really be in
> > use)
> > > > and
> > > > > > > tried
> > > > > > > > > to
> > > > > > > > > > > run
> > > > > > > > > > > >> JMeter as administrator (in Win). No effect.
> > > > > > > > > > > >>
> > > > > > > > > > > >> I also tried increasing heap size, but that didn't
> > help
> > > > > > either.
> > > > > > > > And
> > > > > > > > > I
> > > > > > > > > > > >> think
> > > > > > > > > > > >> memory size is not the root cause here - rather the
> > > > result.
> > > > > > > > > > > >>
> > > > > > > > > > > >> I am normally connected to VPN but I also tried
> > > disabling
> > > > > it.
> > > > > > No
> > > > > > > > > > effect.
> > > > > > > > > > > >>
> > > > > > > > > > > >> I tried disabling with and without IPv6 enabled in
> > > Windows
> > > > > > > network
> > > > > > > > > > > >> connection settings. No effect.
> > > > > > > > > > > >>
> > > > > > > > > > > >> I am running a few extra JMeter plugins. Tried
> cutting
> > > my
> > > > > test
> > > > > > > > > > scenario
> > > > > > > > > > > to
> > > > > > > > > > > >> minimum. No effect.
> > > > > > > > > > > >>
> > > > > > > > > > > >> I am running anti-virus software that I cannot
> > disable,
> > > > due
> > > > > to
> > > > > > > > this
> > > > > > > > > > > being
> > > > > > > > > > > >> forced by IT department. However, it's not a problem
> > > with
> > > > > 2.13
> > > > > > > so
> > > > > > > > I
> > > > > > > > > > > assume
> > > > > > > > > > > >> it shouldn't be a problem here either.
> > > > > > > > > > > >>
> > > > > > > > > > > >> FINALLY, once I set HTTP request implementation from
> > > > Apache
> > > > > > > > > > HttpClient 4
> > > > > > > > > > > >> to
> > > > > > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU
> > stays
> > > > > where
> > > > > > > it
> > > > > > > > > did
> > > > > > > > > > > with
> > > > > > > > > > > >> JMeter 2.13.
> > > > > > > > > > > >>
> > > > > > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update
> > 111
> > > on
> > > > > > > Linux).
> > > > > > > > > > > >>
> > > > > > > > > > > >> Why does (the recommended) HttpClient 4 cause
> problems
> > > > here?
> > > > > > Is
> > > > > > > > > this a
> > > > > > > > > > > >> know
> > > > > > > > > > > >> bug/issue? JMeter 3.x is running newer version
> > compared
> > > to
> > > > > > 2.13,
> > > > > > > > but
> > > > > > > > > > > what
> > > > > > > > > > > >> change in HttpClient 4 might cause the problems
> > > described?
> > > > > > > > > > > >>
> > > > > > > > > > > >> Any ideas?
> > > > > > > > > > > >>
> > > > > > > > > > > >> Regards,
> > > > > > > > > > > >> Tuukka
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > Cordialement.
> > > > > > > > > > > > Philippe Mouawad.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Cordialement.
> > > > > > > > > > > Philippe Mouawad.
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Cordialement.
> > > > > > > > > Philippe Mouawad.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cordialement.
> > > > > Philippe Mouawad.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cordialement.
> > > Philippe Mouawad.
> > >
> >
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

Hi,
I only see 3 files for the Thread dumps.

Did you make it public ?
Thanks

On Wed, Feb 1, 2017 at 1:26 PM, Tuukka Mustonen <tu...@gmail.com>
wrote:

> *dump_20170201_124717_708.hprof* is the file that JMeter created. That
> should be in the linked folder...
>
> Tuukka
>
> Tuukka
>
>
> On Wed, Feb 1, 2017 at 2:09 PM, Philippe Mouawad <
> philippe.mouawad@gmail.com
> > wrote:
>
> > Hi,
> > I don't see any heap dump.
> > When JMeter creates it it shows the file name.
> > Regards
> >
> > On Wed, Feb 1, 2017 at 12:57 PM, Tuukka Mustonen <
> > tuukka.mustonen@gmail.com>
> > wrote:
> >
> > > Ok I re-ran the test.
> > >
> > > Took Heap Dump just before JMeter froze.
> > >
> > > Took Thread Dumps right after JMeter froze.
> > >
> > > https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
> > >
> > > ...
> > >
> > > For the fun of it, I also tried upgrading to the latest HttpClient
> 4.5.3
> > > version. No effect.
> > >
> > > I also noticed that if I ping (google) during the perf testing, Windows
> > > ping reports "General failure." when JMeter breaks. After killing it,
> > > connectivity restores and there are replies from the server again. It
> > seems
> > > like JMeter/HttpClient kills my whole network connection...
> > >
> > > Tuukka
> > >
> > >
> > > On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > > philippe.mouawad@gmail.com> wrote:
> > >
> > > > On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen <
> > > tuukka.mustonen@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
> > > > slightly
> > > > > different behavior in my Linux guest. I'll write more about that
> > later,
> > > > but
> > > > > for simplicity I'm testing only on Windows host now:
> > > > >
> > > > > Thread dumps are downloadable at:
> > > > > https://drive.google.com/drive/folders/
> 0B1uBdVuLED5NS1RFbEFuSS1iejA
> > > > >
> > > > > (Took them with 7-8s interval I think, hope it's ok).
> > > > >
> > > > > I took the thread dumps once JMeter started filling in errors,
> which
> > is
> > > > > somewhat after my CPU usage hit 100% (and music playback started to
> > > > distort
> > > > > etc.).
> > > > >
> > > > > I also created the head dump, note that I took it after running the
> > > test
> > > > > and after restarting JMeter, as JMeter froze during the test and I
> > had
> > > to
> > > > > kill it. DId you mean I should take it *during* the test?
> > > >
> > > >
> > > > YES.
> > > >
> > > >
> > > > > Anyway, before
> > > > > sharing it I want to make sure it can't contain anything
> > confidential?
> > > > > (parts of my test plan, passwords, environment variables,
> > anything...)
> > > > >
> > > > > Tuukka
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > > > > support@ubikloadpack.com> wrote:
> > > > >
> > > > > > Hello,
> > > > > > To investigate the problem:
> > > > > > 1/ to create  thread dumps, install a JDK of same version as the
> > one
> > > > > > running your JMeter.
> > > > > >
> > > > > > Run until you get the problem, then call using same user as the
> one
> > > > > running
> > > > > > JMeter 3 times at 5s interval:
> > > > > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> > > > > >
> > > > > > It will also help if you provide the structure of your Test Plan.
> > > > > >
> > > > > > 2/ It would also help if you can provide a Heap Dump. With JMeter
> > > GUI,
> > > > > use
> > > > > > menu Help > Create a Heap Dump
> > > > > > This will create a Heap Dump (rather big file) , it would help a
> > lot
> > > if
> > > > > you
> > > > > > can make it available.
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > > > > tuukka.mustonen@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > @Philippe:
> > > > > > >
> > > > > > > Yeah, I am normally running in non-GUI mode, but to
> > > > > > > study/inspect/understand this problem I am running in GUI mode.
> > > > JMeter
> > > > > > 2.13
> > > > > > > works just fine in GUI mode for these tests so JMeter 3.x
> should
> > > too.
> > > > > > >
> > > > > > > I'll give the nightly build a try.
> > > > > > >
> > > > > > > Tuukka
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > > > > > > philippe.mouawad@gmail.com
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > > tuukka.mustonen@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > @Philippe:
> > > > > > > > >
> > > > > > > > > 1. Yeah, I compared them with the same plan, same machine.
> > > Should
> > > > > be
> > > > > > > > > everything equal.
> > > > > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
> > -Xmx512m)
> > > > and
> > > > > as
> > > > > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak
> them
> > > much
> > > > > > > > because I
> > > > > > > > > don't believe the issue lies there.
> > > > > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > > > > > > > > 4. I am not familiar with that but I can try. If you have
> > > > > > instructions
> > > > > > > to
> > > > > > > > > point to, please do.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > > 5. I am running with GUI.
> > > > > > > >
> > > > > > > >
> > > > > > > > This is a known anti-pattern .
> > > > > > > > switch to non gui mode if you want correct results.
> > > > > > > > http://jmeter.apache.org/usermanual/best-practices.html
> > > > > > > >
> > > > > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> > > > tuning_tips/
> > > > > > > >
> > > > > > > >
> > > > > > > > > About file sending:
> > > > > > > > >
> > > > > > > > > 6. Nope. I'm just doing simple GETs.
> > > > > > > >
> > > > > > > >  Could you give nightly build a try ?
> > > > > > > > http://jmeter.apache.org/nightly.html
> > > > > > > >
> > > > > > > >
> > > > > > > > > Tuukka
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > > > > > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Also are you sending files in post body ?
> > > > > > > > > >
> > > > > > > > > > Thank you
> > > > > > > > > >
> > > > > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > > > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello,
> > > > > > > > > > > Few questions, please try to answer them all:
> > > > > > > > > > >
> > > > > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
> > > > machine...
> > > > > > > > > everything
> > > > > > > > > > > equal ?
> > > > > > > > > > > 2/ What are your Xmx settings
> > > > > > > > > > > 3/ does your plan download embedded resources ?
> > > > > > > > > > > 4/ when issue happens can you make 3 thread dumps at 5
> > > > seconds
> > > > > > > > interval
> > > > > > > > > > > and send file ? (use jstack)
> > > > > > > > > > > 5/ are you running gui or non gui ?
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Regards
> > > > > > > > > > >
> > > > > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > > > > > > > > <javascript:_e(%7B%7D,'cvml','
> tuukka.mustonen@gmail.com
> > > > > > > > <javascript:;>');>>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Hi,
> > > > > > > > > > >>
> > > > > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high
> > CPU
> > > > > usage
> > > > > > > > that
> > > > > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up
> and
> > > > after
> > > > > > > having
> > > > > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it
> > hits
> > > > 100%
> > > > > > on
> > > > > > > my
> > > > > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I
> have
> > to
> > > > > kill
> > > > > > > it.
> > > > > > > > > > >>
> > > > > > > > > > >> On 2.13 everything goes smoothly (with default
> > settings).
> > > > CPU
> > > > > > > usage
> > > > > > > > > > stays
> > > > > > > > > > >> at ~30% even with 500 users.
> > > > > > > > > > >>
> > > > > > > > > > >> The experience is similar on both Windows 10 and Linux
> > > > (Ubuntu
> > > > > > > 14.04
> > > > > > > > > LTS
> > > > > > > > > > >> guest in Virtualbox).
> > > > > > > > > > >>
> > > > > > > > > > >> The problems I saw in JMeter 3:
> > > > > > > > > > >>
> > > > > > > > > > >> - java.net.SocketException: Permission denied: connect
> > > > > > > > > > >> - After a while it runs out of heap space
> > > > > > > > > > >>
> > > > > > > > > > >> For "Permission denied: connect" I tried to set
> > > > > > > > > > >> java.net.preferIPv4Stack=true
> > > > > > > > > > >> (in a few places to make sure it would really be in
> use)
> > > and
> > > > > > tried
> > > > > > > > to
> > > > > > > > > > run
> > > > > > > > > > >> JMeter as administrator (in Win). No effect.
> > > > > > > > > > >>
> > > > > > > > > > >> I also tried increasing heap size, but that didn't
> help
> > > > > either.
> > > > > > > And
> > > > > > > > I
> > > > > > > > > > >> think
> > > > > > > > > > >> memory size is not the root cause here - rather the
> > > result.
> > > > > > > > > > >>
> > > > > > > > > > >> I am normally connected to VPN but I also tried
> > disabling
> > > > it.
> > > > > No
> > > > > > > > > effect.
> > > > > > > > > > >>
> > > > > > > > > > >> I tried disabling with and without IPv6 enabled in
> > Windows
> > > > > > network
> > > > > > > > > > >> connection settings. No effect.
> > > > > > > > > > >>
> > > > > > > > > > >> I am running a few extra JMeter plugins. Tried cutting
> > my
> > > > test
> > > > > > > > > scenario
> > > > > > > > > > to
> > > > > > > > > > >> minimum. No effect.
> > > > > > > > > > >>
> > > > > > > > > > >> I am running anti-virus software that I cannot
> disable,
> > > due
> > > > to
> > > > > > > this
> > > > > > > > > > being
> > > > > > > > > > >> forced by IT department. However, it's not a problem
> > with
> > > > 2.13
> > > > > > so
> > > > > > > I
> > > > > > > > > > assume
> > > > > > > > > > >> it shouldn't be a problem here either.
> > > > > > > > > > >>
> > > > > > > > > > >> FINALLY, once I set HTTP request implementation from
> > > Apache
> > > > > > > > > HttpClient 4
> > > > > > > > > > >> to
> > > > > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU
> stays
> > > > where
> > > > > > it
> > > > > > > > did
> > > > > > > > > > with
> > > > > > > > > > >> JMeter 2.13.
> > > > > > > > > > >>
> > > > > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update
> 111
> > on
> > > > > > Linux).
> > > > > > > > > > >>
> > > > > > > > > > >> Why does (the recommended) HttpClient 4 cause problems
> > > here?
> > > > > Is
> > > > > > > > this a
> > > > > > > > > > >> know
> > > > > > > > > > >> bug/issue? JMeter 3.x is running newer version
> compared
> > to
> > > > > 2.13,
> > > > > > > but
> > > > > > > > > > what
> > > > > > > > > > >> change in HttpClient 4 might cause the problems
> > described?
> > > > > > > > > > >>
> > > > > > > > > > >> Any ideas?
> > > > > > > > > > >>
> > > > > > > > > > >> Regards,
> > > > > > > > > > >> Tuukka
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Cordialement.
> > > > > > > > > > > Philippe Mouawad.
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Cordialement.
> > > > > > > > > > Philippe Mouawad.
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Cordialement.
> > > > > > > > Philippe Mouawad.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Cordialement.
> > > > Philippe Mouawad.
> > > >
> > >
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
>



-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

*dump_20170201_124717_708.hprof* is the file that JMeter created. That
should be in the linked folder...

Tuukka

Tuukka


On Wed, Feb 1, 2017 at 2:09 PM, Philippe Mouawad <philippe.mouawad@gmail.com
> wrote:

> Hi,
> I don't see any heap dump.
> When JMeter creates it it shows the file name.
> Regards
>
> On Wed, Feb 1, 2017 at 12:57 PM, Tuukka Mustonen <
> tuukka.mustonen@gmail.com>
> wrote:
>
> > Ok I re-ran the test.
> >
> > Took Heap Dump just before JMeter froze.
> >
> > Took Thread Dumps right after JMeter froze.
> >
> > https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
> >
> > ...
> >
> > For the fun of it, I also tried upgrading to the latest HttpClient 4.5.3
> > version. No effect.
> >
> > I also noticed that if I ping (google) during the perf testing, Windows
> > ping reports "General failure." when JMeter breaks. After killing it,
> > connectivity restores and there are replies from the server again. It
> seems
> > like JMeter/HttpClient kills my whole network connection...
> >
> > Tuukka
> >
> >
> > On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > philippe.mouawad@gmail.com> wrote:
> >
> > > On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen <
> > tuukka.mustonen@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
> > > slightly
> > > > different behavior in my Linux guest. I'll write more about that
> later,
> > > but
> > > > for simplicity I'm testing only on Windows host now:
> > > >
> > > > Thread dumps are downloadable at:
> > > > https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> > > >
> > > > (Took them with 7-8s interval I think, hope it's ok).
> > > >
> > > > I took the thread dumps once JMeter started filling in errors, which
> is
> > > > somewhat after my CPU usage hit 100% (and music playback started to
> > > distort
> > > > etc.).
> > > >
> > > > I also created the head dump, note that I took it after running the
> > test
> > > > and after restarting JMeter, as JMeter froze during the test and I
> had
> > to
> > > > kill it. DId you mean I should take it *during* the test?
> > >
> > >
> > > YES.
> > >
> > >
> > > > Anyway, before
> > > > sharing it I want to make sure it can't contain anything
> confidential?
> > > > (parts of my test plan, passwords, environment variables,
> anything...)
> > > >
> > > > Tuukka
> > > >
> > > >
> > > >
> > > > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > > > support@ubikloadpack.com> wrote:
> > > >
> > > > > Hello,
> > > > > To investigate the problem:
> > > > > 1/ to create  thread dumps, install a JDK of same version as the
> one
> > > > > running your JMeter.
> > > > >
> > > > > Run until you get the problem, then call using same user as the one
> > > > running
> > > > > JMeter 3 times at 5s interval:
> > > > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> > > > >
> > > > > It will also help if you provide the structure of your Test Plan.
> > > > >
> > > > > 2/ It would also help if you can provide a Heap Dump. With JMeter
> > GUI,
> > > > use
> > > > > menu Help > Create a Heap Dump
> > > > > This will create a Heap Dump (rather big file) , it would help a
> lot
> > if
> > > > you
> > > > > can make it available.
> > > > >
> > > > > Regards
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > > > tuukka.mustonen@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > @Philippe:
> > > > > >
> > > > > > Yeah, I am normally running in non-GUI mode, but to
> > > > > > study/inspect/understand this problem I am running in GUI mode.
> > > JMeter
> > > > > 2.13
> > > > > > works just fine in GUI mode for these tests so JMeter 3.x should
> > too.
> > > > > >
> > > > > > I'll give the nightly build a try.
> > > > > >
> > > > > > Tuukka
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > > > > > philippe.mouawad@gmail.com
> > > > > > > wrote:
> > > > > >
> > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > tuukka.mustonen@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > @Philippe:
> > > > > > > >
> > > > > > > > 1. Yeah, I compared them with the same plan, same machine.
> > Should
> > > > be
> > > > > > > > everything equal.
> > > > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
> -Xmx512m)
> > > and
> > > > as
> > > > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them
> > much
> > > > > > > because I
> > > > > > > > don't believe the issue lies there.
> > > > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > > > > > > > 4. I am not familiar with that but I can try. If you have
> > > > > instructions
> > > > > > to
> > > > > > > > point to, please do.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > 5. I am running with GUI.
> > > > > > >
> > > > > > >
> > > > > > > This is a known anti-pattern .
> > > > > > > switch to non gui mode if you want correct results.
> > > > > > > http://jmeter.apache.org/usermanual/best-practices.html
> > > > > > >
> > > > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> > > tuning_tips/
> > > > > > >
> > > > > > >
> > > > > > > > About file sending:
> > > > > > > >
> > > > > > > > 6. Nope. I'm just doing simple GETs.
> > > > > > >
> > > > > > >  Could you give nightly build a try ?
> > > > > > > http://jmeter.apache.org/nightly.html
> > > > > > >
> > > > > > >
> > > > > > > > Tuukka
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > > > > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Also are you sending files in post body ?
> > > > > > > > >
> > > > > > > > > Thank you
> > > > > > > > >
> > > > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > > Few questions, please try to answer them all:
> > > > > > > > > >
> > > > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
> > > machine...
> > > > > > > > everything
> > > > > > > > > > equal ?
> > > > > > > > > > 2/ What are your Xmx settings
> > > > > > > > > > 3/ does your plan download embedded resources ?
> > > > > > > > > > 4/ when issue happens can you make 3 thread dumps at 5
> > > seconds
> > > > > > > interval
> > > > > > > > > > and send file ? (use jstack)
> > > > > > > > > > 5/ are you running gui or non gui ?
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Regards
> > > > > > > > > >
> > > > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > > > > > > <javascript:;>');>>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi,
> > > > > > > > > >>
> > > > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high
> CPU
> > > > usage
> > > > > > > that
> > > > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up and
> > > after
> > > > > > having
> > > > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it
> hits
> > > 100%
> > > > > on
> > > > > > my
> > > > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I have
> to
> > > > kill
> > > > > > it.
> > > > > > > > > >>
> > > > > > > > > >> On 2.13 everything goes smoothly (with default
> settings).
> > > CPU
> > > > > > usage
> > > > > > > > > stays
> > > > > > > > > >> at ~30% even with 500 users.
> > > > > > > > > >>
> > > > > > > > > >> The experience is similar on both Windows 10 and Linux
> > > (Ubuntu
> > > > > > 14.04
> > > > > > > > LTS
> > > > > > > > > >> guest in Virtualbox).
> > > > > > > > > >>
> > > > > > > > > >> The problems I saw in JMeter 3:
> > > > > > > > > >>
> > > > > > > > > >> - java.net.SocketException: Permission denied: connect
> > > > > > > > > >> - After a while it runs out of heap space
> > > > > > > > > >>
> > > > > > > > > >> For "Permission denied: connect" I tried to set
> > > > > > > > > >> java.net.preferIPv4Stack=true
> > > > > > > > > >> (in a few places to make sure it would really be in use)
> > and
> > > > > tried
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > >> JMeter as administrator (in Win). No effect.
> > > > > > > > > >>
> > > > > > > > > >> I also tried increasing heap size, but that didn't help
> > > > either.
> > > > > > And
> > > > > > > I
> > > > > > > > > >> think
> > > > > > > > > >> memory size is not the root cause here - rather the
> > result.
> > > > > > > > > >>
> > > > > > > > > >> I am normally connected to VPN but I also tried
> disabling
> > > it.
> > > > No
> > > > > > > > effect.
> > > > > > > > > >>
> > > > > > > > > >> I tried disabling with and without IPv6 enabled in
> Windows
> > > > > network
> > > > > > > > > >> connection settings. No effect.
> > > > > > > > > >>
> > > > > > > > > >> I am running a few extra JMeter plugins. Tried cutting
> my
> > > test
> > > > > > > > scenario
> > > > > > > > > to
> > > > > > > > > >> minimum. No effect.
> > > > > > > > > >>
> > > > > > > > > >> I am running anti-virus software that I cannot disable,
> > due
> > > to
> > > > > > this
> > > > > > > > > being
> > > > > > > > > >> forced by IT department. However, it's not a problem
> with
> > > 2.13
> > > > > so
> > > > > > I
> > > > > > > > > assume
> > > > > > > > > >> it shouldn't be a problem here either.
> > > > > > > > > >>
> > > > > > > > > >> FINALLY, once I set HTTP request implementation from
> > Apache
> > > > > > > > HttpClient 4
> > > > > > > > > >> to
> > > > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays
> > > where
> > > > > it
> > > > > > > did
> > > > > > > > > with
> > > > > > > > > >> JMeter 2.13.
> > > > > > > > > >>
> > > > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update 111
> on
> > > > > Linux).
> > > > > > > > > >>
> > > > > > > > > >> Why does (the recommended) HttpClient 4 cause problems
> > here?
> > > > Is
> > > > > > > this a
> > > > > > > > > >> know
> > > > > > > > > >> bug/issue? JMeter 3.x is running newer version compared
> to
> > > > 2.13,
> > > > > > but
> > > > > > > > > what
> > > > > > > > > >> change in HttpClient 4 might cause the problems
> described?
> > > > > > > > > >>
> > > > > > > > > >> Any ideas?
> > > > > > > > > >>
> > > > > > > > > >> Regards,
> > > > > > > > > >> Tuukka
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Cordialement.
> > > > > > > > > > Philippe Mouawad.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Cordialement.
> > > > > > > > > Philippe Mouawad.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Cordialement.
> > > > > > > Philippe Mouawad.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cordialement.
> > > Philippe Mouawad.
> > >
> >
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

Hi,
I don't see any heap dump.
When JMeter creates it it shows the file name.
Regards

On Wed, Feb 1, 2017 at 12:57 PM, Tuukka Mustonen <tu...@gmail.com>
wrote:

> Ok I re-ran the test.
>
> Took Heap Dump just before JMeter froze.
>
> Took Thread Dumps right after JMeter froze.
>
> https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
>
> ...
>
> For the fun of it, I also tried upgrading to the latest HttpClient 4.5.3
> version. No effect.
>
> I also noticed that if I ping (google) during the perf testing, Windows
> ping reports "General failure." when JMeter breaks. After killing it,
> connectivity restores and there are replies from the server again. It seems
> like JMeter/HttpClient kills my whole network connection...
>
> Tuukka
>
>
> On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> philippe.mouawad@gmail.com> wrote:
>
> > On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen <
> tuukka.mustonen@gmail.com
> > >
> > wrote:
> >
> > > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
> > slightly
> > > different behavior in my Linux guest. I'll write more about that later,
> > but
> > > for simplicity I'm testing only on Windows host now:
> > >
> > > Thread dumps are downloadable at:
> > > https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> > >
> > > (Took them with 7-8s interval I think, hope it's ok).
> > >
> > > I took the thread dumps once JMeter started filling in errors, which is
> > > somewhat after my CPU usage hit 100% (and music playback started to
> > distort
> > > etc.).
> > >
> > > I also created the head dump, note that I took it after running the
> test
> > > and after restarting JMeter, as JMeter froze during the test and I had
> to
> > > kill it. DId you mean I should take it *during* the test?
> >
> >
> > YES.
> >
> >
> > > Anyway, before
> > > sharing it I want to make sure it can't contain anything confidential?
> > > (parts of my test plan, passwords, environment variables, anything...)
> > >
> > > Tuukka
> > >
> > >
> > >
> > > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > > support@ubikloadpack.com> wrote:
> > >
> > > > Hello,
> > > > To investigate the problem:
> > > > 1/ to create  thread dumps, install a JDK of same version as the one
> > > > running your JMeter.
> > > >
> > > > Run until you get the problem, then call using same user as the one
> > > running
> > > > JMeter 3 times at 5s interval:
> > > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> > > >
> > > > It will also help if you provide the structure of your Test Plan.
> > > >
> > > > 2/ It would also help if you can provide a Heap Dump. With JMeter
> GUI,
> > > use
> > > > menu Help > Create a Heap Dump
> > > > This will create a Heap Dump (rather big file) , it would help a lot
> if
> > > you
> > > > can make it available.
> > > >
> > > > Regards
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > > tuukka.mustonen@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > @Philippe:
> > > > >
> > > > > Yeah, I am normally running in non-GUI mode, but to
> > > > > study/inspect/understand this problem I am running in GUI mode.
> > JMeter
> > > > 2.13
> > > > > works just fine in GUI mode for these tests so JMeter 3.x should
> too.
> > > > >
> > > > > I'll give the nightly build a try.
> > > > >
> > > > > Tuukka
> > > > >
> > > > >
> > > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > > > > philippe.mouawad@gmail.com
> > > > > > wrote:
> > > > >
> > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > tuukka.mustonen@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > @Philippe:
> > > > > > >
> > > > > > > 1. Yeah, I compared them with the same plan, same machine.
> Should
> > > be
> > > > > > > everything equal.
> > > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m -Xmx512m)
> > and
> > > as
> > > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them
> much
> > > > > > because I
> > > > > > > don't believe the issue lies there.
> > > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > > > > > > 4. I am not familiar with that but I can try. If you have
> > > > instructions
> > > > > to
> > > > > > > point to, please do.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > 5. I am running with GUI.
> > > > > >
> > > > > >
> > > > > > This is a known anti-pattern .
> > > > > > switch to non gui mode if you want correct results.
> > > > > > http://jmeter.apache.org/usermanual/best-practices.html
> > > > > >
> > > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> > tuning_tips/
> > > > > >
> > > > > >
> > > > > > > About file sending:
> > > > > > >
> > > > > > > 6. Nope. I'm just doing simple GETs.
> > > > > >
> > > > > >  Could you give nightly build a try ?
> > > > > > http://jmeter.apache.org/nightly.html
> > > > > >
> > > > > >
> > > > > > > Tuukka
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > > > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > > > > wrote:
> > > > > > >
> > > > > > > > Also are you sending files in post body ?
> > > > > > > >
> > > > > > > > Thank you
> > > > > > > >
> > > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > > Few questions, please try to answer them all:
> > > > > > > > >
> > > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
> > machine...
> > > > > > > everything
> > > > > > > > > equal ?
> > > > > > > > > 2/ What are your Xmx settings
> > > > > > > > > 3/ does your plan download embedded resources ?
> > > > > > > > > 4/ when issue happens can you make 3 thread dumps at 5
> > seconds
> > > > > > interval
> > > > > > > > > and send file ? (use jstack)
> > > > > > > > > 5/ are you running gui or non gui ?
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Regards
> > > > > > > > >
> > > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > > > > > <javascript:;>');>>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi,
> > > > > > > > >>
> > > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high CPU
> > > usage
> > > > > > that
> > > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up and
> > after
> > > > > having
> > > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it hits
> > 100%
> > > > on
> > > > > my
> > > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I have to
> > > kill
> > > > > it.
> > > > > > > > >>
> > > > > > > > >> On 2.13 everything goes smoothly (with default settings).
> > CPU
> > > > > usage
> > > > > > > > stays
> > > > > > > > >> at ~30% even with 500 users.
> > > > > > > > >>
> > > > > > > > >> The experience is similar on both Windows 10 and Linux
> > (Ubuntu
> > > > > 14.04
> > > > > > > LTS
> > > > > > > > >> guest in Virtualbox).
> > > > > > > > >>
> > > > > > > > >> The problems I saw in JMeter 3:
> > > > > > > > >>
> > > > > > > > >> - java.net.SocketException: Permission denied: connect
> > > > > > > > >> - After a while it runs out of heap space
> > > > > > > > >>
> > > > > > > > >> For "Permission denied: connect" I tried to set
> > > > > > > > >> java.net.preferIPv4Stack=true
> > > > > > > > >> (in a few places to make sure it would really be in use)
> and
> > > > tried
> > > > > > to
> > > > > > > > run
> > > > > > > > >> JMeter as administrator (in Win). No effect.
> > > > > > > > >>
> > > > > > > > >> I also tried increasing heap size, but that didn't help
> > > either.
> > > > > And
> > > > > > I
> > > > > > > > >> think
> > > > > > > > >> memory size is not the root cause here - rather the
> result.
> > > > > > > > >>
> > > > > > > > >> I am normally connected to VPN but I also tried disabling
> > it.
> > > No
> > > > > > > effect.
> > > > > > > > >>
> > > > > > > > >> I tried disabling with and without IPv6 enabled in Windows
> > > > network
> > > > > > > > >> connection settings. No effect.
> > > > > > > > >>
> > > > > > > > >> I am running a few extra JMeter plugins. Tried cutting my
> > test
> > > > > > > scenario
> > > > > > > > to
> > > > > > > > >> minimum. No effect.
> > > > > > > > >>
> > > > > > > > >> I am running anti-virus software that I cannot disable,
> due
> > to
> > > > > this
> > > > > > > > being
> > > > > > > > >> forced by IT department. However, it's not a problem with
> > 2.13
> > > > so
> > > > > I
> > > > > > > > assume
> > > > > > > > >> it shouldn't be a problem here either.
> > > > > > > > >>
> > > > > > > > >> FINALLY, once I set HTTP request implementation from
> Apache
> > > > > > > HttpClient 4
> > > > > > > > >> to
> > > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays
> > where
> > > > it
> > > > > > did
> > > > > > > > with
> > > > > > > > >> JMeter 2.13.
> > > > > > > > >>
> > > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update 111 on
> > > > Linux).
> > > > > > > > >>
> > > > > > > > >> Why does (the recommended) HttpClient 4 cause problems
> here?
> > > Is
> > > > > > this a
> > > > > > > > >> know
> > > > > > > > >> bug/issue? JMeter 3.x is running newer version compared to
> > > 2.13,
> > > > > but
> > > > > > > > what
> > > > > > > > >> change in HttpClient 4 might cause the problems described?
> > > > > > > > >>
> > > > > > > > >> Any ideas?
> > > > > > > > >>
> > > > > > > > >> Regards,
> > > > > > > > >> Tuukka
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Cordialement.
> > > > > > > > > Philippe Mouawad.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Cordialement.
> > > > > > > > Philippe Mouawad.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Cordialement.
> > > > > > Philippe Mouawad.
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
>



-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

> Could you open a bugzilla and attach thread dumps when you reproduce this
> issue please ?

Sure, I will if I bump into it again.

Tuukka


On Mon, Feb 6, 2017 at 8:57 AM, Philippe Mouawad <philippe.mouawad@gmail.com
> wrote:

> On Monday, February 6, 2017, Tuukka Mustonen <tu...@gmail.com>
> wrote:
>
> > > https://tools.ietf.org/id/draft-thomson-hybi-http-
> > timeout-01.html#rfc.section.2.1
> > > But server is allowed not to send it.
> > > But I think for a load test, it is better to adjust the 3 parameters
> to a
> > > value that is similar to the server keep-alive timeout
> >
> > Cool, never seen anyone mention/recommend that header :O
> >
> > I actually tried setting TTL to 60 seconds and while I think it worked,
> the
> > result was that in the end of the test, some threads would never close
> > their connection. Tuning TTL down to 10 seconds fixed this. Not sure what
> > was going on, but yeah, that's a potential separate issue - no need to
> > prolong this thread any longer.
>
>
> Could you open a bugzilla and attach thread dumps when you reproduce this
> issue please ?
>
> >
> > > I mean if server does not send timeout, but timeouts connection on its
> > > side, JMeter might
> > > try to reuse a "stale" connection which has a cost and will only drop
> it
> > on
> > > next use.
> > > For high throughput tests it migh be a problem.
> >
> > Alright.
> >
> > > https://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx
> >
> > Interesting. I googled around a bit and it would appear that with default
> > Windows settings one shouldn't hit into it (ephemeral port exhaustion)
> very
> > easily (so I guess your 2s TTL is ok, although it feels overly short).
> But
> > yeah, maybe we have special configuration or something...
> >
> > Thanks again!
> >
> > Tuukka
> >
> >
> >
> > On Sat, Feb 4, 2017 at 10:22 PM, Philippe Mouawad <
> > philippe.mouawad@gmail.com <javascript:;>> wrote:
> >
> > > On Sat, Feb 4, 2017 at 8:05 PM, Tuukka Mustonen <
> > tuukka.mustonen@gmail.com <javascript:;>
> > > >
> > > wrote:
> > >
> > > > > In the case of NGinx , it seems no header timeout is sent.
> > > > > So if you set a value > to 0, you can control the max idle duration
> > of
> > > a
> > > > > connection from Client (JMeter) side to avoid storing too many
> > > > connections
> > > > > that won't be reused and can impact JMeter heap.
> > > > > By default it's not set so no impact.
> > > >
> > > > Hmm, what do you mean by "header timeout"? Afaik, the server cannot
> > > > communicate a timeout for the keep-alived connection to the client.
> The
> > > > server probably has some timeout (I think in nginx it's 75s by
> default)
> > > but
> > > > the client doesn't know it. Server will just close the connection at
> > some
> > > > point.
> > > >
> > > https://tools.ietf.org/id/draft-thomson-hybi-http-
> > > timeout-01.html#rfc.section.2.1
> > > But server is allowed not to send it.
> > > But I think for a load test, it is better to adjust the 3 parameters
> to a
> > > value that is similar to the server keep-alive timeout
> > >
> > >
> > >
> > > > Also I'm not 100% sure what you mean by "to avoid storing too many
> > > > connections that won't be reused"? The client will re-use connection
> > > until
> > > > it's closed by TTL configuration or if the server closes the
> > connection,
> > > > right?
> > >
> > >
> > > I mean if server does not send timeout, but timeouts connection on its
> > > side, JMeter might
> > > try to reuse a "stale" connection which has a cost and will only drop
> it
> > on
> > > next use.
> > > For high throughput tests it migh be a problem.
> > >
> > >
> > >
> > > > I though with idletimeout I could just make sure that client closes
> > > > the connection before the server would (and avoid any potential
> failed
> > > > requests if client sent a request just while the server closes
> > > connection)?
> > > >
> > > You understood correctly
> > >
> > > >
> > > > I'm not sure, the documentation for that parameter could maybe be
> > > improved
> > > > a tiny bit?
> > > >
> > >
> > > If you want to contribute a patch for that you're welcome. It's in
> > > properties_reference.xml.
> > > http://jmeter.apache.org/building.html
> > > Otherwise we'll try to fix it.
> > >
> > > >
> > > > > But investigating your issue, I discovered that in fact
> > > > > "httpclient4.validate_after_inactivity" has no impact due to
> another
> > > bug
> > > > > here I am also responsible of :-( .
> > > > > validate() method always return true so JMeter always consider
> > > connection
> > > > > as valid.
> > > > > So only "httpclient4.time_to_live" does its job here.
> > > >
> > > > You seem to be author of lots of commits - it's only human to forget
> > > > something every now and then :)
> > > >
> > > > > What's happening here is that you're most probably saturating the
> > > dynamic
> > > > > port range
> > > >
> > > > If you mean that I would be going over the number of available
> dynamic
> > > > ports, I don't think that is it. The test breaks at ~40-50 seconds or
> > so
> > > > and at that point there shouldn't be even 10k of connections
> > > (potentially)
> > > > open. But, I'm definitively hitting some limit. I will contact our IT
> > > > department - who knows what magic configuration we have.
> > > >
> > >
> > > https://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx
> > >
> > > >
> > > > > It's because TTL is > 0, so it will be included in the management
> of
> > > idle
> > > > > connections.
> > > > > ...
> > > > > Because JMeter will close connection immediately and inform server
> to
> > > do
> > > > > the same.
> > > >
> > > > Ah, good point. What's the policy (=timeline) for closing the idle
> > > > connections that go over TTL?
> > > >
> > > It will be closed when a new request is made by same user and it has
> > > expired.
> > >
> > > >
> > > > > Thanks a lot for all your tests which helped us narrow this issue.
> > > > > Besides Vladimir and Felix are working on improving performances in
> > GUI
> > > > > Mode but the team still advises against using it for Load Testing.
> > > >
> > > > Cool. Yeah, I run tests normally from Jenkins in command line mode,
> > but I
> > > > always develop/debug in GUI (as does everybody, I would believe).
> When
> > in
> > > > GUI mode, I don't care about performance really - just that the tests
> > > pass.
> > > > But better performance is welcome of course.
> > > >
> > > > > If you used the nightly build (select Jenkins one) you should be
> able
> > > to
> > > > > see that now :
> > > > > httpclient4.validate_after_inactivity = 1700
> > > > > and has an effect.
> > > >
> > > > Thanks for the quick fixes! I think I will test it next week some
> > point,
> > > > just for the curiosity, to see it in effect in Wireshark. I assume
> this
> > > > check sends some dummy package to the server and should be visible in
> > the
> > > > trace. Correct me if wrong.
> > > >
> > > > Thanks. Your feedback is very welcome.
> > >
> > > Tuukka
> > > >
> > > >
> > > >
> > > > On Sat, Feb 4, 2017 at 5:13 PM, Philippe Mouawad <
> > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > wrote:
> > > >
> > > > > Hello Tukka,
> > > > > I have commited :
> > > > > https://bz.apache.org/bugzilla/show_bug.cgi?id=60690
> > > > > https://bz.apache.org/bugzilla/show_bug.cgi?id=60689
> > > > >
> > > > > If you used the nightly build (select Jenkins one) you should be
> able
> > > to
> > > > > see that now :
> > > > > httpclient4.validate_after_inactivity = 1700
> > > > > and has an effect.
> > > > >
> > > > > Still in your particular case as you've tested you'll need to
> > increase
> > > > > httpclient4.time_to_live to avoid too much connections creations
> that
> > > > your
> > > > > OS/Machine/Network does not correctly handle.
> > > > >
> > > > > But beware that this may not be realistic as if you're server calls
> > are
> > > > > made from clients that do not use keep alive or persist their
> > > connections
> > > > > for a short time.
> > > > > To ensure your server correctly handles this , you should test
> from a
> > > > fine
> > > > > tuned machine that has a correct network with target server.
> > > > >
> > > > >
> > > > > Regards
> > > > > @philmdot
> > > > >
> > > > > On Sat, Feb 4, 2017 at 3:16 PM, Philippe Mouawad <
> > > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > > wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > On Sat, Feb 4, 2017 at 12:58 PM, Tuukka Mustonen <
> > > > > > tuukka.mustonen@gmail.com <javascript:;>> wrote:
> > > > > >
> > > > > >> On point, Philippe!
> > > > > >>
> > > > > >> I think *httpclient4.idletimeout* is not of relevance here - my
> > > server
> > > > > >> sends proper Keep-Alive headers so this option shouldn't have
> > > impact.
> > > > > Good
> > > > > >> to know, for sure.
> > > > > >>
> > > > > > In the case of NGinx , it seems no header timeout is sent.
> > > > > > So if you set a value > to 0, you can control the max idle
> duration
> > > of
> > > > a
> > > > > > connection from Client (JMeter) side to avoid storing too many
> > > > > connections
> > > > > > that won't be reused and can impact JMeter heap.
> > > > > > By default it's not set so no impact.
> > > > > >
> > > > > >>
> > > > > >> *httpclient4.validate_after_inactivity* and
> > > *httpclient4.time_to_live
> > > > > >> default* values (2 seconds) sound conflicting: TTL being the
> same
> > as
> > > > > >> validate_after_inactivity leads to connections never getting
> > checked
> > > > as
> > > > > >> connections are discarded after 2 seconds anyway? So
> > > > > >> validate_after_inactivity is not really in use by default if I
> > > > > understood
> > > > > >> it right.
> > > > > >>
> > > > > > Good catch. It somehow went out of my radar, I remember it was
> > > > something
> > > > > I
> > > > > > had in mind but it seems I failed to change it.
> > > > > > It's a bug or at least bad default value.
> > > > > >
> > > > > > But investigating your issue, I discovered that in fact
> > > > > > "httpclient4.validate_after_inactivity" has no impact due to
> > another
> > > > bug
> > > > > > here I am also responsible of :-( .
> > > > > > validate() method always return true so JMeter always consider
> > > > connection
> > > > > > as valid.
> > > > > > So only "httpclient4.time_to_live" does its job here.
> > > > > >
> > > > > >
> > > > > >>
> > > > > >> So I went on and tried these settings:
> > > > > >>
> > > > > >> 1) httpclient4.validate_after_inactivity=0 +
> > > > > >> httpclient4.time_to_live=2000
> > > > > >> -> CRASH
> > > > > >> 2) httpclient4.validate_after_inactivity=0 +
> > > > > >> httpclient4.time_to_live=1000
> > > > > >> -> CRASH
> > > > > >> 3) httpclient4.validate_after_inactivity=1 +
> > > > > >> httpclient4.time_to_live=2000
> > > > > >> -> CRASH
> > > > > >>
> > > > > >> 4) httpclient4.validate_after_inactivity=5000 +
> > > > > >> httpclient4.time_to_live=10000 -> SUCCESS
> > > > > >> 5) httpclient4.validate_after_inactivity=0 +
> > > > > >> httpclient4.time_to_live=10000
> > > > > >> -> SUCCESS
> > > > > >> 6) httpclient4.validate_after_inactivity=1 +
> > > > > >> httpclient4.time_to_live=10000
> > > > > >> -> SUCCESS
> > > > > >> 7) httpclient4.validate_after_inactivity=0 +
> > > > httpclient4.time_to_live=0
> > > > > >> ->
> > > > > >> SUCCESS
> > > > > >> 8) httpclient4.validate_after_inactivity=10000 +
> > > > > >> httpclient4.time_to_live=10000 -> SUCCESS
> > > > > >>
> > > > > >> Clearly increasing TTL makes the difference for me.
> > > > > >> validate_after_inactivity seems to have no effect. Although
> there
> > is
> > > > > some
> > > > > >> packet loss, the connections themselves don't break.
> > > > > >>
> > > > > > Absolutely.
> > > > > >
> > > > > >>
> > > > > >> A) So... maybe my network chip configuration just cannot
> tolerate
> > > > JMeter
> > > > > >> opening fresh connections every 2 seconds? That translates to
> 250
> > > > > >> connections/s with 500 users- maybe it's enough to bloat my
> > network
> > > IO
> > > > > and
> > > > > >> hit some policy (and I'll then get "connect: Permission
> denied")?
> > > > > Sounds a
> > > > > >> bit weird.
> > > > > >>
> > > > > >
> > > > > > It's also your OS, you need to tune the TCP stack to work with
> port
> > > > reuse
> > > > > > speed.
> > > > > > What's happening here is that you're most probably saturating the
> > > > dynamic
> > > > > > port range
> > > > > >
> > > > > >>
> > > > > >> So I also tried:
> > > > > >>
> > > > > >> 9) httpclient4.validate_after_inactivity=0 +
> > > > httpclient4.time_to_live=1
> > > > > >> ->
> > > > > >> CRASH
> > > > > >>
> > > > > >> And it crashes quite instantly. I guess it's because JMeter is
> > > > opening a
> > > > > >> new connection per request...
> > > > > >>
> > > > > > It's because TTL is > 0, so it will be included in the management
> > of
> > > > idle
> > > > > > connections.
> > > > > >
> > > > > >>
> > > > > >> B) ...but if I keep the default 2s settings and just disable
> > > > Keep-Alive,
> > > > > >> then JMeter is opening new connection per request also, and it
> > > works.
> > > > > Why
> > > > > >> the difference?
> > > > > >>
> > > > > > Because JMeter will close connection immediately and inform
> server
> > to
> > > > do
> > > > > > the same.
> > > > > >
> > > > > >
> > > > > >>
> > > > > >> C) Then, I wonder if your "standard workstation" is supposed to
> > > > tolerate
> > > > > >> these defaults? Can my network chip/driver/configuration just be
> > so
> > > > bad?
> > > > > >>
> > > > > >
> > > > > > Yes at least your OS Tuning defaults are not well suited for Load
> > > > > Testing,
> > > > > > you need to tune.
> > > > > >
> > > > > >>
> > > > > >> I see the *httpclient4.validate_after_inactivity* and
> > > > > >> *httpclient4.time_to_live
> > > > > >> default* properties mentioned in the JMeter 3 changelog:
> > > > > >>
> > > > > >> "Improved and better performing validation mechanism for Stale
> > > > > connections
> > > > > >> and Keep-Alive management, see properties
> > > > > >> httpclient4.validate_after_inactivity and
> > httpclient4.time_to_live"
> > > > > >>
> > > > > >
> > > > > > Yes, it was the intention.
> > > > > >
> > > > > >>
> > > > > >> And when I monitored TCP connection in Wireshark, JMeter 2.13
> > indeed
> > > > > keeps
> > > > > >> connections open waaaay longer (seemed a bit random).
> > > > > >>
> > > > > >
> > > > > > Yes, at that time the only solution was to play with idletimeout
> or
> > > set
> > > > > > "http.connection.stalecheck$Boolean=true" which had important
> > impact
> > > > at
> > > > > > high throughput.
> > > > > >
> > > > > >>
> > > > > >> D) So I the new 2s default sounds a bit risky?
> > > > > >>
> > > > > > In your case yes, but you're in a "bad load testing config" :-)
> > > > > >
> > > > > >>
> > > > > >> In the end, increasing TTL seems to fix the issues for me and I
> > can
> > > > > >> proceed
> > > > > >> upgrading to JMeter 3.
> > > > > >>
> > > > > >> Would love to hear your opinion/analysis on the open questions
> > > (A-D),
> > > > > but
> > > > > >> big thanks for your interest in my problem and the precious help
> > > > already
> > > > > >> at
> > > > > >> this point!
> > > > > >>
> > > > > >
> > > > > > Thanks a lot for all your tests which helped us narrow this
> issue.
> > > > > > Besides Vladimir and Felix are working on improving performances
> in
> > > GUI
> > > > > > Mode but the team still advises against using it for Load
> Testing.
> > > > > >
> > > > > >
> > > > > >> Tuukka
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Feb 3, 2017 at 9:40 PM, Philippe Mouawad <
> > > > > >> philippe.mouawad@gmail.com <javascript:;>
> > > > > >> > wrote:
> > > > > >>
> > > > > >> > On Friday, February 3, 2017, Tuukka Mustonen <
> > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > > >> >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Sorry for the delay, got too busy yesterday.
> > > > > >> > >
> > > > > >> > > I have been running more tests now. I tried to a few
> > variations
> > > to
> > > > > >> > pinpoint
> > > > > >> > > the issue, but first:
> > > > > >> > >
> > > > > >> > > @Philippe:
> > > > > >> > >
> > > > > >> > > 0) Response headers are like:
> > > > > >> > >
> > > > > >> > > Response headers:
> > > > > >> > > HTTP/1.1 200 OK
> > > > > >> > > Date: Fri, 03 Feb 2017 06:47:00 GMT
> > > > > >> > > Content-Type: application/json
> > > > > >> > > Content-Length: 374
> > > > > >> > > Connection: keep-alive
> > > > > >> > > Server: nginx/1.6.2
> > > > > >> > > ETag: 37926:1-37927:1
> > > > > >> > > Cache-Control: no-cache, no-store
> > > > > >> > > Pragma: no-cache
> > > > > >> > > Expires: 0
> > > > > >> > > X-Transaction: 0000-326srxia31tss3tc
> > > > > >> >
> > > > > >> >
> > > > > >> > As you pointed it later that issue is related to keep alive.
> > > > > >> > This is tunable , I 'll try to provide some advices
> > > > > >> >
> > > > > >> > But read:
> > > > > >> > http://jmeter.apache.org/usermanual/properties_reference.
> > > > > >> html#httpclient4
> > > > > >> >
> > > > > >> > And pay attention to:
> > > > > >> >
> > > > > >> > httpclient4.idletimeout
> > > > > >> > httpclient4.validate_after_inactivity
> > > > > >> > httpclient4.time_to_live
> > > > > >> >
> > > > > >> > It appears Default values do not suite your case.
> > > > > >> >
> > > > > >> >
> > > > > >> > > 1) Tried your test plan. Unfortunately, it does not make
> > > > difference.
> > > > > >> >
> > > > > >> > it was not made to fix issue only to make it simpler and
> > > indépendant
> > > > > of
> > > > > >> any
> > > > > >> > 3rd party plugin
> > > > > >> >
> > > > > >> > >
> > > > > >> > > 2) I ran Wireshark during a couple of tests. You can find a
> > few
> > > > > >> > screenshots
> > > > > >> > > in a later link. Note that my whole network kinda crashes
> when
> > > > > JMeter
> > > > > >> 3
> > > > > >> > > gets stuck (ping gets "General failure" and some active
> > > > connections
> > > > > >> > crash,
> > > > > >> > > note that this seems a bit random).
> > > > > >> >
> > > > > >> > Thanks
> > > > > >> >
> > > > > >> > >
> > > > > >> > > 3) I gave it a few minutes to recover between the runs.
> > > > > >> > >
> > > > > >> > > @Felix:
> > > > > >> > >
> > > > > >> > > I ran all tests with loggerpanel.usejsyntaxtext=false.
> > > > > >> > >
> > > > > >> > > No, the amount of seems roughly the same.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Then, about pinpointing the issue:
> > > > > >> > >
> > > > > >> > > I managed to reproduce this by installing nginx on a network
> > > > > computer
> > > > > >> > (sudo
> > > > > >> > > apt-get install nginx on debian 8). I did not configure
> > anything
> > > > > and I
> > > > > >> > > simply bombarded the default nginx welcome page.
> > > > > >> > >
> > > > > >> > > I have attached Heap Dump, Thread Dumps and some other
> > > information
> > > > > in
> > > > > >> > these
> > > > > >> > > links:
> > > > > >> > >
> > > > > >> > > JAVA impl (that doesn't crash):
> > > > > >> > > https://drive.google.com/drive/folders/
> > > > 0B1uBdVuLED5NSFJvc2VSdE13a0k
> > > > > ?
> > > > > >> > > usp=sharing
> > > > > >> > >
> > > > > >> > > HttpCilent 4 impl (that crashes):
> > > > > >> > > https://drive.google.com/drive/folders/
> > > > 0B1uBdVuLED5NUGc1WWFYYkRncTQ
> > > > > ?
> > > > > >> > > usp=sharing
> > > > > >> > >
> > > > > >> > > I used Philippe's test plan and even simplified it a bit,
> the
> > > JMX
> > > > > >> file is
> > > > > >> > > in the latter link. Note that the errors occur (in my case)
> > > during
> > > > > the
> > > > > >> > > ramp-up so I also shortened to run just 30s after reaching
> 500
> > > > > users.
> > > > > >> > >
> > > > > >> > > With JAVA impl, I ran heap dump at ~450 users or so, and
> > thread
> > > > > dumps
> > > > > >> > after
> > > > > >> > > that.
> > > > > >> > >
> > > > > >> > > With HttpClient 4 impl, I ran heap dump immediately after
> > seeing
> > > > the
> > > > > >> > first
> > > > > >> > > error (which was at ~430 users or so) and thread dumps right
> > > after
> > > > > >> that
> > > > > >> > > (the first two dumps I think took a bit longer than normally
> > to
> > > > > >> create).
> > > > > >> > > JMeter goes unresponsive a second or two after seeing the
> > first
> > > > > error,
> > > > > >> > so I
> > > > > >> > > just barely had time to click to take the dump.
> > > > > >> > >
> > > > > >> > > As said, there are some screenshots from Wireshark trace for
> > > > > >> HttpClient 4
> > > > > >> > > run. I am not network expert, so I don't dare to judge them,
> > > but I
> > > > > >> > included
> > > > > >> > > what I saw interesting. I think I can share some parts
> > > (privately)
> > > > > >> should
> > > > > >> > > you want to study the trace in detail.
> > > > > >> > >
> > > > > >> > > Finally, when I disabled keep-alive from nginx
> configuration,
> > > the
> > > > > >> problem
> > > > > >> > > was gone (as earlier). So, the problem is somehow related to
> > > > > >> keep-alive.
> > > > > >> > >
> > > > > >> > > Note that in all scenarios CPU goes to 100% quite
> immediately
> > > > (also
> > > > > on
> > > > > >> > 2.13
> > > > > >> > > now, probably because of high throughput, I assume my
> > > workstation
> > > > is
> > > > > >> the
> > > > > >> > > bottleneck here) but without keep-alive or with JAVA impl,
> > > things
> > > > > work
> > > > > >> > and
> > > > > >> > > with keep-alive enabled and HttpClient 4, things break
> apart.
> > As
> > > > > >> before,
> > > > > >> > on
> > > > > >> > > JMeter 2.13 everything works.
> > > > > >> > >
> > > > > >> > > Tuukka
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <
> > > > > >> > > felix.schumacher@internetallee.de <javascript:;>
> > <javascript:;>> wrote:
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
> > > > > >> > > > tuukka.mustonen@gmail.com <javascript:;> <javascript:;>>:
> > > > > >> > > > >@Philippe:
> > > > > >> > > > >
> > > > > >> > > > >0. Alright, I will.
> > > > > >> > > > >1. Will do.
> > > > > >> > > > >2. Sure.
> > > > > >> > > > >3. To let the network connection state restore? Ok, will
> > do.
> > > > > >> > > > >
> > > > > >> > > > >I am having some other stuff I need to attend today, but
> I
> > > will
> > > > > get
> > > > > >> > > > >back to
> > > > > >> > > > >this in the afternoon at latest.
> > > > > >> > > >
> > > > > >> > > > Could you also try to set loggerpanel.usejsyntaxtext to
> > false
> > > in
> > > > > >> your
> > > > > >> > > > settings?
> > > > > >> > > >
> > > > > >> > > > Are there more log entries, when you use httpclient 4
> > instead
> > > of
> > > > > >> Java?
> > > > > >> > > >
> > > > > >> > > > Vladimir has found signs for a event congestion in the awt
> > > event
> > > > > >> queue,
> > > > > >> > > > that show in your threaddumps.
> > > > > >> > > >
> > > > > >> > > > Regards,
> > > > > >> > > >  Felix
> > > > > >> > > >
> > > > > >> > > > >
> > > > > >> > > > >Tuukka
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
> > > > > >> > > > >philippe.mouawad@gmail.com <javascript:;>
> <javascript:;>>
> > wrote:
> > > > > >> > > > >
> > > > > >> > > > >> Hello,
> > > > > >> > > > >> I've created this bugzilla for investigation and to
> share
> > > > with
> > > > > >> you
> > > > > >> > > > >the plan
> > > > > >> > > > >> I have used to run my own tests:
> > > > > >> > > > >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
> > > > > >> > > > >>
> > > > > >> > > > >> I use a Tomcat 8.5.11 with a localhost and a
> certificate
> > > > > >> generated
> > > > > >> > > > >and auto
> > > > > >> > > > >> signed.
> > > > > >> > > > >>
> > > > > >> > > > >> So here is what I get on a Mac Book Pro Mid 2015 OSX El
> > > > Capitan
> > > > > >> 2.5
> > > > > >> > > > >Ghz
> > > > > >> > > > >> Intel Core i7
> > > > > >> > > > >> Java 8 u121
> > > > > >> > > > >> -Xmx512m -Xms512m
> > > > > >> > > > >> Test in GUI mode (ANTI-PATTERN)
> > > > > >> > > > >>
> > > > > >> > > > >> 2.13:
> > > > > >> > > > >> - CPU is at 100%
> > > > > >> > > > >> - No error
> > > > > >> > > > >> - No leak
> > > > > >> > > > >>
> > > > > >> > > > >> 3.2 nightly with default settings :
> > > > > >> > > > >> - CPU is at 100%
> > > > > >> > > > >> - No error
> > > > > >> > > > >> - No leak
> > > > > >> > > > >>
> > > > > >> > > > >> 3.2 nightly with this in user.properties :
> > > > > >> > > > >> httpclient4.time_to_live=0
> > > > > >> > > > >> httpclient4.validate_after_inactivity=0
> > > > > >> > > > >>
> > > > > >> > > > >> - CPU is at 100%
> > > > > >> > > > >> - No error
> > > > > >> > > > >> - No leak
> > > > > >> > > > >>
> > > > > >> > > > >> I attached the Heap Graphs to bugzilla and the plan.
> > > > > >> > > > >>
> > > > > >> > > > >> So for now I don't  reproduce your issue but I am not
> in
> > > the
> > > > > same
> > > > > >> > > > >> configuration as I am not hitting the same website.
> > > > > >> > > > >>
> > > > > >> > > > >> So
> > > > > >> > > > >> 0/ Could you provide by running only 5 requests, the
> > > response
> > > > > >> header
> > > > > >> > > > >that
> > > > > >> > > > >> your website returns ?
> > > > > >> > > > >> 1/ Could you try my test plan by replacing with your
> > target
> > > > > host
> > > > > >> and
> > > > > >> > > > >see if
> > > > > >> > > > >> behaviour is the same ?
> > > > > >> > > > >> 2/ Ensure nothing is running (No Spotify :-) ) except
> > your
> > > > > JMeter
> > > > > >> > and
> > > > > >> > > > >> monitor network as per Felix note
> > > > > >> > > > >> 3/ Run 1 test after the other but ensure you wait a bit
> > > > between
> > > > > >> each
> > > > > >> > > > >one
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > > >> Thanks
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > > >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
> > > > > >> > > > >> felix.schumacher@internetallee.de <javascript:;>
> > <javascript:;>> wrote:
> > > > > >> > > > >>
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka
> > Mustonen <
> > > > > >> > > > >> > tuukka.mustonen@gmail.com <javascript:;>
> > <javascript:;>>:
> > > > > >> > > > >> > >Ok I re-ran the test.
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >Took Heap Dump just before JMeter froze.
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >Took Thread Dumps right after JMeter froze.
> > > > > >> > > > >> > >
> > > > > >> > > > >> >
> > > > > >> > > > >>https://drive.google.com/drive/folders/
> > 0B1uBdVuLED5NNFNqal
> > > > > >> lGam13MTA
> > > > > >> > > > >> >
> > > > > >> > > > >> > That folder seems to be empty now.
> > > > > >> > > > >> >
> > > > > >> > > > >> > Are you running the tests with a security manager
> > > enabled?
> > > > > >> > > > >> >
> > > > > >> > > > >> > Can you share the test or at least the used requests?
> > > > > >> > > > >> >
> > > > > >> > > > >> > Are you using a proxy and if so, how is it
> configured?
> > > > > >> > > > >> >
> > > > > >> > > > >> > Is jmeter running locally, or over something like
> rdp?
> > > > > >> > > > >> >
> > > > > >> > > > >> > Can you look at the used network connections like
> > netstat
> > > > > >> -etapn
> > > > > >> > > > >would do
> > > > > >> > > > >> > on Linux?
> > > > > >> > > > >> >
> > > > > >> > > > >> > Regards,
> > > > > >> > > > >> >  Felix
> > > > > >> > > > >> >
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >...
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >For the fun of it, I also tried upgrading to the
> > latest
> > > > > >> > HttpClient
> > > > > >> > > > >> > >4.5.3
> > > > > >> > > > >> > >version. No effect.
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >I also noticed that if I ping (google) during the
> perf
> > > > > >> testing,
> > > > > >> > > > >Windows
> > > > > >> > > > >> > >ping reports "General failure." when JMeter breaks.
> > > After
> > > > > >> killing
> > > > > >> > > > >it,
> > > > > >> > > > >> > >connectivity restores and there are replies from the
> > > > server
> > > > > >> > again.
> > > > > >> > > > >It
> > > > > >> > > > >> > >seems
> > > > > >> > > > >> > >like JMeter/HttpClient kills my whole network
> > > > connection...
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >Tuukka
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > > > > >> > > > >> > >philippe.mouawad@gmail.com <javascript:;>
> > <javascript:;>> wrote:
> > > > > >> > > > >> > >
> > > > > >> > > > >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> > > > > >> > > > >> > ><tuukka.mustonen@gmail.com <javascript:;>
> > <javascript:;>
> > > > > >> > > > >> > >> >
> > > > > >> > > > >> > >> wrote:
> > > > > >> > > > >> > >>
> > > > > >> > > > >> > >> > Hi, I'm running solely on Win 10 now. In
> > > re-attempts I
> > > > > >> think
> > > > > >> > I
> > > > > >> > > > >saw
> > > > > >> > > > >> > >> slightly
> > > > > >> > > > >> > >> > different behavior in my Linux guest. I'll write
> > > more
> > > > > >> about
> > > > > >> > > > >that
> > > > > >> > > > >> > >later,
> > > > > >> > > > >> > >> but
> > > > > >> > > > >> > >> > for simplicity I'm testing only on Windows host
> > now:
> > > > > >> > > > >> > >> >
> > > > > >> > > > >> > >> > Thread dumps are downloadable at:
> > > > > >> > > > >> > >> >
> > > > > >> > > > >https://drive.google.com/drive/folders/
> > 0B1uBdVuLED5NS1RFbEF
> > > > > >> uSS1iejA
> > > > > >> > > > >> > >> >
> > > > > >> > > > >> > >> > (Took them with 7-8s interval I think, hope it's
> > > ok).
> > > > > >> > > > >> > >> >
> > > > > >> > > > >> > >> > I took the thread dumps once JMeter started
> > filling
> > > in
> > > > > >> > errors,
> > > > > >> > > > >> > >which is
> > > > > >> > > > >> > >> > somewhat after my CPU usage hit 100% (and music
> > > > playback
> > > > > >> > > > >started to
> > > > > >> > > > >> > >> distort
> > > > > >> > > > >> > >> > etc.).
> > > > > >> > > > >> > >> >
> > > > > >> > > > >> > >> > I also created the head dump, note that I took
> it
> > > > after
> > > > > >> > > > >running the
> > > > > >> > > > >> > >test
> > > > > >> > > > >> > >> > and after restarting JMeter, as JMeter froze
> > during
> > > > the
> > > > > >> test
> > > > > >> > > > >and I
> > > > > >> > > > >> > >had to
> > > > > >> > > > >> > >> > kill it. DId you mean I should take it *during*
> > the
> > > > > test?
> > > > > >> > > > >> > >>
> > > > > >> > > > >> > >>
> > > > > >> > > > >> > >> YES.
> > > > > >> > > > >> > >>
> > > > > >> > > > >> > >>
> > > > > >> > > > >> > >> > Anyway, before
> > > > > >> > > > >> > >> > sharing it I want to make sure it can't contain
> > > > anything
> > > > > >> > > > >> > >confidential?
> > > > > >> > > > >> > >> > (parts of my test plan, passwords, environment
> > > > > variables,
> > > > > >> > > > >> > >anything...)
> > > > > >> > > > >> > >> >
> > > > > >> > > > >> > >> > Tuukka
> > > > > >> > > > >> > >> >
> > > > > >> > > > >> > >> >
> > > > > >> > > > >> > >> >
> > > > > >> > > > >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK
> > > > Support
> > > > > <
> > > > > >> > > > >> > >> > support@ubikloadpack.com <javascript:;>
> > <javascript:;>> wrote:
> > > > > >> > > > >> > >> >
> > > > > >> > > > >> > >> > > Hello,
> > > > > >> > > > >> > >> > > To investigate the problem:
> > > > > >> > > > >> > >> > > 1/ to create  thread dumps, install a JDK of
> > same
> > > > > >> version
> > > > > >> > as
> > > > > >> > > > >the
> > > > > >> > > > >> > >one
> > > > > >> > > > >> > >> > > running your JMeter.
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > > Run until you get the problem, then call using
> > > same
> > > > > >> user as
> > > > > >> > > > >the
> > > > > >> > > > >> > >one
> > > > > >> > > > >> > >> > running
> > > > > >> > > > >> > >> > > JMeter 3 times at 5s interval:
> > > > > >> > > > >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >>
> > > > > >> > output_file.txt
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > > It will also help if you provide the structure
> > of
> > > > your
> > > > > >> Test
> > > > > >> > > > >Plan.
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > > 2/ It would also help if you can provide a
> Heap
> > > > Dump.
> > > > > >> With
> > > > > >> > > > >JMeter
> > > > > >> > > > >> > >GUI,
> > > > > >> > > > >> > >> > use
> > > > > >> > > > >> > >> > > menu Help > Create a Heap Dump
> > > > > >> > > > >> > >> > > This will create a Heap Dump (rather big
> file) ,
> > > it
> > > > > >> would
> > > > > >> > > > >help a
> > > > > >> > > > >> > >lot if
> > > > > >> > > > >> > >> > you
> > > > > >> > > > >> > >> > > can make it available.
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > > Regards
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka
> Mustonen
> > <
> > > > > >> > > > >> > >> > tuukka.mustonen@gmail.com <javascript:;>
> > <javascript:;>
> > > > > >> > > > >> > >> > > >
> > > > > >> > > > >> > >> > > wrote:
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> > > > @Philippe:
> > > > > >> > > > >> > >> > > >
> > > > > >> > > > >> > >> > > > Yeah, I am normally running in non-GUI mode,
> > but
> > > > to
> > > > > >> > > > >> > >> > > > study/inspect/understand this problem I am
> > > running
> > > > > in
> > > > > >> GUI
> > > > > >> > > > >mode.
> > > > > >> > > > >> > >> JMeter
> > > > > >> > > > >> > >> > > 2.13
> > > > > >> > > > >> > >> > > > works just fine in GUI mode for these tests
> so
> > > > > JMeter
> > > > > >> 3.x
> > > > > >> > > > >> > >should too.
> > > > > >> > > > >> > >> > > >
> > > > > >> > > > >> > >> > > > I'll give the nightly build a try.
> > > > > >> > > > >> > >> > > >
> > > > > >> > > > >> > >> > > > Tuukka
> > > > > >> > > > >> > >> > > >
> > > > > >> > > > >> > >> > > >
> > > > > >> > > > >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe
> > > Mouawad <
> > > > > >> > > > >> > >> > > > philippe.mouawad@gmail.com <javascript:;>
> > <javascript:;>
> > > > > >> > > > >> > >> > > > > wrote:
> > > > > >> > > > >> > >> > > >
> > > > > >> > > > >> > >> > > > > On Wednesday, February 1, 2017, Tuukka
> > > Mustonen
> > > > <
> > > > > >> > > > >> > >> > > > tuukka.mustonen@gmail.com <javascript:;>
> > <javascript:;>
> > > > > >> > > > >> > >> > > > > >
> > > > > >> > > > >> > >> > > > > wrote:
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > > > @Philippe:
> > > > > >> > > > >> > >> > > > > >
> > > > > >> > > > >> > >> > > > > > 1. Yeah, I compared them with the same
> > plan,
> > > > > same
> > > > > >> > > > >machine.
> > > > > >> > > > >> > >Should
> > > > > >> > > > >> > >> > be
> > > > > >> > > > >> > >> > > > > > everything equal.
> > > > > >> > > > >> > >> > > > > > 2. I tried both the defaults in
> jmeter.bat
> > > > > >> (-Xms512m
> > > > > >> > > > >> > >-Xmx512m)
> > > > > >> > > > >> > >> and
> > > > > >> > > > >> > >> > as
> > > > > >> > > > >> > >> > > > > > slightly increased (-Xms512m
> -Xmx1024m). I
> > > > > didn't
> > > > > >> > > > >tweak
> > > > > >> > > > >> > >them much
> > > > > >> > > > >> > >> > > > > because I
> > > > > >> > > > >> > >> > > > > > don't believe the issue lies there.
> > > > > >> > > > >> > >> > > > > > 3. No, I do just simple HTTP GETs
> (against
> > > > RESTy
> > > > > >> > API).
> > > > > >> > > > >> > >> > > > > > 4. I am not familiar with that but I can
> > > try.
> > > > If
> > > > > >> you
> > > > > >> > > > >have
> > > > > >> > > > >> > >> > > instructions
> > > > > >> > > > >> > >> > > > to
> > > > > >> > > > >> > >> > > > > > point to, please do.
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > > > 5. I am running with GUI.
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > > This is a known anti-pattern .
> > > > > >> > > > >> > >> > > > > switch to non gui mode if you want correct
> > > > > results.
> > > > > >> > > > >> > >> > > > > http://jmeter.apache.org/
> > > > > >> > usermanual/best-practices.html
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > > http://www.ubik-ingenierie.
> > > > > >> > com/blog/jmeter_performance_
> > > > > >> > > > >> > >> tuning_tips/
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > > > About file sending:
> > > > > >> > > > >> > >> > > > > >
> > > > > >> > > > >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > >  Could you give nightly build a try ?
> > > > > >> > > > >> > >> > > > > http://jmeter.apache.org/nightly.html
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > > > Tuukka
> > > > > >> > > > >> > >> > > > > >
> > > > > >> > > > >> > >> > > > > >
> > > > > >> > > > >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe
> > > > > Mouawad <
> > > > > >> > > > >> > >> > > > > > philippe.mouawad@gmail.com
> <javascript:;>
> > <javascript:;>
> > > > > >> > > <javascript:;>
> > > > > >> > > > >> > >> > > > > > > wrote:
> > > > > >> > > > >> > >> > > > > >
> > > > > >> > > > >> > >> > > > > > > Also are you sending files in post
> body
> > ?
> > > > > >> > > > >> > >> > > > > > >
> > > > > >> > > > >> > >> > > > > > > Thank you
> > > > > >> > > > >> > >> > > > > > >
> > > > > >> > > > >> > >> > > > > > > On Wednesday, February 1, 2017,
> Philippe
> > > > > >> Mouawad <
> > > > > >> > > > >> > >> > > > > > > philippe.mouawad@gmail.com
> > <javascript:;> <javascript:;>
> > > > > >> > > <javascript:;>>
> > > > > >> > > > >> > >> > > > > > > wrote:
> > > > > >> > > > >> > >> > > > > > >
> > > > > >> > > > >> > >> > > > > > > > Hello,
> > > > > >> > > > >> > >> > > > > > > > Few questions, please try to answer
> > them
> > > > > all:
> > > > > >> > > > >> > >> > > > > > > >
> > > > > >> > > > >> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with
> > > same
> > > > > >> > > > >plan/sape
> > > > > >> > > > >> > >> machine...
> > > > > >> > > > >> > >> > > > > > everything
> > > > > >> > > > >> > >> > > > > > > > equal ?
> > > > > >> > > > >> > >> > > > > > > > 2/ What are your Xmx settings
> > > > > >> > > > >> > >> > > > > > > > 3/ does your plan download embedded
> > > > > resources
> > > > > >> ?
> > > > > >> > > > >> > >> > > > > > > > 4/ when issue happens can you make 3
> > > > thread
> > > > > >> dumps
> > > > > >> > > > >at 5
> > > > > >> > > > >> > >> seconds
> > > > > >> > > > >> > >> > > > > interval
> > > > > >> > > > >> > >> > > > > > > > and send file ? (use jstack)
> > > > > >> > > > >> > >> > > > > > > > 5/ are you running gui or non gui ?
> > > > > >> > > > >> > >> > > > > > > >
> > > > > >> > > > >> > >> > > > > > > > Thanks
> > > > > >> > > > >> > >> > > > > > > > Regards
> > > > > >> > > > >> > >> > > > > > > >
> > > > > >> > > > >> > >> > > > > > > > On Wednesday, February 1, 2017,
> Tuukka
> > > > > >> Mustonen <
> > > > > >> > > > >> > >> > > > > > > tuukka.mustonen@gmail.com
> > <javascript:;> <javascript:;>
> > > > > >> > > <javascript:;>
> > > > > >> > > > >> > >> > > > > > > >
> > > > > >> > > > ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > <javascript:;>
> > > > > >> > <javascript:;>
> > > > > >> > > > >> > >> > > > > <javascript:;>');>>
> > > > > >> > > > >> > >> > > > > > wrote:
> > > > > >> > > > >> > >> > > > > > > >
> > > > > >> > > > >> > >> > > > > > > >> Hi,
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> After upgrading to JMeter 3.0 or
> 3.1
> > I
> > > > > >> > experience
> > > > > >> > > > >high
> > > > > >> > > > >> > >CPU
> > > > > >> > > > >> > >> > usage
> > > > > >> > > > >> > >> > > > > that
> > > > > >> > > > >> > >> > > > > > > >> renders JMeter unusable. CPU usage
> > > ramps
> > > > > >> quickly
> > > > > >> > > > >up
> > > > > >> > > > >> > >and
> > > > > >> > > > >> > >> after
> > > > > >> > > > >> > >> > > > having
> > > > > >> > > > >> > >> > > > > > > >> ~100-200 concurrent users or so (in
> > ~30
> > > > > >> > seconds),
> > > > > >> > > > >it
> > > > > >> > > > >> > >hits
> > > > > >> > > > >> > >> 100%
> > > > > >> > > > >> > >> > > on
> > > > > >> > > > >> > >> > > > my
> > > > > >> > > > >> > >> > > > > > > >> i7-4600. Shortly after, whole
> JMeter
> > > > > freezes
> > > > > >> and
> > > > > >> > > > >I
> > > > > >> > > > >> > >have to
> > > > > >> > > > >> > >> > kill
> > > > > >> > > > >> > >> > > > it.
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> On 2.13 everything goes smoothly
> > (with
> > > > > >> default
> > > > > >> > > > >> > >settings).
> > > > > >> > > > >> > >> CPU
> > > > > >> > > > >> > >> > > > usage
> > > > > >> > > > >> > >> > > > > > > stays
> > > > > >> > > > >> > >> > > > > > > >> at ~30% even with 500 users.
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> The experience is similar on both
> > > Windows
> > > > > 10
> > > > > >> and
> > > > > >> > > > >Linux
> > > > > >> > > > >> > >> (Ubuntu
> > > > > >> > > > >> > >> > > > 14.04
> > > > > >> > > > >> > >> > > > > > LTS
> > > > > >> > > > >> > >> > > > > > > >> guest in Virtualbox).
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> The problems I saw in JMeter 3:
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> - java.net.SocketException:
> > Permission
> > > > > >> denied:
> > > > > >> > > > >connect
> > > > > >> > > > >> > >> > > > > > > >> - After a while it runs out of heap
> > > space
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> For "Permission denied: connect" I
> > > tried
> > > > to
> > > > > >> set
> > > > > >> > > > >> > >> > > > > > > >> java.net.preferIPv4Stack=true
> > > > > >> > > > >> > >> > > > > > > >> (in a few places to make sure it
> > would
> > > > > >> really be
> > > > > >> > > > >in
> > > > > >> > > > >> > >use) and
> > > > > >> > > > >> > >> > > tried
> > > > > >> > > > >> > >> > > > > to
> > > > > >> > > > >> > >> > > > > > > run
> > > > > >> > > > >> > >> > > > > > > >> JMeter as administrator (in Win).
> No
> > > > > effect.
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> I also tried increasing heap size,
> > but
> > > > that
> > > > > >> > > > >didn't
> > > > > >> > > > >> > >help
> > > > > >> > > > >> > >> > either.
> > > > > >> > > > >> > >> > > > And
> > > > > >> > > > >> > >> > > > > I
> > > > > >> > > > >> > >> > > > > > > >> think
> > > > > >> > > > >> > >> > > > > > > >> memory size is not the root cause
> > here
> > > -
> > > > > >> rather
> > > > > >> > > > >the
> > > > > >> > > > >> > >result.
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> I am normally connected to VPN but
> I
> > > also
> > > > > >> tried
> > > > > >> > > > >> > >disabling
> > > > > >> > > > >> > >> it.
> > > > > >> > > > >> > >> > No
> > > > > >> > > > >> > >> > > > > > effect.
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> I tried disabling with and without
> > IPv6
> > > > > >> enabled
> > > > > >> > > > >in
> > > > > >> > > > >> > >Windows
> > > > > >> > > > >> > >> > > network
> > > > > >> > > > >> > >> > > > > > > >> connection settings. No effect.
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> I am running a few extra JMeter
> > > plugins.
> > > > > >> Tried
> > > > > >> > > > >cutting
> > > > > >> > > > >> > >my
> > > > > >> > > > >> > >> test
> > > > > >> > > > >> > >> > > > > > scenario
> > > > > >> > > > >> > >> > > > > > > to
> > > > > >> > > > >> > >> > > > > > > >> minimum. No effect.
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> I am running anti-virus software
> > that I
> > > > > >> cannot
> > > > > >> > > > >> > >disable, due
> > > > > >> > > > >> > >> to
> > > > > >> > > > >> > >> > > > this
> > > > > >> > > > >> > >> > > > > > > being
> > > > > >> > > > >> > >> > > > > > > >> forced by IT department. However,
> > it's
> > > > not
> > > > > a
> > > > > >> > > > >problem
> > > > > >> > > > >> > >with
> > > > > >> > > > >> > >> 2.13
> > > > > >> > > > >> > >> > > so
> > > > > >> > > > >> > >> > > > I
> > > > > >> > > > >> > >> > > > > > > assume
> > > > > >> > > > >> > >> > > > > > > >> it shouldn't be a problem here
> > either.
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> FINALLY, once I set HTTP request
> > > > > >> implementation
> > > > > >> > > > >from
> > > > > >> > > > >> > >Apache
> > > > > >> > > > >> > >> > > > > > HttpClient 4
> > > > > >> > > > >> > >> > > > > > > >> to
> > > > > >> > > > >> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem
> > > > > vanishes
> > > > > >> -
> > > > > >> > > > >CPU
> > > > > >> > > > >> > >stays
> > > > > >> > > > >> > >> where
> > > > > >> > > > >> > >> > > it
> > > > > >> > > > >> > >> > > > > did
> > > > > >> > > > >> > >> > > > > > > with
> > > > > >> > > > >> > >> > > > > > > >> JMeter 2.13.
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> I'm running Oracle JVM 8 (update
> 121
> > on
> > > > > Win,
> > > > > >> > > > >update
> > > > > >> > > > >> > >111 on
> > > > > >> > > > >> > >> > > Linux).
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> Why does (the recommended)
> > HttpClient 4
> > > > > cause
> > > > > >> > > > >problems
> > > > > >> > > > >> > >here?
> > > > > >> > > > >> > >> > Is
> > > > > >> > > > >> > >> > > > > this a
> > > > > >> > > > >> > >> > > > > > > >> know
> > > > > >> > > > >> > >> > > > > > > >> bug/issue? JMeter 3.x is running
> > newer
> > > > > >> version
> > > > > >> > > > >> > >compared to
> > > > > >> > > > >> > >> > 2.13,
> > > > > >> > > > >> > >> > > > but
> > > > > >> > > > >> > >> > > > > > > what
> > > > > >> > > > >> > >> > > > > > > >> change in HttpClient 4 might cause
> > the
> > > > > >> problems
> > > > > >> > > > >> > >described?
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> Any ideas?
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >> Regards,
> > > > > >> > > > >> > >> > > > > > > >> Tuukka
> > > > > >> > > > >> > >> > > > > > > >>
> > > > > >> > > > >> > >> > > > > > > >
> > > > > >> > > > >> > >> > > > > > > >
> > > > > >> > > > >> > >> > > > > > > > --
> > > > > >> > > > >> > >> > > > > > > > Cordialement.
> > > > > >> > > > >> > >> > > > > > > > Philippe Mouawad.
> > > > > >> > > > >> > >> > > > > > > >
> > > > > >> > > > >> > >> > > > > > > >
> > > > > >> > > > >> > >> > > > > > > >
> > > > > >> > > > >> > >> > > > > > > >
> > > > > >> > > > >> > >> > > > > > >
> > > > > >> > > > >> > >> > > > > > > --
> > > > > >> > > > >> > >> > > > > > > Cordialement.
> > > > > >> > > > >> > >> > > > > > > Philippe Mouawad.
> > > > > >> > > > >> > >> > > > > > >
> > > > > >> > > > >> > >> > > > > >
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > > > --
> > > > > >> > > > >> > >> > > > > Cordialement.
> > > > > >> > > > >> > >> > > > > Philippe Mouawad.
> > > > > >> > > > >> > >> > > > >
> > > > > >> > > > >> > >> > > >
> > > > > >> > > > >> > >> > >
> > > > > >> > > > >> > >> >
> > > > > >> > > > >> > >>
> > > > > >> > > > >> > >>
> > > > > >> > > > >> > >>
> > > > > >> > > > >> > >> --
> > > > > >> > > > >> > >> Cordialement.
> > > > > >> > > > >> > >> Philippe Mouawad.
> > > > > >> > > > >> > >>
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >-----------------------------
> > ------------------------------
> > > > > >> ----------
> > > > > >> > > > >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.
> > > apache.org
> > > > > >> > > <javascript:;>
> > > > > >> > > > >> > For additional commands, e-mail:
> > > > user-help@jmeter.apache.org <javascript:;>
> > > > > >> > > <javascript:;>
> > > > > >> > > > >> >
> > > > > >> > > > >> >
> > > > > >> > > > >>
> > > > > >> > > > >>
> > > > > >> > > > >> --
> > > > > >> > > > >> Cordialement.
> > > > > >> > > > >> Philippe Mouawad.
> > > > > >> > > > >>
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Cordialement.
> > > > > >> > Philippe Mouawad.
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Cordialement.
> > > > > > Philippe Mouawad.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cordialement.
> > > > > Philippe Mouawad.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cordialement.
> > > Philippe Mouawad.
> > >
> >
>
>
> --
> Cordialement.
> Philippe Mouawad.
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

On Monday, February 6, 2017, Tuukka Mustonen <tu...@gmail.com>
wrote:

> > https://tools.ietf.org/id/draft-thomson-hybi-http-
> timeout-01.html#rfc.section.2.1
> > But server is allowed not to send it.
> > But I think for a load test, it is better to adjust the 3 parameters to a
> > value that is similar to the server keep-alive timeout
>
> Cool, never seen anyone mention/recommend that header :O
>
> I actually tried setting TTL to 60 seconds and while I think it worked, the
> result was that in the end of the test, some threads would never close
> their connection. Tuning TTL down to 10 seconds fixed this. Not sure what
> was going on, but yeah, that's a potential separate issue - no need to
> prolong this thread any longer.


Could you open a bugzilla and attach thread dumps when you reproduce this
issue please ?

>
> > I mean if server does not send timeout, but timeouts connection on its
> > side, JMeter might
> > try to reuse a "stale" connection which has a cost and will only drop it
> on
> > next use.
> > For high throughput tests it migh be a problem.
>
> Alright.
>
> > https://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx
>
> Interesting. I googled around a bit and it would appear that with default
> Windows settings one shouldn't hit into it (ephemeral port exhaustion) very
> easily (so I guess your 2s TTL is ok, although it feels overly short). But
> yeah, maybe we have special configuration or something...
>
> Thanks again!
>
> Tuukka
>
>
>
> On Sat, Feb 4, 2017 at 10:22 PM, Philippe Mouawad <
> philippe.mouawad@gmail.com <javascript:;>> wrote:
>
> > On Sat, Feb 4, 2017 at 8:05 PM, Tuukka Mustonen <
> tuukka.mustonen@gmail.com <javascript:;>
> > >
> > wrote:
> >
> > > > In the case of NGinx , it seems no header timeout is sent.
> > > > So if you set a value > to 0, you can control the max idle duration
> of
> > a
> > > > connection from Client (JMeter) side to avoid storing too many
> > > connections
> > > > that won't be reused and can impact JMeter heap.
> > > > By default it's not set so no impact.
> > >
> > > Hmm, what do you mean by "header timeout"? Afaik, the server cannot
> > > communicate a timeout for the keep-alived connection to the client. The
> > > server probably has some timeout (I think in nginx it's 75s by default)
> > but
> > > the client doesn't know it. Server will just close the connection at
> some
> > > point.
> > >
> > https://tools.ietf.org/id/draft-thomson-hybi-http-
> > timeout-01.html#rfc.section.2.1
> > But server is allowed not to send it.
> > But I think for a load test, it is better to adjust the 3 parameters to a
> > value that is similar to the server keep-alive timeout
> >
> >
> >
> > > Also I'm not 100% sure what you mean by "to avoid storing too many
> > > connections that won't be reused"? The client will re-use connection
> > until
> > > it's closed by TTL configuration or if the server closes the
> connection,
> > > right?
> >
> >
> > I mean if server does not send timeout, but timeouts connection on its
> > side, JMeter might
> > try to reuse a "stale" connection which has a cost and will only drop it
> on
> > next use.
> > For high throughput tests it migh be a problem.
> >
> >
> >
> > > I though with idletimeout I could just make sure that client closes
> > > the connection before the server would (and avoid any potential failed
> > > requests if client sent a request just while the server closes
> > connection)?
> > >
> > You understood correctly
> >
> > >
> > > I'm not sure, the documentation for that parameter could maybe be
> > improved
> > > a tiny bit?
> > >
> >
> > If you want to contribute a patch for that you're welcome. It's in
> > properties_reference.xml.
> > http://jmeter.apache.org/building.html
> > Otherwise we'll try to fix it.
> >
> > >
> > > > But investigating your issue, I discovered that in fact
> > > > "httpclient4.validate_after_inactivity" has no impact due to another
> > bug
> > > > here I am also responsible of :-( .
> > > > validate() method always return true so JMeter always consider
> > connection
> > > > as valid.
> > > > So only "httpclient4.time_to_live" does its job here.
> > >
> > > You seem to be author of lots of commits - it's only human to forget
> > > something every now and then :)
> > >
> > > > What's happening here is that you're most probably saturating the
> > dynamic
> > > > port range
> > >
> > > If you mean that I would be going over the number of available dynamic
> > > ports, I don't think that is it. The test breaks at ~40-50 seconds or
> so
> > > and at that point there shouldn't be even 10k of connections
> > (potentially)
> > > open. But, I'm definitively hitting some limit. I will contact our IT
> > > department - who knows what magic configuration we have.
> > >
> >
> > https://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx
> >
> > >
> > > > It's because TTL is > 0, so it will be included in the management of
> > idle
> > > > connections.
> > > > ...
> > > > Because JMeter will close connection immediately and inform server to
> > do
> > > > the same.
> > >
> > > Ah, good point. What's the policy (=timeline) for closing the idle
> > > connections that go over TTL?
> > >
> > It will be closed when a new request is made by same user and it has
> > expired.
> >
> > >
> > > > Thanks a lot for all your tests which helped us narrow this issue.
> > > > Besides Vladimir and Felix are working on improving performances in
> GUI
> > > > Mode but the team still advises against using it for Load Testing.
> > >
> > > Cool. Yeah, I run tests normally from Jenkins in command line mode,
> but I
> > > always develop/debug in GUI (as does everybody, I would believe). When
> in
> > > GUI mode, I don't care about performance really - just that the tests
> > pass.
> > > But better performance is welcome of course.
> > >
> > > > If you used the nightly build (select Jenkins one) you should be able
> > to
> > > > see that now :
> > > > httpclient4.validate_after_inactivity = 1700
> > > > and has an effect.
> > >
> > > Thanks for the quick fixes! I think I will test it next week some
> point,
> > > just for the curiosity, to see it in effect in Wireshark. I assume this
> > > check sends some dummy package to the server and should be visible in
> the
> > > trace. Correct me if wrong.
> > >
> > > Thanks. Your feedback is very welcome.
> >
> > Tuukka
> > >
> > >
> > >
> > > On Sat, Feb 4, 2017 at 5:13 PM, Philippe Mouawad <
> > > philippe.mouawad@gmail.com <javascript:;>
> > > > wrote:
> > >
> > > > Hello Tukka,
> > > > I have commited :
> > > > https://bz.apache.org/bugzilla/show_bug.cgi?id=60690
> > > > https://bz.apache.org/bugzilla/show_bug.cgi?id=60689
> > > >
> > > > If you used the nightly build (select Jenkins one) you should be able
> > to
> > > > see that now :
> > > > httpclient4.validate_after_inactivity = 1700
> > > > and has an effect.
> > > >
> > > > Still in your particular case as you've tested you'll need to
> increase
> > > > httpclient4.time_to_live to avoid too much connections creations that
> > > your
> > > > OS/Machine/Network does not correctly handle.
> > > >
> > > > But beware that this may not be realistic as if you're server calls
> are
> > > > made from clients that do not use keep alive or persist their
> > connections
> > > > for a short time.
> > > > To ensure your server correctly handles this , you should test from a
> > > fine
> > > > tuned machine that has a correct network with target server.
> > > >
> > > >
> > > > Regards
> > > > @philmdot
> > > >
> > > > On Sat, Feb 4, 2017 at 3:16 PM, Philippe Mouawad <
> > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > On Sat, Feb 4, 2017 at 12:58 PM, Tuukka Mustonen <
> > > > > tuukka.mustonen@gmail.com <javascript:;>> wrote:
> > > > >
> > > > >> On point, Philippe!
> > > > >>
> > > > >> I think *httpclient4.idletimeout* is not of relevance here - my
> > server
> > > > >> sends proper Keep-Alive headers so this option shouldn't have
> > impact.
> > > > Good
> > > > >> to know, for sure.
> > > > >>
> > > > > In the case of NGinx , it seems no header timeout is sent.
> > > > > So if you set a value > to 0, you can control the max idle duration
> > of
> > > a
> > > > > connection from Client (JMeter) side to avoid storing too many
> > > > connections
> > > > > that won't be reused and can impact JMeter heap.
> > > > > By default it's not set so no impact.
> > > > >
> > > > >>
> > > > >> *httpclient4.validate_after_inactivity* and
> > *httpclient4.time_to_live
> > > > >> default* values (2 seconds) sound conflicting: TTL being the same
> as
> > > > >> validate_after_inactivity leads to connections never getting
> checked
> > > as
> > > > >> connections are discarded after 2 seconds anyway? So
> > > > >> validate_after_inactivity is not really in use by default if I
> > > > understood
> > > > >> it right.
> > > > >>
> > > > > Good catch. It somehow went out of my radar, I remember it was
> > > something
> > > > I
> > > > > had in mind but it seems I failed to change it.
> > > > > It's a bug or at least bad default value.
> > > > >
> > > > > But investigating your issue, I discovered that in fact
> > > > > "httpclient4.validate_after_inactivity" has no impact due to
> another
> > > bug
> > > > > here I am also responsible of :-( .
> > > > > validate() method always return true so JMeter always consider
> > > connection
> > > > > as valid.
> > > > > So only "httpclient4.time_to_live" does its job here.
> > > > >
> > > > >
> > > > >>
> > > > >> So I went on and tried these settings:
> > > > >>
> > > > >> 1) httpclient4.validate_after_inactivity=0 +
> > > > >> httpclient4.time_to_live=2000
> > > > >> -> CRASH
> > > > >> 2) httpclient4.validate_after_inactivity=0 +
> > > > >> httpclient4.time_to_live=1000
> > > > >> -> CRASH
> > > > >> 3) httpclient4.validate_after_inactivity=1 +
> > > > >> httpclient4.time_to_live=2000
> > > > >> -> CRASH
> > > > >>
> > > > >> 4) httpclient4.validate_after_inactivity=5000 +
> > > > >> httpclient4.time_to_live=10000 -> SUCCESS
> > > > >> 5) httpclient4.validate_after_inactivity=0 +
> > > > >> httpclient4.time_to_live=10000
> > > > >> -> SUCCESS
> > > > >> 6) httpclient4.validate_after_inactivity=1 +
> > > > >> httpclient4.time_to_live=10000
> > > > >> -> SUCCESS
> > > > >> 7) httpclient4.validate_after_inactivity=0 +
> > > httpclient4.time_to_live=0
> > > > >> ->
> > > > >> SUCCESS
> > > > >> 8) httpclient4.validate_after_inactivity=10000 +
> > > > >> httpclient4.time_to_live=10000 -> SUCCESS
> > > > >>
> > > > >> Clearly increasing TTL makes the difference for me.
> > > > >> validate_after_inactivity seems to have no effect. Although there
> is
> > > > some
> > > > >> packet loss, the connections themselves don't break.
> > > > >>
> > > > > Absolutely.
> > > > >
> > > > >>
> > > > >> A) So... maybe my network chip configuration just cannot tolerate
> > > JMeter
> > > > >> opening fresh connections every 2 seconds? That translates to 250
> > > > >> connections/s with 500 users- maybe it's enough to bloat my
> network
> > IO
> > > > and
> > > > >> hit some policy (and I'll then get "connect: Permission denied")?
> > > > Sounds a
> > > > >> bit weird.
> > > > >>
> > > > >
> > > > > It's also your OS, you need to tune the TCP stack to work with port
> > > reuse
> > > > > speed.
> > > > > What's happening here is that you're most probably saturating the
> > > dynamic
> > > > > port range
> > > > >
> > > > >>
> > > > >> So I also tried:
> > > > >>
> > > > >> 9) httpclient4.validate_after_inactivity=0 +
> > > httpclient4.time_to_live=1
> > > > >> ->
> > > > >> CRASH
> > > > >>
> > > > >> And it crashes quite instantly. I guess it's because JMeter is
> > > opening a
> > > > >> new connection per request...
> > > > >>
> > > > > It's because TTL is > 0, so it will be included in the management
> of
> > > idle
> > > > > connections.
> > > > >
> > > > >>
> > > > >> B) ...but if I keep the default 2s settings and just disable
> > > Keep-Alive,
> > > > >> then JMeter is opening new connection per request also, and it
> > works.
> > > > Why
> > > > >> the difference?
> > > > >>
> > > > > Because JMeter will close connection immediately and inform server
> to
> > > do
> > > > > the same.
> > > > >
> > > > >
> > > > >>
> > > > >> C) Then, I wonder if your "standard workstation" is supposed to
> > > tolerate
> > > > >> these defaults? Can my network chip/driver/configuration just be
> so
> > > bad?
> > > > >>
> > > > >
> > > > > Yes at least your OS Tuning defaults are not well suited for Load
> > > > Testing,
> > > > > you need to tune.
> > > > >
> > > > >>
> > > > >> I see the *httpclient4.validate_after_inactivity* and
> > > > >> *httpclient4.time_to_live
> > > > >> default* properties mentioned in the JMeter 3 changelog:
> > > > >>
> > > > >> "Improved and better performing validation mechanism for Stale
> > > > connections
> > > > >> and Keep-Alive management, see properties
> > > > >> httpclient4.validate_after_inactivity and
> httpclient4.time_to_live"
> > > > >>
> > > > >
> > > > > Yes, it was the intention.
> > > > >
> > > > >>
> > > > >> And when I monitored TCP connection in Wireshark, JMeter 2.13
> indeed
> > > > keeps
> > > > >> connections open waaaay longer (seemed a bit random).
> > > > >>
> > > > >
> > > > > Yes, at that time the only solution was to play with idletimeout or
> > set
> > > > > "http.connection.stalecheck$Boolean=true" which had important
> impact
> > > at
> > > > > high throughput.
> > > > >
> > > > >>
> > > > >> D) So I the new 2s default sounds a bit risky?
> > > > >>
> > > > > In your case yes, but you're in a "bad load testing config" :-)
> > > > >
> > > > >>
> > > > >> In the end, increasing TTL seems to fix the issues for me and I
> can
> > > > >> proceed
> > > > >> upgrading to JMeter 3.
> > > > >>
> > > > >> Would love to hear your opinion/analysis on the open questions
> > (A-D),
> > > > but
> > > > >> big thanks for your interest in my problem and the precious help
> > > already
> > > > >> at
> > > > >> this point!
> > > > >>
> > > > >
> > > > > Thanks a lot for all your tests which helped us narrow this issue.
> > > > > Besides Vladimir and Felix are working on improving performances in
> > GUI
> > > > > Mode but the team still advises against using it for Load Testing.
> > > > >
> > > > >
> > > > >> Tuukka
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Fri, Feb 3, 2017 at 9:40 PM, Philippe Mouawad <
> > > > >> philippe.mouawad@gmail.com <javascript:;>
> > > > >> > wrote:
> > > > >>
> > > > >> > On Friday, February 3, 2017, Tuukka Mustonen <
> > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > >> >
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Sorry for the delay, got too busy yesterday.
> > > > >> > >
> > > > >> > > I have been running more tests now. I tried to a few
> variations
> > to
> > > > >> > pinpoint
> > > > >> > > the issue, but first:
> > > > >> > >
> > > > >> > > @Philippe:
> > > > >> > >
> > > > >> > > 0) Response headers are like:
> > > > >> > >
> > > > >> > > Response headers:
> > > > >> > > HTTP/1.1 200 OK
> > > > >> > > Date: Fri, 03 Feb 2017 06:47:00 GMT
> > > > >> > > Content-Type: application/json
> > > > >> > > Content-Length: 374
> > > > >> > > Connection: keep-alive
> > > > >> > > Server: nginx/1.6.2
> > > > >> > > ETag: 37926:1-37927:1
> > > > >> > > Cache-Control: no-cache, no-store
> > > > >> > > Pragma: no-cache
> > > > >> > > Expires: 0
> > > > >> > > X-Transaction: 0000-326srxia31tss3tc
> > > > >> >
> > > > >> >
> > > > >> > As you pointed it later that issue is related to keep alive.
> > > > >> > This is tunable , I 'll try to provide some advices
> > > > >> >
> > > > >> > But read:
> > > > >> > http://jmeter.apache.org/usermanual/properties_reference.
> > > > >> html#httpclient4
> > > > >> >
> > > > >> > And pay attention to:
> > > > >> >
> > > > >> > httpclient4.idletimeout
> > > > >> > httpclient4.validate_after_inactivity
> > > > >> > httpclient4.time_to_live
> > > > >> >
> > > > >> > It appears Default values do not suite your case.
> > > > >> >
> > > > >> >
> > > > >> > > 1) Tried your test plan. Unfortunately, it does not make
> > > difference.
> > > > >> >
> > > > >> > it was not made to fix issue only to make it simpler and
> > indépendant
> > > > of
> > > > >> any
> > > > >> > 3rd party plugin
> > > > >> >
> > > > >> > >
> > > > >> > > 2) I ran Wireshark during a couple of tests. You can find a
> few
> > > > >> > screenshots
> > > > >> > > in a later link. Note that my whole network kinda crashes when
> > > > JMeter
> > > > >> 3
> > > > >> > > gets stuck (ping gets "General failure" and some active
> > > connections
> > > > >> > crash,
> > > > >> > > note that this seems a bit random).
> > > > >> >
> > > > >> > Thanks
> > > > >> >
> > > > >> > >
> > > > >> > > 3) I gave it a few minutes to recover between the runs.
> > > > >> > >
> > > > >> > > @Felix:
> > > > >> > >
> > > > >> > > I ran all tests with loggerpanel.usejsyntaxtext=false.
> > > > >> > >
> > > > >> > > No, the amount of seems roughly the same.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > Then, about pinpointing the issue:
> > > > >> > >
> > > > >> > > I managed to reproduce this by installing nginx on a network
> > > > computer
> > > > >> > (sudo
> > > > >> > > apt-get install nginx on debian 8). I did not configure
> anything
> > > > and I
> > > > >> > > simply bombarded the default nginx welcome page.
> > > > >> > >
> > > > >> > > I have attached Heap Dump, Thread Dumps and some other
> > information
> > > > in
> > > > >> > these
> > > > >> > > links:
> > > > >> > >
> > > > >> > > JAVA impl (that doesn't crash):
> > > > >> > > https://drive.google.com/drive/folders/
> > > 0B1uBdVuLED5NSFJvc2VSdE13a0k
> > > > ?
> > > > >> > > usp=sharing
> > > > >> > >
> > > > >> > > HttpCilent 4 impl (that crashes):
> > > > >> > > https://drive.google.com/drive/folders/
> > > 0B1uBdVuLED5NUGc1WWFYYkRncTQ
> > > > ?
> > > > >> > > usp=sharing
> > > > >> > >
> > > > >> > > I used Philippe's test plan and even simplified it a bit, the
> > JMX
> > > > >> file is
> > > > >> > > in the latter link. Note that the errors occur (in my case)
> > during
> > > > the
> > > > >> > > ramp-up so I also shortened to run just 30s after reaching 500
> > > > users.
> > > > >> > >
> > > > >> > > With JAVA impl, I ran heap dump at ~450 users or so, and
> thread
> > > > dumps
> > > > >> > after
> > > > >> > > that.
> > > > >> > >
> > > > >> > > With HttpClient 4 impl, I ran heap dump immediately after
> seeing
> > > the
> > > > >> > first
> > > > >> > > error (which was at ~430 users or so) and thread dumps right
> > after
> > > > >> that
> > > > >> > > (the first two dumps I think took a bit longer than normally
> to
> > > > >> create).
> > > > >> > > JMeter goes unresponsive a second or two after seeing the
> first
> > > > error,
> > > > >> > so I
> > > > >> > > just barely had time to click to take the dump.
> > > > >> > >
> > > > >> > > As said, there are some screenshots from Wireshark trace for
> > > > >> HttpClient 4
> > > > >> > > run. I am not network expert, so I don't dare to judge them,
> > but I
> > > > >> > included
> > > > >> > > what I saw interesting. I think I can share some parts
> > (privately)
> > > > >> should
> > > > >> > > you want to study the trace in detail.
> > > > >> > >
> > > > >> > > Finally, when I disabled keep-alive from nginx configuration,
> > the
> > > > >> problem
> > > > >> > > was gone (as earlier). So, the problem is somehow related to
> > > > >> keep-alive.
> > > > >> > >
> > > > >> > > Note that in all scenarios CPU goes to 100% quite immediately
> > > (also
> > > > on
> > > > >> > 2.13
> > > > >> > > now, probably because of high throughput, I assume my
> > workstation
> > > is
> > > > >> the
> > > > >> > > bottleneck here) but without keep-alive or with JAVA impl,
> > things
> > > > work
> > > > >> > and
> > > > >> > > with keep-alive enabled and HttpClient 4, things break apart.
> As
> > > > >> before,
> > > > >> > on
> > > > >> > > JMeter 2.13 everything works.
> > > > >> > >
> > > > >> > > Tuukka
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <
> > > > >> > > felix.schumacher@internetallee.de <javascript:;>
> <javascript:;>> wrote:
> > > > >> > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
> > > > >> > > > tuukka.mustonen@gmail.com <javascript:;> <javascript:;>>:
> > > > >> > > > >@Philippe:
> > > > >> > > > >
> > > > >> > > > >0. Alright, I will.
> > > > >> > > > >1. Will do.
> > > > >> > > > >2. Sure.
> > > > >> > > > >3. To let the network connection state restore? Ok, will
> do.
> > > > >> > > > >
> > > > >> > > > >I am having some other stuff I need to attend today, but I
> > will
> > > > get
> > > > >> > > > >back to
> > > > >> > > > >this in the afternoon at latest.
> > > > >> > > >
> > > > >> > > > Could you also try to set loggerpanel.usejsyntaxtext to
> false
> > in
> > > > >> your
> > > > >> > > > settings?
> > > > >> > > >
> > > > >> > > > Are there more log entries, when you use httpclient 4
> instead
> > of
> > > > >> Java?
> > > > >> > > >
> > > > >> > > > Vladimir has found signs for a event congestion in the awt
> > event
> > > > >> queue,
> > > > >> > > > that show in your threaddumps.
> > > > >> > > >
> > > > >> > > > Regards,
> > > > >> > > >  Felix
> > > > >> > > >
> > > > >> > > > >
> > > > >> > > > >Tuukka
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
> > > > >> > > > >philippe.mouawad@gmail.com <javascript:;> <javascript:;>>
> wrote:
> > > > >> > > > >
> > > > >> > > > >> Hello,
> > > > >> > > > >> I've created this bugzilla for investigation and to share
> > > with
> > > > >> you
> > > > >> > > > >the plan
> > > > >> > > > >> I have used to run my own tests:
> > > > >> > > > >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
> > > > >> > > > >>
> > > > >> > > > >> I use a Tomcat 8.5.11 with a localhost and a certificate
> > > > >> generated
> > > > >> > > > >and auto
> > > > >> > > > >> signed.
> > > > >> > > > >>
> > > > >> > > > >> So here is what I get on a Mac Book Pro Mid 2015 OSX El
> > > Capitan
> > > > >> 2.5
> > > > >> > > > >Ghz
> > > > >> > > > >> Intel Core i7
> > > > >> > > > >> Java 8 u121
> > > > >> > > > >> -Xmx512m -Xms512m
> > > > >> > > > >> Test in GUI mode (ANTI-PATTERN)
> > > > >> > > > >>
> > > > >> > > > >> 2.13:
> > > > >> > > > >> - CPU is at 100%
> > > > >> > > > >> - No error
> > > > >> > > > >> - No leak
> > > > >> > > > >>
> > > > >> > > > >> 3.2 nightly with default settings :
> > > > >> > > > >> - CPU is at 100%
> > > > >> > > > >> - No error
> > > > >> > > > >> - No leak
> > > > >> > > > >>
> > > > >> > > > >> 3.2 nightly with this in user.properties :
> > > > >> > > > >> httpclient4.time_to_live=0
> > > > >> > > > >> httpclient4.validate_after_inactivity=0
> > > > >> > > > >>
> > > > >> > > > >> - CPU is at 100%
> > > > >> > > > >> - No error
> > > > >> > > > >> - No leak
> > > > >> > > > >>
> > > > >> > > > >> I attached the Heap Graphs to bugzilla and the plan.
> > > > >> > > > >>
> > > > >> > > > >> So for now I don't  reproduce your issue but I am not in
> > the
> > > > same
> > > > >> > > > >> configuration as I am not hitting the same website.
> > > > >> > > > >>
> > > > >> > > > >> So
> > > > >> > > > >> 0/ Could you provide by running only 5 requests, the
> > response
> > > > >> header
> > > > >> > > > >that
> > > > >> > > > >> your website returns ?
> > > > >> > > > >> 1/ Could you try my test plan by replacing with your
> target
> > > > host
> > > > >> and
> > > > >> > > > >see if
> > > > >> > > > >> behaviour is the same ?
> > > > >> > > > >> 2/ Ensure nothing is running (No Spotify :-) ) except
> your
> > > > JMeter
> > > > >> > and
> > > > >> > > > >> monitor network as per Felix note
> > > > >> > > > >> 3/ Run 1 test after the other but ensure you wait a bit
> > > between
> > > > >> each
> > > > >> > > > >one
> > > > >> > > > >>
> > > > >> > > > >>
> > > > >> > > > >> Thanks
> > > > >> > > > >>
> > > > >> > > > >>
> > > > >> > > > >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
> > > > >> > > > >> felix.schumacher@internetallee.de <javascript:;>
> <javascript:;>> wrote:
> > > > >> > > > >>
> > > > >> > > > >> >
> > > > >> > > > >> >
> > > > >> > > > >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka
> Mustonen <
> > > > >> > > > >> > tuukka.mustonen@gmail.com <javascript:;>
> <javascript:;>>:
> > > > >> > > > >> > >Ok I re-ran the test.
> > > > >> > > > >> > >
> > > > >> > > > >> > >Took Heap Dump just before JMeter froze.
> > > > >> > > > >> > >
> > > > >> > > > >> > >Took Thread Dumps right after JMeter froze.
> > > > >> > > > >> > >
> > > > >> > > > >> >
> > > > >> > > > >>https://drive.google.com/drive/folders/
> 0B1uBdVuLED5NNFNqal
> > > > >> lGam13MTA
> > > > >> > > > >> >
> > > > >> > > > >> > That folder seems to be empty now.
> > > > >> > > > >> >
> > > > >> > > > >> > Are you running the tests with a security manager
> > enabled?
> > > > >> > > > >> >
> > > > >> > > > >> > Can you share the test or at least the used requests?
> > > > >> > > > >> >
> > > > >> > > > >> > Are you using a proxy and if so, how is it configured?
> > > > >> > > > >> >
> > > > >> > > > >> > Is jmeter running locally, or over something like rdp?
> > > > >> > > > >> >
> > > > >> > > > >> > Can you look at the used network connections like
> netstat
> > > > >> -etapn
> > > > >> > > > >would do
> > > > >> > > > >> > on Linux?
> > > > >> > > > >> >
> > > > >> > > > >> > Regards,
> > > > >> > > > >> >  Felix
> > > > >> > > > >> >
> > > > >> > > > >> > >
> > > > >> > > > >> > >...
> > > > >> > > > >> > >
> > > > >> > > > >> > >For the fun of it, I also tried upgrading to the
> latest
> > > > >> > HttpClient
> > > > >> > > > >> > >4.5.3
> > > > >> > > > >> > >version. No effect.
> > > > >> > > > >> > >
> > > > >> > > > >> > >I also noticed that if I ping (google) during the perf
> > > > >> testing,
> > > > >> > > > >Windows
> > > > >> > > > >> > >ping reports "General failure." when JMeter breaks.
> > After
> > > > >> killing
> > > > >> > > > >it,
> > > > >> > > > >> > >connectivity restores and there are replies from the
> > > server
> > > > >> > again.
> > > > >> > > > >It
> > > > >> > > > >> > >seems
> > > > >> > > > >> > >like JMeter/HttpClient kills my whole network
> > > connection...
> > > > >> > > > >> > >
> > > > >> > > > >> > >Tuukka
> > > > >> > > > >> > >
> > > > >> > > > >> > >
> > > > >> > > > >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > > > >> > > > >> > >philippe.mouawad@gmail.com <javascript:;>
> <javascript:;>> wrote:
> > > > >> > > > >> > >
> > > > >> > > > >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> > > > >> > > > >> > ><tuukka.mustonen@gmail.com <javascript:;>
> <javascript:;>
> > > > >> > > > >> > >> >
> > > > >> > > > >> > >> wrote:
> > > > >> > > > >> > >>
> > > > >> > > > >> > >> > Hi, I'm running solely on Win 10 now. In
> > re-attempts I
> > > > >> think
> > > > >> > I
> > > > >> > > > >saw
> > > > >> > > > >> > >> slightly
> > > > >> > > > >> > >> > different behavior in my Linux guest. I'll write
> > more
> > > > >> about
> > > > >> > > > >that
> > > > >> > > > >> > >later,
> > > > >> > > > >> > >> but
> > > > >> > > > >> > >> > for simplicity I'm testing only on Windows host
> now:
> > > > >> > > > >> > >> >
> > > > >> > > > >> > >> > Thread dumps are downloadable at:
> > > > >> > > > >> > >> >
> > > > >> > > > >https://drive.google.com/drive/folders/
> 0B1uBdVuLED5NS1RFbEF
> > > > >> uSS1iejA
> > > > >> > > > >> > >> >
> > > > >> > > > >> > >> > (Took them with 7-8s interval I think, hope it's
> > ok).
> > > > >> > > > >> > >> >
> > > > >> > > > >> > >> > I took the thread dumps once JMeter started
> filling
> > in
> > > > >> > errors,
> > > > >> > > > >> > >which is
> > > > >> > > > >> > >> > somewhat after my CPU usage hit 100% (and music
> > > playback
> > > > >> > > > >started to
> > > > >> > > > >> > >> distort
> > > > >> > > > >> > >> > etc.).
> > > > >> > > > >> > >> >
> > > > >> > > > >> > >> > I also created the head dump, note that I took it
> > > after
> > > > >> > > > >running the
> > > > >> > > > >> > >test
> > > > >> > > > >> > >> > and after restarting JMeter, as JMeter froze
> during
> > > the
> > > > >> test
> > > > >> > > > >and I
> > > > >> > > > >> > >had to
> > > > >> > > > >> > >> > kill it. DId you mean I should take it *during*
> the
> > > > test?
> > > > >> > > > >> > >>
> > > > >> > > > >> > >>
> > > > >> > > > >> > >> YES.
> > > > >> > > > >> > >>
> > > > >> > > > >> > >>
> > > > >> > > > >> > >> > Anyway, before
> > > > >> > > > >> > >> > sharing it I want to make sure it can't contain
> > > anything
> > > > >> > > > >> > >confidential?
> > > > >> > > > >> > >> > (parts of my test plan, passwords, environment
> > > > variables,
> > > > >> > > > >> > >anything...)
> > > > >> > > > >> > >> >
> > > > >> > > > >> > >> > Tuukka
> > > > >> > > > >> > >> >
> > > > >> > > > >> > >> >
> > > > >> > > > >> > >> >
> > > > >> > > > >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK
> > > Support
> > > > <
> > > > >> > > > >> > >> > support@ubikloadpack.com <javascript:;>
> <javascript:;>> wrote:
> > > > >> > > > >> > >> >
> > > > >> > > > >> > >> > > Hello,
> > > > >> > > > >> > >> > > To investigate the problem:
> > > > >> > > > >> > >> > > 1/ to create  thread dumps, install a JDK of
> same
> > > > >> version
> > > > >> > as
> > > > >> > > > >the
> > > > >> > > > >> > >one
> > > > >> > > > >> > >> > > running your JMeter.
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > > Run until you get the problem, then call using
> > same
> > > > >> user as
> > > > >> > > > >the
> > > > >> > > > >> > >one
> > > > >> > > > >> > >> > running
> > > > >> > > > >> > >> > > JMeter 3 times at 5s interval:
> > > > >> > > > >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >>
> > > > >> > output_file.txt
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > > It will also help if you provide the structure
> of
> > > your
> > > > >> Test
> > > > >> > > > >Plan.
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > > 2/ It would also help if you can provide a Heap
> > > Dump.
> > > > >> With
> > > > >> > > > >JMeter
> > > > >> > > > >> > >GUI,
> > > > >> > > > >> > >> > use
> > > > >> > > > >> > >> > > menu Help > Create a Heap Dump
> > > > >> > > > >> > >> > > This will create a Heap Dump (rather big file) ,
> > it
> > > > >> would
> > > > >> > > > >help a
> > > > >> > > > >> > >lot if
> > > > >> > > > >> > >> > you
> > > > >> > > > >> > >> > > can make it available.
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > > Regards
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen
> <
> > > > >> > > > >> > >> > tuukka.mustonen@gmail.com <javascript:;>
> <javascript:;>
> > > > >> > > > >> > >> > > >
> > > > >> > > > >> > >> > > wrote:
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> > > > @Philippe:
> > > > >> > > > >> > >> > > >
> > > > >> > > > >> > >> > > > Yeah, I am normally running in non-GUI mode,
> but
> > > to
> > > > >> > > > >> > >> > > > study/inspect/understand this problem I am
> > running
> > > > in
> > > > >> GUI
> > > > >> > > > >mode.
> > > > >> > > > >> > >> JMeter
> > > > >> > > > >> > >> > > 2.13
> > > > >> > > > >> > >> > > > works just fine in GUI mode for these tests so
> > > > JMeter
> > > > >> 3.x
> > > > >> > > > >> > >should too.
> > > > >> > > > >> > >> > > >
> > > > >> > > > >> > >> > > > I'll give the nightly build a try.
> > > > >> > > > >> > >> > > >
> > > > >> > > > >> > >> > > > Tuukka
> > > > >> > > > >> > >> > > >
> > > > >> > > > >> > >> > > >
> > > > >> > > > >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe
> > Mouawad <
> > > > >> > > > >> > >> > > > philippe.mouawad@gmail.com <javascript:;>
> <javascript:;>
> > > > >> > > > >> > >> > > > > wrote:
> > > > >> > > > >> > >> > > >
> > > > >> > > > >> > >> > > > > On Wednesday, February 1, 2017, Tuukka
> > Mustonen
> > > <
> > > > >> > > > >> > >> > > > tuukka.mustonen@gmail.com <javascript:;>
> <javascript:;>
> > > > >> > > > >> > >> > > > > >
> > > > >> > > > >> > >> > > > > wrote:
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > > > @Philippe:
> > > > >> > > > >> > >> > > > > >
> > > > >> > > > >> > >> > > > > > 1. Yeah, I compared them with the same
> plan,
> > > > same
> > > > >> > > > >machine.
> > > > >> > > > >> > >Should
> > > > >> > > > >> > >> > be
> > > > >> > > > >> > >> > > > > > everything equal.
> > > > >> > > > >> > >> > > > > > 2. I tried both the defaults in jmeter.bat
> > > > >> (-Xms512m
> > > > >> > > > >> > >-Xmx512m)
> > > > >> > > > >> > >> and
> > > > >> > > > >> > >> > as
> > > > >> > > > >> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I
> > > > didn't
> > > > >> > > > >tweak
> > > > >> > > > >> > >them much
> > > > >> > > > >> > >> > > > > because I
> > > > >> > > > >> > >> > > > > > don't believe the issue lies there.
> > > > >> > > > >> > >> > > > > > 3. No, I do just simple HTTP GETs (against
> > > RESTy
> > > > >> > API).
> > > > >> > > > >> > >> > > > > > 4. I am not familiar with that but I can
> > try.
> > > If
> > > > >> you
> > > > >> > > > >have
> > > > >> > > > >> > >> > > instructions
> > > > >> > > > >> > >> > > > to
> > > > >> > > > >> > >> > > > > > point to, please do.
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > > > 5. I am running with GUI.
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > > This is a known anti-pattern .
> > > > >> > > > >> > >> > > > > switch to non gui mode if you want correct
> > > > results.
> > > > >> > > > >> > >> > > > > http://jmeter.apache.org/
> > > > >> > usermanual/best-practices.html
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > > http://www.ubik-ingenierie.
> > > > >> > com/blog/jmeter_performance_
> > > > >> > > > >> > >> tuning_tips/
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > > > About file sending:
> > > > >> > > > >> > >> > > > > >
> > > > >> > > > >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > >  Could you give nightly build a try ?
> > > > >> > > > >> > >> > > > > http://jmeter.apache.org/nightly.html
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > > > Tuukka
> > > > >> > > > >> > >> > > > > >
> > > > >> > > > >> > >> > > > > >
> > > > >> > > > >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe
> > > > Mouawad <
> > > > >> > > > >> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> <javascript:;>
> > > > >> > > <javascript:;>
> > > > >> > > > >> > >> > > > > > > wrote:
> > > > >> > > > >> > >> > > > > >
> > > > >> > > > >> > >> > > > > > > Also are you sending files in post body
> ?
> > > > >> > > > >> > >> > > > > > >
> > > > >> > > > >> > >> > > > > > > Thank you
> > > > >> > > > >> > >> > > > > > >
> > > > >> > > > >> > >> > > > > > > On Wednesday, February 1, 2017, Philippe
> > > > >> Mouawad <
> > > > >> > > > >> > >> > > > > > > philippe.mouawad@gmail.com
> <javascript:;> <javascript:;>
> > > > >> > > <javascript:;>>
> > > > >> > > > >> > >> > > > > > > wrote:
> > > > >> > > > >> > >> > > > > > >
> > > > >> > > > >> > >> > > > > > > > Hello,
> > > > >> > > > >> > >> > > > > > > > Few questions, please try to answer
> them
> > > > all:
> > > > >> > > > >> > >> > > > > > > >
> > > > >> > > > >> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with
> > same
> > > > >> > > > >plan/sape
> > > > >> > > > >> > >> machine...
> > > > >> > > > >> > >> > > > > > everything
> > > > >> > > > >> > >> > > > > > > > equal ?
> > > > >> > > > >> > >> > > > > > > > 2/ What are your Xmx settings
> > > > >> > > > >> > >> > > > > > > > 3/ does your plan download embedded
> > > > resources
> > > > >> ?
> > > > >> > > > >> > >> > > > > > > > 4/ when issue happens can you make 3
> > > thread
> > > > >> dumps
> > > > >> > > > >at 5
> > > > >> > > > >> > >> seconds
> > > > >> > > > >> > >> > > > > interval
> > > > >> > > > >> > >> > > > > > > > and send file ? (use jstack)
> > > > >> > > > >> > >> > > > > > > > 5/ are you running gui or non gui ?
> > > > >> > > > >> > >> > > > > > > >
> > > > >> > > > >> > >> > > > > > > > Thanks
> > > > >> > > > >> > >> > > > > > > > Regards
> > > > >> > > > >> > >> > > > > > > >
> > > > >> > > > >> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka
> > > > >> Mustonen <
> > > > >> > > > >> > >> > > > > > > tuukka.mustonen@gmail.com
> <javascript:;> <javascript:;>
> > > > >> > > <javascript:;>
> > > > >> > > > >> > >> > > > > > > >
> > > > >> > > > ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> <javascript:;>
> > > > >> > <javascript:;>
> > > > >> > > > >> > >> > > > > <javascript:;>');>>
> > > > >> > > > >> > >> > > > > > wrote:
> > > > >> > > > >> > >> > > > > > > >
> > > > >> > > > >> > >> > > > > > > >> Hi,
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1
> I
> > > > >> > experience
> > > > >> > > > >high
> > > > >> > > > >> > >CPU
> > > > >> > > > >> > >> > usage
> > > > >> > > > >> > >> > > > > that
> > > > >> > > > >> > >> > > > > > > >> renders JMeter unusable. CPU usage
> > ramps
> > > > >> quickly
> > > > >> > > > >up
> > > > >> > > > >> > >and
> > > > >> > > > >> > >> after
> > > > >> > > > >> > >> > > > having
> > > > >> > > > >> > >> > > > > > > >> ~100-200 concurrent users or so (in
> ~30
> > > > >> > seconds),
> > > > >> > > > >it
> > > > >> > > > >> > >hits
> > > > >> > > > >> > >> 100%
> > > > >> > > > >> > >> > > on
> > > > >> > > > >> > >> > > > my
> > > > >> > > > >> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter
> > > > freezes
> > > > >> and
> > > > >> > > > >I
> > > > >> > > > >> > >have to
> > > > >> > > > >> > >> > kill
> > > > >> > > > >> > >> > > > it.
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> On 2.13 everything goes smoothly
> (with
> > > > >> default
> > > > >> > > > >> > >settings).
> > > > >> > > > >> > >> CPU
> > > > >> > > > >> > >> > > > usage
> > > > >> > > > >> > >> > > > > > > stays
> > > > >> > > > >> > >> > > > > > > >> at ~30% even with 500 users.
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> The experience is similar on both
> > Windows
> > > > 10
> > > > >> and
> > > > >> > > > >Linux
> > > > >> > > > >> > >> (Ubuntu
> > > > >> > > > >> > >> > > > 14.04
> > > > >> > > > >> > >> > > > > > LTS
> > > > >> > > > >> > >> > > > > > > >> guest in Virtualbox).
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> The problems I saw in JMeter 3:
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> - java.net.SocketException:
> Permission
> > > > >> denied:
> > > > >> > > > >connect
> > > > >> > > > >> > >> > > > > > > >> - After a while it runs out of heap
> > space
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> For "Permission denied: connect" I
> > tried
> > > to
> > > > >> set
> > > > >> > > > >> > >> > > > > > > >> java.net.preferIPv4Stack=true
> > > > >> > > > >> > >> > > > > > > >> (in a few places to make sure it
> would
> > > > >> really be
> > > > >> > > > >in
> > > > >> > > > >> > >use) and
> > > > >> > > > >> > >> > > tried
> > > > >> > > > >> > >> > > > > to
> > > > >> > > > >> > >> > > > > > > run
> > > > >> > > > >> > >> > > > > > > >> JMeter as administrator (in Win). No
> > > > effect.
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> I also tried increasing heap size,
> but
> > > that
> > > > >> > > > >didn't
> > > > >> > > > >> > >help
> > > > >> > > > >> > >> > either.
> > > > >> > > > >> > >> > > > And
> > > > >> > > > >> > >> > > > > I
> > > > >> > > > >> > >> > > > > > > >> think
> > > > >> > > > >> > >> > > > > > > >> memory size is not the root cause
> here
> > -
> > > > >> rather
> > > > >> > > > >the
> > > > >> > > > >> > >result.
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> I am normally connected to VPN but I
> > also
> > > > >> tried
> > > > >> > > > >> > >disabling
> > > > >> > > > >> > >> it.
> > > > >> > > > >> > >> > No
> > > > >> > > > >> > >> > > > > > effect.
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> I tried disabling with and without
> IPv6
> > > > >> enabled
> > > > >> > > > >in
> > > > >> > > > >> > >Windows
> > > > >> > > > >> > >> > > network
> > > > >> > > > >> > >> > > > > > > >> connection settings. No effect.
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> I am running a few extra JMeter
> > plugins.
> > > > >> Tried
> > > > >> > > > >cutting
> > > > >> > > > >> > >my
> > > > >> > > > >> > >> test
> > > > >> > > > >> > >> > > > > > scenario
> > > > >> > > > >> > >> > > > > > > to
> > > > >> > > > >> > >> > > > > > > >> minimum. No effect.
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> I am running anti-virus software
> that I
> > > > >> cannot
> > > > >> > > > >> > >disable, due
> > > > >> > > > >> > >> to
> > > > >> > > > >> > >> > > > this
> > > > >> > > > >> > >> > > > > > > being
> > > > >> > > > >> > >> > > > > > > >> forced by IT department. However,
> it's
> > > not
> > > > a
> > > > >> > > > >problem
> > > > >> > > > >> > >with
> > > > >> > > > >> > >> 2.13
> > > > >> > > > >> > >> > > so
> > > > >> > > > >> > >> > > > I
> > > > >> > > > >> > >> > > > > > > assume
> > > > >> > > > >> > >> > > > > > > >> it shouldn't be a problem here
> either.
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> FINALLY, once I set HTTP request
> > > > >> implementation
> > > > >> > > > >from
> > > > >> > > > >> > >Apache
> > > > >> > > > >> > >> > > > > > HttpClient 4
> > > > >> > > > >> > >> > > > > > > >> to
> > > > >> > > > >> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem
> > > > vanishes
> > > > >> -
> > > > >> > > > >CPU
> > > > >> > > > >> > >stays
> > > > >> > > > >> > >> where
> > > > >> > > > >> > >> > > it
> > > > >> > > > >> > >> > > > > did
> > > > >> > > > >> > >> > > > > > > with
> > > > >> > > > >> > >> > > > > > > >> JMeter 2.13.
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121
> on
> > > > Win,
> > > > >> > > > >update
> > > > >> > > > >> > >111 on
> > > > >> > > > >> > >> > > Linux).
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> Why does (the recommended)
> HttpClient 4
> > > > cause
> > > > >> > > > >problems
> > > > >> > > > >> > >here?
> > > > >> > > > >> > >> > Is
> > > > >> > > > >> > >> > > > > this a
> > > > >> > > > >> > >> > > > > > > >> know
> > > > >> > > > >> > >> > > > > > > >> bug/issue? JMeter 3.x is running
> newer
> > > > >> version
> > > > >> > > > >> > >compared to
> > > > >> > > > >> > >> > 2.13,
> > > > >> > > > >> > >> > > > but
> > > > >> > > > >> > >> > > > > > > what
> > > > >> > > > >> > >> > > > > > > >> change in HttpClient 4 might cause
> the
> > > > >> problems
> > > > >> > > > >> > >described?
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> Any ideas?
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >> Regards,
> > > > >> > > > >> > >> > > > > > > >> Tuukka
> > > > >> > > > >> > >> > > > > > > >>
> > > > >> > > > >> > >> > > > > > > >
> > > > >> > > > >> > >> > > > > > > >
> > > > >> > > > >> > >> > > > > > > > --
> > > > >> > > > >> > >> > > > > > > > Cordialement.
> > > > >> > > > >> > >> > > > > > > > Philippe Mouawad.
> > > > >> > > > >> > >> > > > > > > >
> > > > >> > > > >> > >> > > > > > > >
> > > > >> > > > >> > >> > > > > > > >
> > > > >> > > > >> > >> > > > > > > >
> > > > >> > > > >> > >> > > > > > >
> > > > >> > > > >> > >> > > > > > > --
> > > > >> > > > >> > >> > > > > > > Cordialement.
> > > > >> > > > >> > >> > > > > > > Philippe Mouawad.
> > > > >> > > > >> > >> > > > > > >
> > > > >> > > > >> > >> > > > > >
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > > > --
> > > > >> > > > >> > >> > > > > Cordialement.
> > > > >> > > > >> > >> > > > > Philippe Mouawad.
> > > > >> > > > >> > >> > > > >
> > > > >> > > > >> > >> > > >
> > > > >> > > > >> > >> > >
> > > > >> > > > >> > >> >
> > > > >> > > > >> > >>
> > > > >> > > > >> > >>
> > > > >> > > > >> > >>
> > > > >> > > > >> > >> --
> > > > >> > > > >> > >> Cordialement.
> > > > >> > > > >> > >> Philippe Mouawad.
> > > > >> > > > >> > >>
> > > > >> > > > >> >
> > > > >> > > > >> >
> > > > >> > > > >-----------------------------
> ------------------------------
> > > > >> ----------
> > > > >> > > > >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.
> > apache.org
> > > > >> > > <javascript:;>
> > > > >> > > > >> > For additional commands, e-mail:
> > > user-help@jmeter.apache.org <javascript:;>
> > > > >> > > <javascript:;>
> > > > >> > > > >> >
> > > > >> > > > >> >
> > > > >> > > > >>
> > > > >> > > > >>
> > > > >> > > > >> --
> > > > >> > > > >> Cordialement.
> > > > >> > > > >> Philippe Mouawad.
> > > > >> > > > >>
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > Cordialement.
> > > > >> > Philippe Mouawad.
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cordialement.
> > > > > Philippe Mouawad.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Cordialement.
> > > > Philippe Mouawad.
> > > >
> > >
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
>


-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

> https://tools.ietf.org/id/draft-thomson-hybi-http-
timeout-01.html#rfc.section.2.1
> But server is allowed not to send it.
> But I think for a load test, it is better to adjust the 3 parameters to a
> value that is similar to the server keep-alive timeout

Cool, never seen anyone mention/recommend that header :O

I actually tried setting TTL to 60 seconds and while I think it worked, the
result was that in the end of the test, some threads would never close
their connection. Tuning TTL down to 10 seconds fixed this. Not sure what
was going on, but yeah, that's a potential separate issue - no need to
prolong this thread any longer.

> I mean if server does not send timeout, but timeouts connection on its
> side, JMeter might
> try to reuse a "stale" connection which has a cost and will only drop it
on
> next use.
> For high throughput tests it migh be a problem.

Alright.

> https://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx

Interesting. I googled around a bit and it would appear that with default
Windows settings one shouldn't hit into it (ephemeral port exhaustion) very
easily (so I guess your 2s TTL is ok, although it feels overly short). But
yeah, maybe we have special configuration or something...

Thanks again!

Tuukka



On Sat, Feb 4, 2017 at 10:22 PM, Philippe Mouawad <
philippe.mouawad@gmail.com> wrote:

> On Sat, Feb 4, 2017 at 8:05 PM, Tuukka Mustonen <tuukka.mustonen@gmail.com
> >
> wrote:
>
> > > In the case of NGinx , it seems no header timeout is sent.
> > > So if you set a value > to 0, you can control the max idle duration of
> a
> > > connection from Client (JMeter) side to avoid storing too many
> > connections
> > > that won't be reused and can impact JMeter heap.
> > > By default it's not set so no impact.
> >
> > Hmm, what do you mean by "header timeout"? Afaik, the server cannot
> > communicate a timeout for the keep-alived connection to the client. The
> > server probably has some timeout (I think in nginx it's 75s by default)
> but
> > the client doesn't know it. Server will just close the connection at some
> > point.
> >
> https://tools.ietf.org/id/draft-thomson-hybi-http-
> timeout-01.html#rfc.section.2.1
> But server is allowed not to send it.
> But I think for a load test, it is better to adjust the 3 parameters to a
> value that is similar to the server keep-alive timeout
>
>
>
> > Also I'm not 100% sure what you mean by "to avoid storing too many
> > connections that won't be reused"? The client will re-use connection
> until
> > it's closed by TTL configuration or if the server closes the connection,
> > right?
>
>
> I mean if server does not send timeout, but timeouts connection on its
> side, JMeter might
> try to reuse a "stale" connection which has a cost and will only drop it on
> next use.
> For high throughput tests it migh be a problem.
>
>
>
> > I though with idletimeout I could just make sure that client closes
> > the connection before the server would (and avoid any potential failed
> > requests if client sent a request just while the server closes
> connection)?
> >
> You understood correctly
>
> >
> > I'm not sure, the documentation for that parameter could maybe be
> improved
> > a tiny bit?
> >
>
> If you want to contribute a patch for that you're welcome. It's in
> properties_reference.xml.
> http://jmeter.apache.org/building.html
> Otherwise we'll try to fix it.
>
> >
> > > But investigating your issue, I discovered that in fact
> > > "httpclient4.validate_after_inactivity" has no impact due to another
> bug
> > > here I am also responsible of :-( .
> > > validate() method always return true so JMeter always consider
> connection
> > > as valid.
> > > So only "httpclient4.time_to_live" does its job here.
> >
> > You seem to be author of lots of commits - it's only human to forget
> > something every now and then :)
> >
> > > What's happening here is that you're most probably saturating the
> dynamic
> > > port range
> >
> > If you mean that I would be going over the number of available dynamic
> > ports, I don't think that is it. The test breaks at ~40-50 seconds or so
> > and at that point there shouldn't be even 10k of connections
> (potentially)
> > open. But, I'm definitively hitting some limit. I will contact our IT
> > department - who knows what magic configuration we have.
> >
>
> https://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx
>
> >
> > > It's because TTL is > 0, so it will be included in the management of
> idle
> > > connections.
> > > ...
> > > Because JMeter will close connection immediately and inform server to
> do
> > > the same.
> >
> > Ah, good point. What's the policy (=timeline) for closing the idle
> > connections that go over TTL?
> >
> It will be closed when a new request is made by same user and it has
> expired.
>
> >
> > > Thanks a lot for all your tests which helped us narrow this issue.
> > > Besides Vladimir and Felix are working on improving performances in GUI
> > > Mode but the team still advises against using it for Load Testing.
> >
> > Cool. Yeah, I run tests normally from Jenkins in command line mode, but I
> > always develop/debug in GUI (as does everybody, I would believe). When in
> > GUI mode, I don't care about performance really - just that the tests
> pass.
> > But better performance is welcome of course.
> >
> > > If you used the nightly build (select Jenkins one) you should be able
> to
> > > see that now :
> > > httpclient4.validate_after_inactivity = 1700
> > > and has an effect.
> >
> > Thanks for the quick fixes! I think I will test it next week some point,
> > just for the curiosity, to see it in effect in Wireshark. I assume this
> > check sends some dummy package to the server and should be visible in the
> > trace. Correct me if wrong.
> >
> > Thanks. Your feedback is very welcome.
>
> Tuukka
> >
> >
> >
> > On Sat, Feb 4, 2017 at 5:13 PM, Philippe Mouawad <
> > philippe.mouawad@gmail.com
> > > wrote:
> >
> > > Hello Tukka,
> > > I have commited :
> > > https://bz.apache.org/bugzilla/show_bug.cgi?id=60690
> > > https://bz.apache.org/bugzilla/show_bug.cgi?id=60689
> > >
> > > If you used the nightly build (select Jenkins one) you should be able
> to
> > > see that now :
> > > httpclient4.validate_after_inactivity = 1700
> > > and has an effect.
> > >
> > > Still in your particular case as you've tested you'll need to increase
> > > httpclient4.time_to_live to avoid too much connections creations that
> > your
> > > OS/Machine/Network does not correctly handle.
> > >
> > > But beware that this may not be realistic as if you're server calls are
> > > made from clients that do not use keep alive or persist their
> connections
> > > for a short time.
> > > To ensure your server correctly handles this , you should test from a
> > fine
> > > tuned machine that has a correct network with target server.
> > >
> > >
> > > Regards
> > > @philmdot
> > >
> > > On Sat, Feb 4, 2017 at 3:16 PM, Philippe Mouawad <
> > > philippe.mouawad@gmail.com
> > > > wrote:
> > >
> > > >
> > > >
> > > > On Sat, Feb 4, 2017 at 12:58 PM, Tuukka Mustonen <
> > > > tuukka.mustonen@gmail.com> wrote:
> > > >
> > > >> On point, Philippe!
> > > >>
> > > >> I think *httpclient4.idletimeout* is not of relevance here - my
> server
> > > >> sends proper Keep-Alive headers so this option shouldn't have
> impact.
> > > Good
> > > >> to know, for sure.
> > > >>
> > > > In the case of NGinx , it seems no header timeout is sent.
> > > > So if you set a value > to 0, you can control the max idle duration
> of
> > a
> > > > connection from Client (JMeter) side to avoid storing too many
> > > connections
> > > > that won't be reused and can impact JMeter heap.
> > > > By default it's not set so no impact.
> > > >
> > > >>
> > > >> *httpclient4.validate_after_inactivity* and
> *httpclient4.time_to_live
> > > >> default* values (2 seconds) sound conflicting: TTL being the same as
> > > >> validate_after_inactivity leads to connections never getting checked
> > as
> > > >> connections are discarded after 2 seconds anyway? So
> > > >> validate_after_inactivity is not really in use by default if I
> > > understood
> > > >> it right.
> > > >>
> > > > Good catch. It somehow went out of my radar, I remember it was
> > something
> > > I
> > > > had in mind but it seems I failed to change it.
> > > > It's a bug or at least bad default value.
> > > >
> > > > But investigating your issue, I discovered that in fact
> > > > "httpclient4.validate_after_inactivity" has no impact due to another
> > bug
> > > > here I am also responsible of :-( .
> > > > validate() method always return true so JMeter always consider
> > connection
> > > > as valid.
> > > > So only "httpclient4.time_to_live" does its job here.
> > > >
> > > >
> > > >>
> > > >> So I went on and tried these settings:
> > > >>
> > > >> 1) httpclient4.validate_after_inactivity=0 +
> > > >> httpclient4.time_to_live=2000
> > > >> -> CRASH
> > > >> 2) httpclient4.validate_after_inactivity=0 +
> > > >> httpclient4.time_to_live=1000
> > > >> -> CRASH
> > > >> 3) httpclient4.validate_after_inactivity=1 +
> > > >> httpclient4.time_to_live=2000
> > > >> -> CRASH
> > > >>
> > > >> 4) httpclient4.validate_after_inactivity=5000 +
> > > >> httpclient4.time_to_live=10000 -> SUCCESS
> > > >> 5) httpclient4.validate_after_inactivity=0 +
> > > >> httpclient4.time_to_live=10000
> > > >> -> SUCCESS
> > > >> 6) httpclient4.validate_after_inactivity=1 +
> > > >> httpclient4.time_to_live=10000
> > > >> -> SUCCESS
> > > >> 7) httpclient4.validate_after_inactivity=0 +
> > httpclient4.time_to_live=0
> > > >> ->
> > > >> SUCCESS
> > > >> 8) httpclient4.validate_after_inactivity=10000 +
> > > >> httpclient4.time_to_live=10000 -> SUCCESS
> > > >>
> > > >> Clearly increasing TTL makes the difference for me.
> > > >> validate_after_inactivity seems to have no effect. Although there is
> > > some
> > > >> packet loss, the connections themselves don't break.
> > > >>
> > > > Absolutely.
> > > >
> > > >>
> > > >> A) So... maybe my network chip configuration just cannot tolerate
> > JMeter
> > > >> opening fresh connections every 2 seconds? That translates to 250
> > > >> connections/s with 500 users- maybe it's enough to bloat my network
> IO
> > > and
> > > >> hit some policy (and I'll then get "connect: Permission denied")?
> > > Sounds a
> > > >> bit weird.
> > > >>
> > > >
> > > > It's also your OS, you need to tune the TCP stack to work with port
> > reuse
> > > > speed.
> > > > What's happening here is that you're most probably saturating the
> > dynamic
> > > > port range
> > > >
> > > >>
> > > >> So I also tried:
> > > >>
> > > >> 9) httpclient4.validate_after_inactivity=0 +
> > httpclient4.time_to_live=1
> > > >> ->
> > > >> CRASH
> > > >>
> > > >> And it crashes quite instantly. I guess it's because JMeter is
> > opening a
> > > >> new connection per request...
> > > >>
> > > > It's because TTL is > 0, so it will be included in the management of
> > idle
> > > > connections.
> > > >
> > > >>
> > > >> B) ...but if I keep the default 2s settings and just disable
> > Keep-Alive,
> > > >> then JMeter is opening new connection per request also, and it
> works.
> > > Why
> > > >> the difference?
> > > >>
> > > > Because JMeter will close connection immediately and inform server to
> > do
> > > > the same.
> > > >
> > > >
> > > >>
> > > >> C) Then, I wonder if your "standard workstation" is supposed to
> > tolerate
> > > >> these defaults? Can my network chip/driver/configuration just be so
> > bad?
> > > >>
> > > >
> > > > Yes at least your OS Tuning defaults are not well suited for Load
> > > Testing,
> > > > you need to tune.
> > > >
> > > >>
> > > >> I see the *httpclient4.validate_after_inactivity* and
> > > >> *httpclient4.time_to_live
> > > >> default* properties mentioned in the JMeter 3 changelog:
> > > >>
> > > >> "Improved and better performing validation mechanism for Stale
> > > connections
> > > >> and Keep-Alive management, see properties
> > > >> httpclient4.validate_after_inactivity and httpclient4.time_to_live"
> > > >>
> > > >
> > > > Yes, it was the intention.
> > > >
> > > >>
> > > >> And when I monitored TCP connection in Wireshark, JMeter 2.13 indeed
> > > keeps
> > > >> connections open waaaay longer (seemed a bit random).
> > > >>
> > > >
> > > > Yes, at that time the only solution was to play with idletimeout or
> set
> > > > "http.connection.stalecheck$Boolean=true" which had important impact
> > at
> > > > high throughput.
> > > >
> > > >>
> > > >> D) So I the new 2s default sounds a bit risky?
> > > >>
> > > > In your case yes, but you're in a "bad load testing config" :-)
> > > >
> > > >>
> > > >> In the end, increasing TTL seems to fix the issues for me and I can
> > > >> proceed
> > > >> upgrading to JMeter 3.
> > > >>
> > > >> Would love to hear your opinion/analysis on the open questions
> (A-D),
> > > but
> > > >> big thanks for your interest in my problem and the precious help
> > already
> > > >> at
> > > >> this point!
> > > >>
> > > >
> > > > Thanks a lot for all your tests which helped us narrow this issue.
> > > > Besides Vladimir and Felix are working on improving performances in
> GUI
> > > > Mode but the team still advises against using it for Load Testing.
> > > >
> > > >
> > > >> Tuukka
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Feb 3, 2017 at 9:40 PM, Philippe Mouawad <
> > > >> philippe.mouawad@gmail.com
> > > >> > wrote:
> > > >>
> > > >> > On Friday, February 3, 2017, Tuukka Mustonen <
> > > tuukka.mustonen@gmail.com
> > > >> >
> > > >> > wrote:
> > > >> >
> > > >> > > Sorry for the delay, got too busy yesterday.
> > > >> > >
> > > >> > > I have been running more tests now. I tried to a few variations
> to
> > > >> > pinpoint
> > > >> > > the issue, but first:
> > > >> > >
> > > >> > > @Philippe:
> > > >> > >
> > > >> > > 0) Response headers are like:
> > > >> > >
> > > >> > > Response headers:
> > > >> > > HTTP/1.1 200 OK
> > > >> > > Date: Fri, 03 Feb 2017 06:47:00 GMT
> > > >> > > Content-Type: application/json
> > > >> > > Content-Length: 374
> > > >> > > Connection: keep-alive
> > > >> > > Server: nginx/1.6.2
> > > >> > > ETag: 37926:1-37927:1
> > > >> > > Cache-Control: no-cache, no-store
> > > >> > > Pragma: no-cache
> > > >> > > Expires: 0
> > > >> > > X-Transaction: 0000-326srxia31tss3tc
> > > >> >
> > > >> >
> > > >> > As you pointed it later that issue is related to keep alive.
> > > >> > This is tunable , I 'll try to provide some advices
> > > >> >
> > > >> > But read:
> > > >> > http://jmeter.apache.org/usermanual/properties_reference.
> > > >> html#httpclient4
> > > >> >
> > > >> > And pay attention to:
> > > >> >
> > > >> > httpclient4.idletimeout
> > > >> > httpclient4.validate_after_inactivity
> > > >> > httpclient4.time_to_live
> > > >> >
> > > >> > It appears Default values do not suite your case.
> > > >> >
> > > >> >
> > > >> > > 1) Tried your test plan. Unfortunately, it does not make
> > difference.
> > > >> >
> > > >> > it was not made to fix issue only to make it simpler and
> indépendant
> > > of
> > > >> any
> > > >> > 3rd party plugin
> > > >> >
> > > >> > >
> > > >> > > 2) I ran Wireshark during a couple of tests. You can find a few
> > > >> > screenshots
> > > >> > > in a later link. Note that my whole network kinda crashes when
> > > JMeter
> > > >> 3
> > > >> > > gets stuck (ping gets "General failure" and some active
> > connections
> > > >> > crash,
> > > >> > > note that this seems a bit random).
> > > >> >
> > > >> > Thanks
> > > >> >
> > > >> > >
> > > >> > > 3) I gave it a few minutes to recover between the runs.
> > > >> > >
> > > >> > > @Felix:
> > > >> > >
> > > >> > > I ran all tests with loggerpanel.usejsyntaxtext=false.
> > > >> > >
> > > >> > > No, the amount of seems roughly the same.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > Then, about pinpointing the issue:
> > > >> > >
> > > >> > > I managed to reproduce this by installing nginx on a network
> > > computer
> > > >> > (sudo
> > > >> > > apt-get install nginx on debian 8). I did not configure anything
> > > and I
> > > >> > > simply bombarded the default nginx welcome page.
> > > >> > >
> > > >> > > I have attached Heap Dump, Thread Dumps and some other
> information
> > > in
> > > >> > these
> > > >> > > links:
> > > >> > >
> > > >> > > JAVA impl (that doesn't crash):
> > > >> > > https://drive.google.com/drive/folders/
> > 0B1uBdVuLED5NSFJvc2VSdE13a0k
> > > ?
> > > >> > > usp=sharing
> > > >> > >
> > > >> > > HttpCilent 4 impl (that crashes):
> > > >> > > https://drive.google.com/drive/folders/
> > 0B1uBdVuLED5NUGc1WWFYYkRncTQ
> > > ?
> > > >> > > usp=sharing
> > > >> > >
> > > >> > > I used Philippe's test plan and even simplified it a bit, the
> JMX
> > > >> file is
> > > >> > > in the latter link. Note that the errors occur (in my case)
> during
> > > the
> > > >> > > ramp-up so I also shortened to run just 30s after reaching 500
> > > users.
> > > >> > >
> > > >> > > With JAVA impl, I ran heap dump at ~450 users or so, and thread
> > > dumps
> > > >> > after
> > > >> > > that.
> > > >> > >
> > > >> > > With HttpClient 4 impl, I ran heap dump immediately after seeing
> > the
> > > >> > first
> > > >> > > error (which was at ~430 users or so) and thread dumps right
> after
> > > >> that
> > > >> > > (the first two dumps I think took a bit longer than normally to
> > > >> create).
> > > >> > > JMeter goes unresponsive a second or two after seeing the first
> > > error,
> > > >> > so I
> > > >> > > just barely had time to click to take the dump.
> > > >> > >
> > > >> > > As said, there are some screenshots from Wireshark trace for
> > > >> HttpClient 4
> > > >> > > run. I am not network expert, so I don't dare to judge them,
> but I
> > > >> > included
> > > >> > > what I saw interesting. I think I can share some parts
> (privately)
> > > >> should
> > > >> > > you want to study the trace in detail.
> > > >> > >
> > > >> > > Finally, when I disabled keep-alive from nginx configuration,
> the
> > > >> problem
> > > >> > > was gone (as earlier). So, the problem is somehow related to
> > > >> keep-alive.
> > > >> > >
> > > >> > > Note that in all scenarios CPU goes to 100% quite immediately
> > (also
> > > on
> > > >> > 2.13
> > > >> > > now, probably because of high throughput, I assume my
> workstation
> > is
> > > >> the
> > > >> > > bottleneck here) but without keep-alive or with JAVA impl,
> things
> > > work
> > > >> > and
> > > >> > > with keep-alive enabled and HttpClient 4, things break apart. As
> > > >> before,
> > > >> > on
> > > >> > > JMeter 2.13 everything works.
> > > >> > >
> > > >> > > Tuukka
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <
> > > >> > > felix.schumacher@internetallee.de <javascript:;>> wrote:
> > > >> > >
> > > >> > > >
> > > >> > > >
> > > >> > > > Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
> > > >> > > > tuukka.mustonen@gmail.com <javascript:;>>:
> > > >> > > > >@Philippe:
> > > >> > > > >
> > > >> > > > >0. Alright, I will.
> > > >> > > > >1. Will do.
> > > >> > > > >2. Sure.
> > > >> > > > >3. To let the network connection state restore? Ok, will do.
> > > >> > > > >
> > > >> > > > >I am having some other stuff I need to attend today, but I
> will
> > > get
> > > >> > > > >back to
> > > >> > > > >this in the afternoon at latest.
> > > >> > > >
> > > >> > > > Could you also try to set loggerpanel.usejsyntaxtext to false
> in
> > > >> your
> > > >> > > > settings?
> > > >> > > >
> > > >> > > > Are there more log entries, when you use httpclient 4 instead
> of
> > > >> Java?
> > > >> > > >
> > > >> > > > Vladimir has found signs for a event congestion in the awt
> event
> > > >> queue,
> > > >> > > > that show in your threaddumps.
> > > >> > > >
> > > >> > > > Regards,
> > > >> > > >  Felix
> > > >> > > >
> > > >> > > > >
> > > >> > > > >Tuukka
> > > >> > > > >
> > > >> > > > >
> > > >> > > > >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
> > > >> > > > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> > > >> > > > >
> > > >> > > > >> Hello,
> > > >> > > > >> I've created this bugzilla for investigation and to share
> > with
> > > >> you
> > > >> > > > >the plan
> > > >> > > > >> I have used to run my own tests:
> > > >> > > > >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
> > > >> > > > >>
> > > >> > > > >> I use a Tomcat 8.5.11 with a localhost and a certificate
> > > >> generated
> > > >> > > > >and auto
> > > >> > > > >> signed.
> > > >> > > > >>
> > > >> > > > >> So here is what I get on a Mac Book Pro Mid 2015 OSX El
> > Capitan
> > > >> 2.5
> > > >> > > > >Ghz
> > > >> > > > >> Intel Core i7
> > > >> > > > >> Java 8 u121
> > > >> > > > >> -Xmx512m -Xms512m
> > > >> > > > >> Test in GUI mode (ANTI-PATTERN)
> > > >> > > > >>
> > > >> > > > >> 2.13:
> > > >> > > > >> - CPU is at 100%
> > > >> > > > >> - No error
> > > >> > > > >> - No leak
> > > >> > > > >>
> > > >> > > > >> 3.2 nightly with default settings :
> > > >> > > > >> - CPU is at 100%
> > > >> > > > >> - No error
> > > >> > > > >> - No leak
> > > >> > > > >>
> > > >> > > > >> 3.2 nightly with this in user.properties :
> > > >> > > > >> httpclient4.time_to_live=0
> > > >> > > > >> httpclient4.validate_after_inactivity=0
> > > >> > > > >>
> > > >> > > > >> - CPU is at 100%
> > > >> > > > >> - No error
> > > >> > > > >> - No leak
> > > >> > > > >>
> > > >> > > > >> I attached the Heap Graphs to bugzilla and the plan.
> > > >> > > > >>
> > > >> > > > >> So for now I don't  reproduce your issue but I am not in
> the
> > > same
> > > >> > > > >> configuration as I am not hitting the same website.
> > > >> > > > >>
> > > >> > > > >> So
> > > >> > > > >> 0/ Could you provide by running only 5 requests, the
> response
> > > >> header
> > > >> > > > >that
> > > >> > > > >> your website returns ?
> > > >> > > > >> 1/ Could you try my test plan by replacing with your target
> > > host
> > > >> and
> > > >> > > > >see if
> > > >> > > > >> behaviour is the same ?
> > > >> > > > >> 2/ Ensure nothing is running (No Spotify :-) ) except your
> > > JMeter
> > > >> > and
> > > >> > > > >> monitor network as per Felix note
> > > >> > > > >> 3/ Run 1 test after the other but ensure you wait a bit
> > between
> > > >> each
> > > >> > > > >one
> > > >> > > > >>
> > > >> > > > >>
> > > >> > > > >> Thanks
> > > >> > > > >>
> > > >> > > > >>
> > > >> > > > >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
> > > >> > > > >> felix.schumacher@internetallee.de <javascript:;>> wrote:
> > > >> > > > >>
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
> > > >> > > > >> > tuukka.mustonen@gmail.com <javascript:;>>:
> > > >> > > > >> > >Ok I re-ran the test.
> > > >> > > > >> > >
> > > >> > > > >> > >Took Heap Dump just before JMeter froze.
> > > >> > > > >> > >
> > > >> > > > >> > >Took Thread Dumps right after JMeter froze.
> > > >> > > > >> > >
> > > >> > > > >> >
> > > >> > > > >>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqal
> > > >> lGam13MTA
> > > >> > > > >> >
> > > >> > > > >> > That folder seems to be empty now.
> > > >> > > > >> >
> > > >> > > > >> > Are you running the tests with a security manager
> enabled?
> > > >> > > > >> >
> > > >> > > > >> > Can you share the test or at least the used requests?
> > > >> > > > >> >
> > > >> > > > >> > Are you using a proxy and if so, how is it configured?
> > > >> > > > >> >
> > > >> > > > >> > Is jmeter running locally, or over something like rdp?
> > > >> > > > >> >
> > > >> > > > >> > Can you look at the used network connections like netstat
> > > >> -etapn
> > > >> > > > >would do
> > > >> > > > >> > on Linux?
> > > >> > > > >> >
> > > >> > > > >> > Regards,
> > > >> > > > >> >  Felix
> > > >> > > > >> >
> > > >> > > > >> > >
> > > >> > > > >> > >...
> > > >> > > > >> > >
> > > >> > > > >> > >For the fun of it, I also tried upgrading to the latest
> > > >> > HttpClient
> > > >> > > > >> > >4.5.3
> > > >> > > > >> > >version. No effect.
> > > >> > > > >> > >
> > > >> > > > >> > >I also noticed that if I ping (google) during the perf
> > > >> testing,
> > > >> > > > >Windows
> > > >> > > > >> > >ping reports "General failure." when JMeter breaks.
> After
> > > >> killing
> > > >> > > > >it,
> > > >> > > > >> > >connectivity restores and there are replies from the
> > server
> > > >> > again.
> > > >> > > > >It
> > > >> > > > >> > >seems
> > > >> > > > >> > >like JMeter/HttpClient kills my whole network
> > connection...
> > > >> > > > >> > >
> > > >> > > > >> > >Tuukka
> > > >> > > > >> > >
> > > >> > > > >> > >
> > > >> > > > >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > > >> > > > >> > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> > > >> > > > >> > >
> > > >> > > > >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> > > >> > > > >> > ><tuukka.mustonen@gmail.com <javascript:;>
> > > >> > > > >> > >> >
> > > >> > > > >> > >> wrote:
> > > >> > > > >> > >>
> > > >> > > > >> > >> > Hi, I'm running solely on Win 10 now. In
> re-attempts I
> > > >> think
> > > >> > I
> > > >> > > > >saw
> > > >> > > > >> > >> slightly
> > > >> > > > >> > >> > different behavior in my Linux guest. I'll write
> more
> > > >> about
> > > >> > > > >that
> > > >> > > > >> > >later,
> > > >> > > > >> > >> but
> > > >> > > > >> > >> > for simplicity I'm testing only on Windows host now:
> > > >> > > > >> > >> >
> > > >> > > > >> > >> > Thread dumps are downloadable at:
> > > >> > > > >> > >> >
> > > >> > > > >https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEF
> > > >> uSS1iejA
> > > >> > > > >> > >> >
> > > >> > > > >> > >> > (Took them with 7-8s interval I think, hope it's
> ok).
> > > >> > > > >> > >> >
> > > >> > > > >> > >> > I took the thread dumps once JMeter started filling
> in
> > > >> > errors,
> > > >> > > > >> > >which is
> > > >> > > > >> > >> > somewhat after my CPU usage hit 100% (and music
> > playback
> > > >> > > > >started to
> > > >> > > > >> > >> distort
> > > >> > > > >> > >> > etc.).
> > > >> > > > >> > >> >
> > > >> > > > >> > >> > I also created the head dump, note that I took it
> > after
> > > >> > > > >running the
> > > >> > > > >> > >test
> > > >> > > > >> > >> > and after restarting JMeter, as JMeter froze during
> > the
> > > >> test
> > > >> > > > >and I
> > > >> > > > >> > >had to
> > > >> > > > >> > >> > kill it. DId you mean I should take it *during* the
> > > test?
> > > >> > > > >> > >>
> > > >> > > > >> > >>
> > > >> > > > >> > >> YES.
> > > >> > > > >> > >>
> > > >> > > > >> > >>
> > > >> > > > >> > >> > Anyway, before
> > > >> > > > >> > >> > sharing it I want to make sure it can't contain
> > anything
> > > >> > > > >> > >confidential?
> > > >> > > > >> > >> > (parts of my test plan, passwords, environment
> > > variables,
> > > >> > > > >> > >anything...)
> > > >> > > > >> > >> >
> > > >> > > > >> > >> > Tuukka
> > > >> > > > >> > >> >
> > > >> > > > >> > >> >
> > > >> > > > >> > >> >
> > > >> > > > >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK
> > Support
> > > <
> > > >> > > > >> > >> > support@ubikloadpack.com <javascript:;>> wrote:
> > > >> > > > >> > >> >
> > > >> > > > >> > >> > > Hello,
> > > >> > > > >> > >> > > To investigate the problem:
> > > >> > > > >> > >> > > 1/ to create  thread dumps, install a JDK of same
> > > >> version
> > > >> > as
> > > >> > > > >the
> > > >> > > > >> > >one
> > > >> > > > >> > >> > > running your JMeter.
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > > Run until you get the problem, then call using
> same
> > > >> user as
> > > >> > > > >the
> > > >> > > > >> > >one
> > > >> > > > >> > >> > running
> > > >> > > > >> > >> > > JMeter 3 times at 5s interval:
> > > >> > > > >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >>
> > > >> > output_file.txt
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > > It will also help if you provide the structure of
> > your
> > > >> Test
> > > >> > > > >Plan.
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > > 2/ It would also help if you can provide a Heap
> > Dump.
> > > >> With
> > > >> > > > >JMeter
> > > >> > > > >> > >GUI,
> > > >> > > > >> > >> > use
> > > >> > > > >> > >> > > menu Help > Create a Heap Dump
> > > >> > > > >> > >> > > This will create a Heap Dump (rather big file) ,
> it
> > > >> would
> > > >> > > > >help a
> > > >> > > > >> > >lot if
> > > >> > > > >> > >> > you
> > > >> > > > >> > >> > > can make it available.
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > > Regards
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > > >> > > > >> > >> > tuukka.mustonen@gmail.com <javascript:;>
> > > >> > > > >> > >> > > >
> > > >> > > > >> > >> > > wrote:
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> > > > @Philippe:
> > > >> > > > >> > >> > > >
> > > >> > > > >> > >> > > > Yeah, I am normally running in non-GUI mode, but
> > to
> > > >> > > > >> > >> > > > study/inspect/understand this problem I am
> running
> > > in
> > > >> GUI
> > > >> > > > >mode.
> > > >> > > > >> > >> JMeter
> > > >> > > > >> > >> > > 2.13
> > > >> > > > >> > >> > > > works just fine in GUI mode for these tests so
> > > JMeter
> > > >> 3.x
> > > >> > > > >> > >should too.
> > > >> > > > >> > >> > > >
> > > >> > > > >> > >> > > > I'll give the nightly build a try.
> > > >> > > > >> > >> > > >
> > > >> > > > >> > >> > > > Tuukka
> > > >> > > > >> > >> > > >
> > > >> > > > >> > >> > > >
> > > >> > > > >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe
> Mouawad <
> > > >> > > > >> > >> > > > philippe.mouawad@gmail.com <javascript:;>
> > > >> > > > >> > >> > > > > wrote:
> > > >> > > > >> > >> > > >
> > > >> > > > >> > >> > > > > On Wednesday, February 1, 2017, Tuukka
> Mustonen
> > <
> > > >> > > > >> > >> > > > tuukka.mustonen@gmail.com <javascript:;>
> > > >> > > > >> > >> > > > > >
> > > >> > > > >> > >> > > > > wrote:
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > > > @Philippe:
> > > >> > > > >> > >> > > > > >
> > > >> > > > >> > >> > > > > > 1. Yeah, I compared them with the same plan,
> > > same
> > > >> > > > >machine.
> > > >> > > > >> > >Should
> > > >> > > > >> > >> > be
> > > >> > > > >> > >> > > > > > everything equal.
> > > >> > > > >> > >> > > > > > 2. I tried both the defaults in jmeter.bat
> > > >> (-Xms512m
> > > >> > > > >> > >-Xmx512m)
> > > >> > > > >> > >> and
> > > >> > > > >> > >> > as
> > > >> > > > >> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I
> > > didn't
> > > >> > > > >tweak
> > > >> > > > >> > >them much
> > > >> > > > >> > >> > > > > because I
> > > >> > > > >> > >> > > > > > don't believe the issue lies there.
> > > >> > > > >> > >> > > > > > 3. No, I do just simple HTTP GETs (against
> > RESTy
> > > >> > API).
> > > >> > > > >> > >> > > > > > 4. I am not familiar with that but I can
> try.
> > If
> > > >> you
> > > >> > > > >have
> > > >> > > > >> > >> > > instructions
> > > >> > > > >> > >> > > > to
> > > >> > > > >> > >> > > > > > point to, please do.
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > > > 5. I am running with GUI.
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > > This is a known anti-pattern .
> > > >> > > > >> > >> > > > > switch to non gui mode if you want correct
> > > results.
> > > >> > > > >> > >> > > > > http://jmeter.apache.org/
> > > >> > usermanual/best-practices.html
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > > http://www.ubik-ingenierie.
> > > >> > com/blog/jmeter_performance_
> > > >> > > > >> > >> tuning_tips/
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > > > About file sending:
> > > >> > > > >> > >> > > > > >
> > > >> > > > >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > >  Could you give nightly build a try ?
> > > >> > > > >> > >> > > > > http://jmeter.apache.org/nightly.html
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > > > Tuukka
> > > >> > > > >> > >> > > > > >
> > > >> > > > >> > >> > > > > >
> > > >> > > > >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe
> > > Mouawad <
> > > >> > > > >> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> > > >> > > <javascript:;>
> > > >> > > > >> > >> > > > > > > wrote:
> > > >> > > > >> > >> > > > > >
> > > >> > > > >> > >> > > > > > > Also are you sending files in post body ?
> > > >> > > > >> > >> > > > > > >
> > > >> > > > >> > >> > > > > > > Thank you
> > > >> > > > >> > >> > > > > > >
> > > >> > > > >> > >> > > > > > > On Wednesday, February 1, 2017, Philippe
> > > >> Mouawad <
> > > >> > > > >> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>
> > > >> > > <javascript:;>>
> > > >> > > > >> > >> > > > > > > wrote:
> > > >> > > > >> > >> > > > > > >
> > > >> > > > >> > >> > > > > > > > Hello,
> > > >> > > > >> > >> > > > > > > > Few questions, please try to answer them
> > > all:
> > > >> > > > >> > >> > > > > > > >
> > > >> > > > >> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with
> same
> > > >> > > > >plan/sape
> > > >> > > > >> > >> machine...
> > > >> > > > >> > >> > > > > > everything
> > > >> > > > >> > >> > > > > > > > equal ?
> > > >> > > > >> > >> > > > > > > > 2/ What are your Xmx settings
> > > >> > > > >> > >> > > > > > > > 3/ does your plan download embedded
> > > resources
> > > >> ?
> > > >> > > > >> > >> > > > > > > > 4/ when issue happens can you make 3
> > thread
> > > >> dumps
> > > >> > > > >at 5
> > > >> > > > >> > >> seconds
> > > >> > > > >> > >> > > > > interval
> > > >> > > > >> > >> > > > > > > > and send file ? (use jstack)
> > > >> > > > >> > >> > > > > > > > 5/ are you running gui or non gui ?
> > > >> > > > >> > >> > > > > > > >
> > > >> > > > >> > >> > > > > > > > Thanks
> > > >> > > > >> > >> > > > > > > > Regards
> > > >> > > > >> > >> > > > > > > >
> > > >> > > > >> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka
> > > >> Mustonen <
> > > >> > > > >> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > >> > > <javascript:;>
> > > >> > > > >> > >> > > > > > > >
> > > >> > > > ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > > >> > <javascript:;>
> > > >> > > > >> > >> > > > > <javascript:;>');>>
> > > >> > > > >> > >> > > > > > wrote:
> > > >> > > > >> > >> > > > > > > >
> > > >> > > > >> > >> > > > > > > >> Hi,
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I
> > > >> > experience
> > > >> > > > >high
> > > >> > > > >> > >CPU
> > > >> > > > >> > >> > usage
> > > >> > > > >> > >> > > > > that
> > > >> > > > >> > >> > > > > > > >> renders JMeter unusable. CPU usage
> ramps
> > > >> quickly
> > > >> > > > >up
> > > >> > > > >> > >and
> > > >> > > > >> > >> after
> > > >> > > > >> > >> > > > having
> > > >> > > > >> > >> > > > > > > >> ~100-200 concurrent users or so (in ~30
> > > >> > seconds),
> > > >> > > > >it
> > > >> > > > >> > >hits
> > > >> > > > >> > >> 100%
> > > >> > > > >> > >> > > on
> > > >> > > > >> > >> > > > my
> > > >> > > > >> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter
> > > freezes
> > > >> and
> > > >> > > > >I
> > > >> > > > >> > >have to
> > > >> > > > >> > >> > kill
> > > >> > > > >> > >> > > > it.
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> On 2.13 everything goes smoothly (with
> > > >> default
> > > >> > > > >> > >settings).
> > > >> > > > >> > >> CPU
> > > >> > > > >> > >> > > > usage
> > > >> > > > >> > >> > > > > > > stays
> > > >> > > > >> > >> > > > > > > >> at ~30% even with 500 users.
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> The experience is similar on both
> Windows
> > > 10
> > > >> and
> > > >> > > > >Linux
> > > >> > > > >> > >> (Ubuntu
> > > >> > > > >> > >> > > > 14.04
> > > >> > > > >> > >> > > > > > LTS
> > > >> > > > >> > >> > > > > > > >> guest in Virtualbox).
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> The problems I saw in JMeter 3:
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> - java.net.SocketException: Permission
> > > >> denied:
> > > >> > > > >connect
> > > >> > > > >> > >> > > > > > > >> - After a while it runs out of heap
> space
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> For "Permission denied: connect" I
> tried
> > to
> > > >> set
> > > >> > > > >> > >> > > > > > > >> java.net.preferIPv4Stack=true
> > > >> > > > >> > >> > > > > > > >> (in a few places to make sure it would
> > > >> really be
> > > >> > > > >in
> > > >> > > > >> > >use) and
> > > >> > > > >> > >> > > tried
> > > >> > > > >> > >> > > > > to
> > > >> > > > >> > >> > > > > > > run
> > > >> > > > >> > >> > > > > > > >> JMeter as administrator (in Win). No
> > > effect.
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> I also tried increasing heap size, but
> > that
> > > >> > > > >didn't
> > > >> > > > >> > >help
> > > >> > > > >> > >> > either.
> > > >> > > > >> > >> > > > And
> > > >> > > > >> > >> > > > > I
> > > >> > > > >> > >> > > > > > > >> think
> > > >> > > > >> > >> > > > > > > >> memory size is not the root cause here
> -
> > > >> rather
> > > >> > > > >the
> > > >> > > > >> > >result.
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> I am normally connected to VPN but I
> also
> > > >> tried
> > > >> > > > >> > >disabling
> > > >> > > > >> > >> it.
> > > >> > > > >> > >> > No
> > > >> > > > >> > >> > > > > > effect.
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> I tried disabling with and without IPv6
> > > >> enabled
> > > >> > > > >in
> > > >> > > > >> > >Windows
> > > >> > > > >> > >> > > network
> > > >> > > > >> > >> > > > > > > >> connection settings. No effect.
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> I am running a few extra JMeter
> plugins.
> > > >> Tried
> > > >> > > > >cutting
> > > >> > > > >> > >my
> > > >> > > > >> > >> test
> > > >> > > > >> > >> > > > > > scenario
> > > >> > > > >> > >> > > > > > > to
> > > >> > > > >> > >> > > > > > > >> minimum. No effect.
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> I am running anti-virus software that I
> > > >> cannot
> > > >> > > > >> > >disable, due
> > > >> > > > >> > >> to
> > > >> > > > >> > >> > > > this
> > > >> > > > >> > >> > > > > > > being
> > > >> > > > >> > >> > > > > > > >> forced by IT department. However, it's
> > not
> > > a
> > > >> > > > >problem
> > > >> > > > >> > >with
> > > >> > > > >> > >> 2.13
> > > >> > > > >> > >> > > so
> > > >> > > > >> > >> > > > I
> > > >> > > > >> > >> > > > > > > assume
> > > >> > > > >> > >> > > > > > > >> it shouldn't be a problem here either.
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> FINALLY, once I set HTTP request
> > > >> implementation
> > > >> > > > >from
> > > >> > > > >> > >Apache
> > > >> > > > >> > >> > > > > > HttpClient 4
> > > >> > > > >> > >> > > > > > > >> to
> > > >> > > > >> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem
> > > vanishes
> > > >> -
> > > >> > > > >CPU
> > > >> > > > >> > >stays
> > > >> > > > >> > >> where
> > > >> > > > >> > >> > > it
> > > >> > > > >> > >> > > > > did
> > > >> > > > >> > >> > > > > > > with
> > > >> > > > >> > >> > > > > > > >> JMeter 2.13.
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on
> > > Win,
> > > >> > > > >update
> > > >> > > > >> > >111 on
> > > >> > > > >> > >> > > Linux).
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> Why does (the recommended) HttpClient 4
> > > cause
> > > >> > > > >problems
> > > >> > > > >> > >here?
> > > >> > > > >> > >> > Is
> > > >> > > > >> > >> > > > > this a
> > > >> > > > >> > >> > > > > > > >> know
> > > >> > > > >> > >> > > > > > > >> bug/issue? JMeter 3.x is running newer
> > > >> version
> > > >> > > > >> > >compared to
> > > >> > > > >> > >> > 2.13,
> > > >> > > > >> > >> > > > but
> > > >> > > > >> > >> > > > > > > what
> > > >> > > > >> > >> > > > > > > >> change in HttpClient 4 might cause the
> > > >> problems
> > > >> > > > >> > >described?
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> Any ideas?
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >> Regards,
> > > >> > > > >> > >> > > > > > > >> Tuukka
> > > >> > > > >> > >> > > > > > > >>
> > > >> > > > >> > >> > > > > > > >
> > > >> > > > >> > >> > > > > > > >
> > > >> > > > >> > >> > > > > > > > --
> > > >> > > > >> > >> > > > > > > > Cordialement.
> > > >> > > > >> > >> > > > > > > > Philippe Mouawad.
> > > >> > > > >> > >> > > > > > > >
> > > >> > > > >> > >> > > > > > > >
> > > >> > > > >> > >> > > > > > > >
> > > >> > > > >> > >> > > > > > > >
> > > >> > > > >> > >> > > > > > >
> > > >> > > > >> > >> > > > > > > --
> > > >> > > > >> > >> > > > > > > Cordialement.
> > > >> > > > >> > >> > > > > > > Philippe Mouawad.
> > > >> > > > >> > >> > > > > > >
> > > >> > > > >> > >> > > > > >
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > > > --
> > > >> > > > >> > >> > > > > Cordialement.
> > > >> > > > >> > >> > > > > Philippe Mouawad.
> > > >> > > > >> > >> > > > >
> > > >> > > > >> > >> > > >
> > > >> > > > >> > >> > >
> > > >> > > > >> > >> >
> > > >> > > > >> > >>
> > > >> > > > >> > >>
> > > >> > > > >> > >>
> > > >> > > > >> > >> --
> > > >> > > > >> > >> Cordialement.
> > > >> > > > >> > >> Philippe Mouawad.
> > > >> > > > >> > >>
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >-----------------------------------------------------------
> > > >> ----------
> > > >> > > > >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.
> apache.org
> > > >> > > <javascript:;>
> > > >> > > > >> > For additional commands, e-mail:
> > user-help@jmeter.apache.org
> > > >> > > <javascript:;>
> > > >> > > > >> >
> > > >> > > > >> >
> > > >> > > > >>
> > > >> > > > >>
> > > >> > > > >> --
> > > >> > > > >> Cordialement.
> > > >> > > > >> Philippe Mouawad.
> > > >> > > > >>
> > > >> > > >
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Cordialement.
> > > >> > Philippe Mouawad.
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Cordialement.
> > > > Philippe Mouawad.
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > Cordialement.
> > > Philippe Mouawad.
> > >
> >
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

On Sat, Feb 4, 2017 at 8:05 PM, Tuukka Mustonen <tu...@gmail.com>
wrote:

> > In the case of NGinx , it seems no header timeout is sent.
> > So if you set a value > to 0, you can control the max idle duration of a
> > connection from Client (JMeter) side to avoid storing too many
> connections
> > that won't be reused and can impact JMeter heap.
> > By default it's not set so no impact.
>
> Hmm, what do you mean by "header timeout"? Afaik, the server cannot
> communicate a timeout for the keep-alived connection to the client. The
> server probably has some timeout (I think in nginx it's 75s by default) but
> the client doesn't know it. Server will just close the connection at some
> point.
>
https://tools.ietf.org/id/draft-thomson-hybi-http-timeout-01.html#rfc.section.2.1
But server is allowed not to send it.
But I think for a load test, it is better to adjust the 3 parameters to a
value that is similar to the server keep-alive timeout



> Also I'm not 100% sure what you mean by "to avoid storing too many
> connections that won't be reused"? The client will re-use connection until
> it's closed by TTL configuration or if the server closes the connection,
> right?


I mean if server does not send timeout, but timeouts connection on its
side, JMeter might
try to reuse a "stale" connection which has a cost and will only drop it on
next use.
For high throughput tests it migh be a problem.



> I though with idletimeout I could just make sure that client closes
> the connection before the server would (and avoid any potential failed
> requests if client sent a request just while the server closes connection)?
>
You understood correctly

>
> I'm not sure, the documentation for that parameter could maybe be improved
> a tiny bit?
>

If you want to contribute a patch for that you're welcome. It's in
properties_reference.xml.
http://jmeter.apache.org/building.html
Otherwise we'll try to fix it.

>
> > But investigating your issue, I discovered that in fact
> > "httpclient4.validate_after_inactivity" has no impact due to another bug
> > here I am also responsible of :-( .
> > validate() method always return true so JMeter always consider connection
> > as valid.
> > So only "httpclient4.time_to_live" does its job here.
>
> You seem to be author of lots of commits - it's only human to forget
> something every now and then :)
>
> > What's happening here is that you're most probably saturating the dynamic
> > port range
>
> If you mean that I would be going over the number of available dynamic
> ports, I don't think that is it. The test breaks at ~40-50 seconds or so
> and at that point there shouldn't be even 10k of connections (potentially)
> open. But, I'm definitively hitting some limit. I will contact our IT
> department - who knows what magic configuration we have.
>

https://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx

>
> > It's because TTL is > 0, so it will be included in the management of idle
> > connections.
> > ...
> > Because JMeter will close connection immediately and inform server to do
> > the same.
>
> Ah, good point. What's the policy (=timeline) for closing the idle
> connections that go over TTL?
>
It will be closed when a new request is made by same user and it has
expired.

>
> > Thanks a lot for all your tests which helped us narrow this issue.
> > Besides Vladimir and Felix are working on improving performances in GUI
> > Mode but the team still advises against using it for Load Testing.
>
> Cool. Yeah, I run tests normally from Jenkins in command line mode, but I
> always develop/debug in GUI (as does everybody, I would believe). When in
> GUI mode, I don't care about performance really - just that the tests pass.
> But better performance is welcome of course.
>
> > If you used the nightly build (select Jenkins one) you should be able to
> > see that now :
> > httpclient4.validate_after_inactivity = 1700
> > and has an effect.
>
> Thanks for the quick fixes! I think I will test it next week some point,
> just for the curiosity, to see it in effect in Wireshark. I assume this
> check sends some dummy package to the server and should be visible in the
> trace. Correct me if wrong.
>
> Thanks. Your feedback is very welcome.

Tuukka
>
>
>
> On Sat, Feb 4, 2017 at 5:13 PM, Philippe Mouawad <
> philippe.mouawad@gmail.com
> > wrote:
>
> > Hello Tukka,
> > I have commited :
> > https://bz.apache.org/bugzilla/show_bug.cgi?id=60690
> > https://bz.apache.org/bugzilla/show_bug.cgi?id=60689
> >
> > If you used the nightly build (select Jenkins one) you should be able to
> > see that now :
> > httpclient4.validate_after_inactivity = 1700
> > and has an effect.
> >
> > Still in your particular case as you've tested you'll need to increase
> > httpclient4.time_to_live to avoid too much connections creations that
> your
> > OS/Machine/Network does not correctly handle.
> >
> > But beware that this may not be realistic as if you're server calls are
> > made from clients that do not use keep alive or persist their connections
> > for a short time.
> > To ensure your server correctly handles this , you should test from a
> fine
> > tuned machine that has a correct network with target server.
> >
> >
> > Regards
> > @philmdot
> >
> > On Sat, Feb 4, 2017 at 3:16 PM, Philippe Mouawad <
> > philippe.mouawad@gmail.com
> > > wrote:
> >
> > >
> > >
> > > On Sat, Feb 4, 2017 at 12:58 PM, Tuukka Mustonen <
> > > tuukka.mustonen@gmail.com> wrote:
> > >
> > >> On point, Philippe!
> > >>
> > >> I think *httpclient4.idletimeout* is not of relevance here - my server
> > >> sends proper Keep-Alive headers so this option shouldn't have impact.
> > Good
> > >> to know, for sure.
> > >>
> > > In the case of NGinx , it seems no header timeout is sent.
> > > So if you set a value > to 0, you can control the max idle duration of
> a
> > > connection from Client (JMeter) side to avoid storing too many
> > connections
> > > that won't be reused and can impact JMeter heap.
> > > By default it's not set so no impact.
> > >
> > >>
> > >> *httpclient4.validate_after_inactivity* and *httpclient4.time_to_live
> > >> default* values (2 seconds) sound conflicting: TTL being the same as
> > >> validate_after_inactivity leads to connections never getting checked
> as
> > >> connections are discarded after 2 seconds anyway? So
> > >> validate_after_inactivity is not really in use by default if I
> > understood
> > >> it right.
> > >>
> > > Good catch. It somehow went out of my radar, I remember it was
> something
> > I
> > > had in mind but it seems I failed to change it.
> > > It's a bug or at least bad default value.
> > >
> > > But investigating your issue, I discovered that in fact
> > > "httpclient4.validate_after_inactivity" has no impact due to another
> bug
> > > here I am also responsible of :-( .
> > > validate() method always return true so JMeter always consider
> connection
> > > as valid.
> > > So only "httpclient4.time_to_live" does its job here.
> > >
> > >
> > >>
> > >> So I went on and tried these settings:
> > >>
> > >> 1) httpclient4.validate_after_inactivity=0 +
> > >> httpclient4.time_to_live=2000
> > >> -> CRASH
> > >> 2) httpclient4.validate_after_inactivity=0 +
> > >> httpclient4.time_to_live=1000
> > >> -> CRASH
> > >> 3) httpclient4.validate_after_inactivity=1 +
> > >> httpclient4.time_to_live=2000
> > >> -> CRASH
> > >>
> > >> 4) httpclient4.validate_after_inactivity=5000 +
> > >> httpclient4.time_to_live=10000 -> SUCCESS
> > >> 5) httpclient4.validate_after_inactivity=0 +
> > >> httpclient4.time_to_live=10000
> > >> -> SUCCESS
> > >> 6) httpclient4.validate_after_inactivity=1 +
> > >> httpclient4.time_to_live=10000
> > >> -> SUCCESS
> > >> 7) httpclient4.validate_after_inactivity=0 +
> httpclient4.time_to_live=0
> > >> ->
> > >> SUCCESS
> > >> 8) httpclient4.validate_after_inactivity=10000 +
> > >> httpclient4.time_to_live=10000 -> SUCCESS
> > >>
> > >> Clearly increasing TTL makes the difference for me.
> > >> validate_after_inactivity seems to have no effect. Although there is
> > some
> > >> packet loss, the connections themselves don't break.
> > >>
> > > Absolutely.
> > >
> > >>
> > >> A) So... maybe my network chip configuration just cannot tolerate
> JMeter
> > >> opening fresh connections every 2 seconds? That translates to 250
> > >> connections/s with 500 users- maybe it's enough to bloat my network IO
> > and
> > >> hit some policy (and I'll then get "connect: Permission denied")?
> > Sounds a
> > >> bit weird.
> > >>
> > >
> > > It's also your OS, you need to tune the TCP stack to work with port
> reuse
> > > speed.
> > > What's happening here is that you're most probably saturating the
> dynamic
> > > port range
> > >
> > >>
> > >> So I also tried:
> > >>
> > >> 9) httpclient4.validate_after_inactivity=0 +
> httpclient4.time_to_live=1
> > >> ->
> > >> CRASH
> > >>
> > >> And it crashes quite instantly. I guess it's because JMeter is
> opening a
> > >> new connection per request...
> > >>
> > > It's because TTL is > 0, so it will be included in the management of
> idle
> > > connections.
> > >
> > >>
> > >> B) ...but if I keep the default 2s settings and just disable
> Keep-Alive,
> > >> then JMeter is opening new connection per request also, and it works.
> > Why
> > >> the difference?
> > >>
> > > Because JMeter will close connection immediately and inform server to
> do
> > > the same.
> > >
> > >
> > >>
> > >> C) Then, I wonder if your "standard workstation" is supposed to
> tolerate
> > >> these defaults? Can my network chip/driver/configuration just be so
> bad?
> > >>
> > >
> > > Yes at least your OS Tuning defaults are not well suited for Load
> > Testing,
> > > you need to tune.
> > >
> > >>
> > >> I see the *httpclient4.validate_after_inactivity* and
> > >> *httpclient4.time_to_live
> > >> default* properties mentioned in the JMeter 3 changelog:
> > >>
> > >> "Improved and better performing validation mechanism for Stale
> > connections
> > >> and Keep-Alive management, see properties
> > >> httpclient4.validate_after_inactivity and httpclient4.time_to_live"
> > >>
> > >
> > > Yes, it was the intention.
> > >
> > >>
> > >> And when I monitored TCP connection in Wireshark, JMeter 2.13 indeed
> > keeps
> > >> connections open waaaay longer (seemed a bit random).
> > >>
> > >
> > > Yes, at that time the only solution was to play with idletimeout or set
> > > "http.connection.stalecheck$Boolean=true" which had important impact
> at
> > > high throughput.
> > >
> > >>
> > >> D) So I the new 2s default sounds a bit risky?
> > >>
> > > In your case yes, but you're in a "bad load testing config" :-)
> > >
> > >>
> > >> In the end, increasing TTL seems to fix the issues for me and I can
> > >> proceed
> > >> upgrading to JMeter 3.
> > >>
> > >> Would love to hear your opinion/analysis on the open questions (A-D),
> > but
> > >> big thanks for your interest in my problem and the precious help
> already
> > >> at
> > >> this point!
> > >>
> > >
> > > Thanks a lot for all your tests which helped us narrow this issue.
> > > Besides Vladimir and Felix are working on improving performances in GUI
> > > Mode but the team still advises against using it for Load Testing.
> > >
> > >
> > >> Tuukka
> > >>
> > >>
> > >>
> > >> On Fri, Feb 3, 2017 at 9:40 PM, Philippe Mouawad <
> > >> philippe.mouawad@gmail.com
> > >> > wrote:
> > >>
> > >> > On Friday, February 3, 2017, Tuukka Mustonen <
> > tuukka.mustonen@gmail.com
> > >> >
> > >> > wrote:
> > >> >
> > >> > > Sorry for the delay, got too busy yesterday.
> > >> > >
> > >> > > I have been running more tests now. I tried to a few variations to
> > >> > pinpoint
> > >> > > the issue, but first:
> > >> > >
> > >> > > @Philippe:
> > >> > >
> > >> > > 0) Response headers are like:
> > >> > >
> > >> > > Response headers:
> > >> > > HTTP/1.1 200 OK
> > >> > > Date: Fri, 03 Feb 2017 06:47:00 GMT
> > >> > > Content-Type: application/json
> > >> > > Content-Length: 374
> > >> > > Connection: keep-alive
> > >> > > Server: nginx/1.6.2
> > >> > > ETag: 37926:1-37927:1
> > >> > > Cache-Control: no-cache, no-store
> > >> > > Pragma: no-cache
> > >> > > Expires: 0
> > >> > > X-Transaction: 0000-326srxia31tss3tc
> > >> >
> > >> >
> > >> > As you pointed it later that issue is related to keep alive.
> > >> > This is tunable , I 'll try to provide some advices
> > >> >
> > >> > But read:
> > >> > http://jmeter.apache.org/usermanual/properties_reference.
> > >> html#httpclient4
> > >> >
> > >> > And pay attention to:
> > >> >
> > >> > httpclient4.idletimeout
> > >> > httpclient4.validate_after_inactivity
> > >> > httpclient4.time_to_live
> > >> >
> > >> > It appears Default values do not suite your case.
> > >> >
> > >> >
> > >> > > 1) Tried your test plan. Unfortunately, it does not make
> difference.
> > >> >
> > >> > it was not made to fix issue only to make it simpler and indépendant
> > of
> > >> any
> > >> > 3rd party plugin
> > >> >
> > >> > >
> > >> > > 2) I ran Wireshark during a couple of tests. You can find a few
> > >> > screenshots
> > >> > > in a later link. Note that my whole network kinda crashes when
> > JMeter
> > >> 3
> > >> > > gets stuck (ping gets "General failure" and some active
> connections
> > >> > crash,
> > >> > > note that this seems a bit random).
> > >> >
> > >> > Thanks
> > >> >
> > >> > >
> > >> > > 3) I gave it a few minutes to recover between the runs.
> > >> > >
> > >> > > @Felix:
> > >> > >
> > >> > > I ran all tests with loggerpanel.usejsyntaxtext=false.
> > >> > >
> > >> > > No, the amount of seems roughly the same.
> > >> > >
> > >> > >
> > >> > >
> > >> > > Then, about pinpointing the issue:
> > >> > >
> > >> > > I managed to reproduce this by installing nginx on a network
> > computer
> > >> > (sudo
> > >> > > apt-get install nginx on debian 8). I did not configure anything
> > and I
> > >> > > simply bombarded the default nginx welcome page.
> > >> > >
> > >> > > I have attached Heap Dump, Thread Dumps and some other information
> > in
> > >> > these
> > >> > > links:
> > >> > >
> > >> > > JAVA impl (that doesn't crash):
> > >> > > https://drive.google.com/drive/folders/
> 0B1uBdVuLED5NSFJvc2VSdE13a0k
> > ?
> > >> > > usp=sharing
> > >> > >
> > >> > > HttpCilent 4 impl (that crashes):
> > >> > > https://drive.google.com/drive/folders/
> 0B1uBdVuLED5NUGc1WWFYYkRncTQ
> > ?
> > >> > > usp=sharing
> > >> > >
> > >> > > I used Philippe's test plan and even simplified it a bit, the JMX
> > >> file is
> > >> > > in the latter link. Note that the errors occur (in my case) during
> > the
> > >> > > ramp-up so I also shortened to run just 30s after reaching 500
> > users.
> > >> > >
> > >> > > With JAVA impl, I ran heap dump at ~450 users or so, and thread
> > dumps
> > >> > after
> > >> > > that.
> > >> > >
> > >> > > With HttpClient 4 impl, I ran heap dump immediately after seeing
> the
> > >> > first
> > >> > > error (which was at ~430 users or so) and thread dumps right after
> > >> that
> > >> > > (the first two dumps I think took a bit longer than normally to
> > >> create).
> > >> > > JMeter goes unresponsive a second or two after seeing the first
> > error,
> > >> > so I
> > >> > > just barely had time to click to take the dump.
> > >> > >
> > >> > > As said, there are some screenshots from Wireshark trace for
> > >> HttpClient 4
> > >> > > run. I am not network expert, so I don't dare to judge them, but I
> > >> > included
> > >> > > what I saw interesting. I think I can share some parts (privately)
> > >> should
> > >> > > you want to study the trace in detail.
> > >> > >
> > >> > > Finally, when I disabled keep-alive from nginx configuration, the
> > >> problem
> > >> > > was gone (as earlier). So, the problem is somehow related to
> > >> keep-alive.
> > >> > >
> > >> > > Note that in all scenarios CPU goes to 100% quite immediately
> (also
> > on
> > >> > 2.13
> > >> > > now, probably because of high throughput, I assume my workstation
> is
> > >> the
> > >> > > bottleneck here) but without keep-alive or with JAVA impl, things
> > work
> > >> > and
> > >> > > with keep-alive enabled and HttpClient 4, things break apart. As
> > >> before,
> > >> > on
> > >> > > JMeter 2.13 everything works.
> > >> > >
> > >> > > Tuukka
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <
> > >> > > felix.schumacher@internetallee.de <javascript:;>> wrote:
> > >> > >
> > >> > > >
> > >> > > >
> > >> > > > Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
> > >> > > > tuukka.mustonen@gmail.com <javascript:;>>:
> > >> > > > >@Philippe:
> > >> > > > >
> > >> > > > >0. Alright, I will.
> > >> > > > >1. Will do.
> > >> > > > >2. Sure.
> > >> > > > >3. To let the network connection state restore? Ok, will do.
> > >> > > > >
> > >> > > > >I am having some other stuff I need to attend today, but I will
> > get
> > >> > > > >back to
> > >> > > > >this in the afternoon at latest.
> > >> > > >
> > >> > > > Could you also try to set loggerpanel.usejsyntaxtext to false in
> > >> your
> > >> > > > settings?
> > >> > > >
> > >> > > > Are there more log entries, when you use httpclient 4 instead of
> > >> Java?
> > >> > > >
> > >> > > > Vladimir has found signs for a event congestion in the awt event
> > >> queue,
> > >> > > > that show in your threaddumps.
> > >> > > >
> > >> > > > Regards,
> > >> > > >  Felix
> > >> > > >
> > >> > > > >
> > >> > > > >Tuukka
> > >> > > > >
> > >> > > > >
> > >> > > > >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
> > >> > > > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> > >> > > > >
> > >> > > > >> Hello,
> > >> > > > >> I've created this bugzilla for investigation and to share
> with
> > >> you
> > >> > > > >the plan
> > >> > > > >> I have used to run my own tests:
> > >> > > > >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
> > >> > > > >>
> > >> > > > >> I use a Tomcat 8.5.11 with a localhost and a certificate
> > >> generated
> > >> > > > >and auto
> > >> > > > >> signed.
> > >> > > > >>
> > >> > > > >> So here is what I get on a Mac Book Pro Mid 2015 OSX El
> Capitan
> > >> 2.5
> > >> > > > >Ghz
> > >> > > > >> Intel Core i7
> > >> > > > >> Java 8 u121
> > >> > > > >> -Xmx512m -Xms512m
> > >> > > > >> Test in GUI mode (ANTI-PATTERN)
> > >> > > > >>
> > >> > > > >> 2.13:
> > >> > > > >> - CPU is at 100%
> > >> > > > >> - No error
> > >> > > > >> - No leak
> > >> > > > >>
> > >> > > > >> 3.2 nightly with default settings :
> > >> > > > >> - CPU is at 100%
> > >> > > > >> - No error
> > >> > > > >> - No leak
> > >> > > > >>
> > >> > > > >> 3.2 nightly with this in user.properties :
> > >> > > > >> httpclient4.time_to_live=0
> > >> > > > >> httpclient4.validate_after_inactivity=0
> > >> > > > >>
> > >> > > > >> - CPU is at 100%
> > >> > > > >> - No error
> > >> > > > >> - No leak
> > >> > > > >>
> > >> > > > >> I attached the Heap Graphs to bugzilla and the plan.
> > >> > > > >>
> > >> > > > >> So for now I don't  reproduce your issue but I am not in the
> > same
> > >> > > > >> configuration as I am not hitting the same website.
> > >> > > > >>
> > >> > > > >> So
> > >> > > > >> 0/ Could you provide by running only 5 requests, the response
> > >> header
> > >> > > > >that
> > >> > > > >> your website returns ?
> > >> > > > >> 1/ Could you try my test plan by replacing with your target
> > host
> > >> and
> > >> > > > >see if
> > >> > > > >> behaviour is the same ?
> > >> > > > >> 2/ Ensure nothing is running (No Spotify :-) ) except your
> > JMeter
> > >> > and
> > >> > > > >> monitor network as per Felix note
> > >> > > > >> 3/ Run 1 test after the other but ensure you wait a bit
> between
> > >> each
> > >> > > > >one
> > >> > > > >>
> > >> > > > >>
> > >> > > > >> Thanks
> > >> > > > >>
> > >> > > > >>
> > >> > > > >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
> > >> > > > >> felix.schumacher@internetallee.de <javascript:;>> wrote:
> > >> > > > >>
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
> > >> > > > >> > tuukka.mustonen@gmail.com <javascript:;>>:
> > >> > > > >> > >Ok I re-ran the test.
> > >> > > > >> > >
> > >> > > > >> > >Took Heap Dump just before JMeter froze.
> > >> > > > >> > >
> > >> > > > >> > >Took Thread Dumps right after JMeter froze.
> > >> > > > >> > >
> > >> > > > >> >
> > >> > > > >>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqal
> > >> lGam13MTA
> > >> > > > >> >
> > >> > > > >> > That folder seems to be empty now.
> > >> > > > >> >
> > >> > > > >> > Are you running the tests with a security manager enabled?
> > >> > > > >> >
> > >> > > > >> > Can you share the test or at least the used requests?
> > >> > > > >> >
> > >> > > > >> > Are you using a proxy and if so, how is it configured?
> > >> > > > >> >
> > >> > > > >> > Is jmeter running locally, or over something like rdp?
> > >> > > > >> >
> > >> > > > >> > Can you look at the used network connections like netstat
> > >> -etapn
> > >> > > > >would do
> > >> > > > >> > on Linux?
> > >> > > > >> >
> > >> > > > >> > Regards,
> > >> > > > >> >  Felix
> > >> > > > >> >
> > >> > > > >> > >
> > >> > > > >> > >...
> > >> > > > >> > >
> > >> > > > >> > >For the fun of it, I also tried upgrading to the latest
> > >> > HttpClient
> > >> > > > >> > >4.5.3
> > >> > > > >> > >version. No effect.
> > >> > > > >> > >
> > >> > > > >> > >I also noticed that if I ping (google) during the perf
> > >> testing,
> > >> > > > >Windows
> > >> > > > >> > >ping reports "General failure." when JMeter breaks. After
> > >> killing
> > >> > > > >it,
> > >> > > > >> > >connectivity restores and there are replies from the
> server
> > >> > again.
> > >> > > > >It
> > >> > > > >> > >seems
> > >> > > > >> > >like JMeter/HttpClient kills my whole network
> connection...
> > >> > > > >> > >
> > >> > > > >> > >Tuukka
> > >> > > > >> > >
> > >> > > > >> > >
> > >> > > > >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > >> > > > >> > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> > >> > > > >> > >
> > >> > > > >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> > >> > > > >> > ><tuukka.mustonen@gmail.com <javascript:;>
> > >> > > > >> > >> >
> > >> > > > >> > >> wrote:
> > >> > > > >> > >>
> > >> > > > >> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I
> > >> think
> > >> > I
> > >> > > > >saw
> > >> > > > >> > >> slightly
> > >> > > > >> > >> > different behavior in my Linux guest. I'll write more
> > >> about
> > >> > > > >that
> > >> > > > >> > >later,
> > >> > > > >> > >> but
> > >> > > > >> > >> > for simplicity I'm testing only on Windows host now:
> > >> > > > >> > >> >
> > >> > > > >> > >> > Thread dumps are downloadable at:
> > >> > > > >> > >> >
> > >> > > > >https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEF
> > >> uSS1iejA
> > >> > > > >> > >> >
> > >> > > > >> > >> > (Took them with 7-8s interval I think, hope it's ok).
> > >> > > > >> > >> >
> > >> > > > >> > >> > I took the thread dumps once JMeter started filling in
> > >> > errors,
> > >> > > > >> > >which is
> > >> > > > >> > >> > somewhat after my CPU usage hit 100% (and music
> playback
> > >> > > > >started to
> > >> > > > >> > >> distort
> > >> > > > >> > >> > etc.).
> > >> > > > >> > >> >
> > >> > > > >> > >> > I also created the head dump, note that I took it
> after
> > >> > > > >running the
> > >> > > > >> > >test
> > >> > > > >> > >> > and after restarting JMeter, as JMeter froze during
> the
> > >> test
> > >> > > > >and I
> > >> > > > >> > >had to
> > >> > > > >> > >> > kill it. DId you mean I should take it *during* the
> > test?
> > >> > > > >> > >>
> > >> > > > >> > >>
> > >> > > > >> > >> YES.
> > >> > > > >> > >>
> > >> > > > >> > >>
> > >> > > > >> > >> > Anyway, before
> > >> > > > >> > >> > sharing it I want to make sure it can't contain
> anything
> > >> > > > >> > >confidential?
> > >> > > > >> > >> > (parts of my test plan, passwords, environment
> > variables,
> > >> > > > >> > >anything...)
> > >> > > > >> > >> >
> > >> > > > >> > >> > Tuukka
> > >> > > > >> > >> >
> > >> > > > >> > >> >
> > >> > > > >> > >> >
> > >> > > > >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK
> Support
> > <
> > >> > > > >> > >> > support@ubikloadpack.com <javascript:;>> wrote:
> > >> > > > >> > >> >
> > >> > > > >> > >> > > Hello,
> > >> > > > >> > >> > > To investigate the problem:
> > >> > > > >> > >> > > 1/ to create  thread dumps, install a JDK of same
> > >> version
> > >> > as
> > >> > > > >the
> > >> > > > >> > >one
> > >> > > > >> > >> > > running your JMeter.
> > >> > > > >> > >> > >
> > >> > > > >> > >> > > Run until you get the problem, then call using same
> > >> user as
> > >> > > > >the
> > >> > > > >> > >one
> > >> > > > >> > >> > running
> > >> > > > >> > >> > > JMeter 3 times at 5s interval:
> > >> > > > >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >>
> > >> > output_file.txt
> > >> > > > >> > >> > >
> > >> > > > >> > >> > > It will also help if you provide the structure of
> your
> > >> Test
> > >> > > > >Plan.
> > >> > > > >> > >> > >
> > >> > > > >> > >> > > 2/ It would also help if you can provide a Heap
> Dump.
> > >> With
> > >> > > > >JMeter
> > >> > > > >> > >GUI,
> > >> > > > >> > >> > use
> > >> > > > >> > >> > > menu Help > Create a Heap Dump
> > >> > > > >> > >> > > This will create a Heap Dump (rather big file) , it
> > >> would
> > >> > > > >help a
> > >> > > > >> > >lot if
> > >> > > > >> > >> > you
> > >> > > > >> > >> > > can make it available.
> > >> > > > >> > >> > >
> > >> > > > >> > >> > > Regards
> > >> > > > >> > >> > >
> > >> > > > >> > >> > >
> > >> > > > >> > >> > >
> > >> > > > >> > >> > >
> > >> > > > >> > >> > >
> > >> > > > >> > >> > >
> > >> > > > >> > >> > >
> > >> > > > >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > >> > > > >> > >> > tuukka.mustonen@gmail.com <javascript:;>
> > >> > > > >> > >> > > >
> > >> > > > >> > >> > > wrote:
> > >> > > > >> > >> > >
> > >> > > > >> > >> > > > @Philippe:
> > >> > > > >> > >> > > >
> > >> > > > >> > >> > > > Yeah, I am normally running in non-GUI mode, but
> to
> > >> > > > >> > >> > > > study/inspect/understand this problem I am running
> > in
> > >> GUI
> > >> > > > >mode.
> > >> > > > >> > >> JMeter
> > >> > > > >> > >> > > 2.13
> > >> > > > >> > >> > > > works just fine in GUI mode for these tests so
> > JMeter
> > >> 3.x
> > >> > > > >> > >should too.
> > >> > > > >> > >> > > >
> > >> > > > >> > >> > > > I'll give the nightly build a try.
> > >> > > > >> > >> > > >
> > >> > > > >> > >> > > > Tuukka
> > >> > > > >> > >> > > >
> > >> > > > >> > >> > > >
> > >> > > > >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > >> > > > >> > >> > > > philippe.mouawad@gmail.com <javascript:;>
> > >> > > > >> > >> > > > > wrote:
> > >> > > > >> > >> > > >
> > >> > > > >> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen
> <
> > >> > > > >> > >> > > > tuukka.mustonen@gmail.com <javascript:;>
> > >> > > > >> > >> > > > > >
> > >> > > > >> > >> > > > > wrote:
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > > > @Philippe:
> > >> > > > >> > >> > > > > >
> > >> > > > >> > >> > > > > > 1. Yeah, I compared them with the same plan,
> > same
> > >> > > > >machine.
> > >> > > > >> > >Should
> > >> > > > >> > >> > be
> > >> > > > >> > >> > > > > > everything equal.
> > >> > > > >> > >> > > > > > 2. I tried both the defaults in jmeter.bat
> > >> (-Xms512m
> > >> > > > >> > >-Xmx512m)
> > >> > > > >> > >> and
> > >> > > > >> > >> > as
> > >> > > > >> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I
> > didn't
> > >> > > > >tweak
> > >> > > > >> > >them much
> > >> > > > >> > >> > > > > because I
> > >> > > > >> > >> > > > > > don't believe the issue lies there.
> > >> > > > >> > >> > > > > > 3. No, I do just simple HTTP GETs (against
> RESTy
> > >> > API).
> > >> > > > >> > >> > > > > > 4. I am not familiar with that but I can try.
> If
> > >> you
> > >> > > > >have
> > >> > > > >> > >> > > instructions
> > >> > > > >> > >> > > > to
> > >> > > > >> > >> > > > > > point to, please do.
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > > > 5. I am running with GUI.
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > > This is a known anti-pattern .
> > >> > > > >> > >> > > > > switch to non gui mode if you want correct
> > results.
> > >> > > > >> > >> > > > > http://jmeter.apache.org/
> > >> > usermanual/best-practices.html
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > > http://www.ubik-ingenierie.
> > >> > com/blog/jmeter_performance_
> > >> > > > >> > >> tuning_tips/
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > > > About file sending:
> > >> > > > >> > >> > > > > >
> > >> > > > >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > >  Could you give nightly build a try ?
> > >> > > > >> > >> > > > > http://jmeter.apache.org/nightly.html
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > > > Tuukka
> > >> > > > >> > >> > > > > >
> > >> > > > >> > >> > > > > >
> > >> > > > >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe
> > Mouawad <
> > >> > > > >> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> > >> > > <javascript:;>
> > >> > > > >> > >> > > > > > > wrote:
> > >> > > > >> > >> > > > > >
> > >> > > > >> > >> > > > > > > Also are you sending files in post body ?
> > >> > > > >> > >> > > > > > >
> > >> > > > >> > >> > > > > > > Thank you
> > >> > > > >> > >> > > > > > >
> > >> > > > >> > >> > > > > > > On Wednesday, February 1, 2017, Philippe
> > >> Mouawad <
> > >> > > > >> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>
> > >> > > <javascript:;>>
> > >> > > > >> > >> > > > > > > wrote:
> > >> > > > >> > >> > > > > > >
> > >> > > > >> > >> > > > > > > > Hello,
> > >> > > > >> > >> > > > > > > > Few questions, please try to answer them
> > all:
> > >> > > > >> > >> > > > > > > >
> > >> > > > >> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same
> > >> > > > >plan/sape
> > >> > > > >> > >> machine...
> > >> > > > >> > >> > > > > > everything
> > >> > > > >> > >> > > > > > > > equal ?
> > >> > > > >> > >> > > > > > > > 2/ What are your Xmx settings
> > >> > > > >> > >> > > > > > > > 3/ does your plan download embedded
> > resources
> > >> ?
> > >> > > > >> > >> > > > > > > > 4/ when issue happens can you make 3
> thread
> > >> dumps
> > >> > > > >at 5
> > >> > > > >> > >> seconds
> > >> > > > >> > >> > > > > interval
> > >> > > > >> > >> > > > > > > > and send file ? (use jstack)
> > >> > > > >> > >> > > > > > > > 5/ are you running gui or non gui ?
> > >> > > > >> > >> > > > > > > >
> > >> > > > >> > >> > > > > > > > Thanks
> > >> > > > >> > >> > > > > > > > Regards
> > >> > > > >> > >> > > > > > > >
> > >> > > > >> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka
> > >> Mustonen <
> > >> > > > >> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > >> > > <javascript:;>
> > >> > > > >> > >> > > > > > > >
> > >> > > > ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > >> > <javascript:;>
> > >> > > > >> > >> > > > > <javascript:;>');>>
> > >> > > > >> > >> > > > > > wrote:
> > >> > > > >> > >> > > > > > > >
> > >> > > > >> > >> > > > > > > >> Hi,
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I
> > >> > experience
> > >> > > > >high
> > >> > > > >> > >CPU
> > >> > > > >> > >> > usage
> > >> > > > >> > >> > > > > that
> > >> > > > >> > >> > > > > > > >> renders JMeter unusable. CPU usage ramps
> > >> quickly
> > >> > > > >up
> > >> > > > >> > >and
> > >> > > > >> > >> after
> > >> > > > >> > >> > > > having
> > >> > > > >> > >> > > > > > > >> ~100-200 concurrent users or so (in ~30
> > >> > seconds),
> > >> > > > >it
> > >> > > > >> > >hits
> > >> > > > >> > >> 100%
> > >> > > > >> > >> > > on
> > >> > > > >> > >> > > > my
> > >> > > > >> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter
> > freezes
> > >> and
> > >> > > > >I
> > >> > > > >> > >have to
> > >> > > > >> > >> > kill
> > >> > > > >> > >> > > > it.
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> On 2.13 everything goes smoothly (with
> > >> default
> > >> > > > >> > >settings).
> > >> > > > >> > >> CPU
> > >> > > > >> > >> > > > usage
> > >> > > > >> > >> > > > > > > stays
> > >> > > > >> > >> > > > > > > >> at ~30% even with 500 users.
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> The experience is similar on both Windows
> > 10
> > >> and
> > >> > > > >Linux
> > >> > > > >> > >> (Ubuntu
> > >> > > > >> > >> > > > 14.04
> > >> > > > >> > >> > > > > > LTS
> > >> > > > >> > >> > > > > > > >> guest in Virtualbox).
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> The problems I saw in JMeter 3:
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> - java.net.SocketException: Permission
> > >> denied:
> > >> > > > >connect
> > >> > > > >> > >> > > > > > > >> - After a while it runs out of heap space
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> For "Permission denied: connect" I tried
> to
> > >> set
> > >> > > > >> > >> > > > > > > >> java.net.preferIPv4Stack=true
> > >> > > > >> > >> > > > > > > >> (in a few places to make sure it would
> > >> really be
> > >> > > > >in
> > >> > > > >> > >use) and
> > >> > > > >> > >> > > tried
> > >> > > > >> > >> > > > > to
> > >> > > > >> > >> > > > > > > run
> > >> > > > >> > >> > > > > > > >> JMeter as administrator (in Win). No
> > effect.
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> I also tried increasing heap size, but
> that
> > >> > > > >didn't
> > >> > > > >> > >help
> > >> > > > >> > >> > either.
> > >> > > > >> > >> > > > And
> > >> > > > >> > >> > > > > I
> > >> > > > >> > >> > > > > > > >> think
> > >> > > > >> > >> > > > > > > >> memory size is not the root cause here -
> > >> rather
> > >> > > > >the
> > >> > > > >> > >result.
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> I am normally connected to VPN but I also
> > >> tried
> > >> > > > >> > >disabling
> > >> > > > >> > >> it.
> > >> > > > >> > >> > No
> > >> > > > >> > >> > > > > > effect.
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> I tried disabling with and without IPv6
> > >> enabled
> > >> > > > >in
> > >> > > > >> > >Windows
> > >> > > > >> > >> > > network
> > >> > > > >> > >> > > > > > > >> connection settings. No effect.
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> I am running a few extra JMeter plugins.
> > >> Tried
> > >> > > > >cutting
> > >> > > > >> > >my
> > >> > > > >> > >> test
> > >> > > > >> > >> > > > > > scenario
> > >> > > > >> > >> > > > > > > to
> > >> > > > >> > >> > > > > > > >> minimum. No effect.
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> I am running anti-virus software that I
> > >> cannot
> > >> > > > >> > >disable, due
> > >> > > > >> > >> to
> > >> > > > >> > >> > > > this
> > >> > > > >> > >> > > > > > > being
> > >> > > > >> > >> > > > > > > >> forced by IT department. However, it's
> not
> > a
> > >> > > > >problem
> > >> > > > >> > >with
> > >> > > > >> > >> 2.13
> > >> > > > >> > >> > > so
> > >> > > > >> > >> > > > I
> > >> > > > >> > >> > > > > > > assume
> > >> > > > >> > >> > > > > > > >> it shouldn't be a problem here either.
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> FINALLY, once I set HTTP request
> > >> implementation
> > >> > > > >from
> > >> > > > >> > >Apache
> > >> > > > >> > >> > > > > > HttpClient 4
> > >> > > > >> > >> > > > > > > >> to
> > >> > > > >> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem
> > vanishes
> > >> -
> > >> > > > >CPU
> > >> > > > >> > >stays
> > >> > > > >> > >> where
> > >> > > > >> > >> > > it
> > >> > > > >> > >> > > > > did
> > >> > > > >> > >> > > > > > > with
> > >> > > > >> > >> > > > > > > >> JMeter 2.13.
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on
> > Win,
> > >> > > > >update
> > >> > > > >> > >111 on
> > >> > > > >> > >> > > Linux).
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> Why does (the recommended) HttpClient 4
> > cause
> > >> > > > >problems
> > >> > > > >> > >here?
> > >> > > > >> > >> > Is
> > >> > > > >> > >> > > > > this a
> > >> > > > >> > >> > > > > > > >> know
> > >> > > > >> > >> > > > > > > >> bug/issue? JMeter 3.x is running newer
> > >> version
> > >> > > > >> > >compared to
> > >> > > > >> > >> > 2.13,
> > >> > > > >> > >> > > > but
> > >> > > > >> > >> > > > > > > what
> > >> > > > >> > >> > > > > > > >> change in HttpClient 4 might cause the
> > >> problems
> > >> > > > >> > >described?
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> Any ideas?
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >> Regards,
> > >> > > > >> > >> > > > > > > >> Tuukka
> > >> > > > >> > >> > > > > > > >>
> > >> > > > >> > >> > > > > > > >
> > >> > > > >> > >> > > > > > > >
> > >> > > > >> > >> > > > > > > > --
> > >> > > > >> > >> > > > > > > > Cordialement.
> > >> > > > >> > >> > > > > > > > Philippe Mouawad.
> > >> > > > >> > >> > > > > > > >
> > >> > > > >> > >> > > > > > > >
> > >> > > > >> > >> > > > > > > >
> > >> > > > >> > >> > > > > > > >
> > >> > > > >> > >> > > > > > >
> > >> > > > >> > >> > > > > > > --
> > >> > > > >> > >> > > > > > > Cordialement.
> > >> > > > >> > >> > > > > > > Philippe Mouawad.
> > >> > > > >> > >> > > > > > >
> > >> > > > >> > >> > > > > >
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > > > --
> > >> > > > >> > >> > > > > Cordialement.
> > >> > > > >> > >> > > > > Philippe Mouawad.
> > >> > > > >> > >> > > > >
> > >> > > > >> > >> > > >
> > >> > > > >> > >> > >
> > >> > > > >> > >> >
> > >> > > > >> > >>
> > >> > > > >> > >>
> > >> > > > >> > >>
> > >> > > > >> > >> --
> > >> > > > >> > >> Cordialement.
> > >> > > > >> > >> Philippe Mouawad.
> > >> > > > >> > >>
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >-----------------------------------------------------------
> > >> ----------
> > >> > > > >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > >> > > <javascript:;>
> > >> > > > >> > For additional commands, e-mail:
> user-help@jmeter.apache.org
> > >> > > <javascript:;>
> > >> > > > >> >
> > >> > > > >> >
> > >> > > > >>
> > >> > > > >>
> > >> > > > >> --
> > >> > > > >> Cordialement.
> > >> > > > >> Philippe Mouawad.
> > >> > > > >>
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > Cordialement.
> > >> > Philippe Mouawad.
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Cordialement.
> > > Philippe Mouawad.
> > >
> > >
> > >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
>



-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

> In the case of NGinx , it seems no header timeout is sent.
> So if you set a value > to 0, you can control the max idle duration of a
> connection from Client (JMeter) side to avoid storing too many connections
> that won't be reused and can impact JMeter heap.
> By default it's not set so no impact.

Hmm, what do you mean by "header timeout"? Afaik, the server cannot
communicate a timeout for the keep-alived connection to the client. The
server probably has some timeout (I think in nginx it's 75s by default) but
the client doesn't know it. Server will just close the connection at some
point.

Also I'm not 100% sure what you mean by "to avoid storing too many
connections that won't be reused"? The client will re-use connection until
it's closed by TTL configuration or if the server closes the connection,
right? I though with idletimeout I could just make sure that client closes
the connection before the server would (and avoid any potential failed
requests if client sent a request just while the server closes connection)?

I'm not sure, the documentation for that parameter could maybe be improved
a tiny bit?

> But investigating your issue, I discovered that in fact
> "httpclient4.validate_after_inactivity" has no impact due to another bug
> here I am also responsible of :-( .
> validate() method always return true so JMeter always consider connection
> as valid.
> So only "httpclient4.time_to_live" does its job here.

You seem to be author of lots of commits - it's only human to forget
something every now and then :)

> What's happening here is that you're most probably saturating the dynamic
> port range

If you mean that I would be going over the number of available dynamic
ports, I don't think that is it. The test breaks at ~40-50 seconds or so
and at that point there shouldn't be even 10k of connections (potentially)
open. But, I'm definitively hitting some limit. I will contact our IT
department - who knows what magic configuration we have.

> It's because TTL is > 0, so it will be included in the management of idle
> connections.
> ...
> Because JMeter will close connection immediately and inform server to do
> the same.

Ah, good point. What's the policy (=timeline) for closing the idle
connections that go over TTL?

> Thanks a lot for all your tests which helped us narrow this issue.
> Besides Vladimir and Felix are working on improving performances in GUI
> Mode but the team still advises against using it for Load Testing.

Cool. Yeah, I run tests normally from Jenkins in command line mode, but I
always develop/debug in GUI (as does everybody, I would believe). When in
GUI mode, I don't care about performance really - just that the tests pass.
But better performance is welcome of course.

> If you used the nightly build (select Jenkins one) you should be able to
> see that now :
> httpclient4.validate_after_inactivity = 1700
> and has an effect.

Thanks for the quick fixes! I think I will test it next week some point,
just for the curiosity, to see it in effect in Wireshark. I assume this
check sends some dummy package to the server and should be visible in the
trace. Correct me if wrong.

Tuukka



On Sat, Feb 4, 2017 at 5:13 PM, Philippe Mouawad <philippe.mouawad@gmail.com
> wrote:

> Hello Tukka,
> I have commited :
> https://bz.apache.org/bugzilla/show_bug.cgi?id=60690
> https://bz.apache.org/bugzilla/show_bug.cgi?id=60689
>
> If you used the nightly build (select Jenkins one) you should be able to
> see that now :
> httpclient4.validate_after_inactivity = 1700
> and has an effect.
>
> Still in your particular case as you've tested you'll need to increase
> httpclient4.time_to_live to avoid too much connections creations that your
> OS/Machine/Network does not correctly handle.
>
> But beware that this may not be realistic as if you're server calls are
> made from clients that do not use keep alive or persist their connections
> for a short time.
> To ensure your server correctly handles this , you should test from a fine
> tuned machine that has a correct network with target server.
>
>
> Regards
> @philmdot
>
> On Sat, Feb 4, 2017 at 3:16 PM, Philippe Mouawad <
> philippe.mouawad@gmail.com
> > wrote:
>
> >
> >
> > On Sat, Feb 4, 2017 at 12:58 PM, Tuukka Mustonen <
> > tuukka.mustonen@gmail.com> wrote:
> >
> >> On point, Philippe!
> >>
> >> I think *httpclient4.idletimeout* is not of relevance here - my server
> >> sends proper Keep-Alive headers so this option shouldn't have impact.
> Good
> >> to know, for sure.
> >>
> > In the case of NGinx , it seems no header timeout is sent.
> > So if you set a value > to 0, you can control the max idle duration of a
> > connection from Client (JMeter) side to avoid storing too many
> connections
> > that won't be reused and can impact JMeter heap.
> > By default it's not set so no impact.
> >
> >>
> >> *httpclient4.validate_after_inactivity* and *httpclient4.time_to_live
> >> default* values (2 seconds) sound conflicting: TTL being the same as
> >> validate_after_inactivity leads to connections never getting checked as
> >> connections are discarded after 2 seconds anyway? So
> >> validate_after_inactivity is not really in use by default if I
> understood
> >> it right.
> >>
> > Good catch. It somehow went out of my radar, I remember it was something
> I
> > had in mind but it seems I failed to change it.
> > It's a bug or at least bad default value.
> >
> > But investigating your issue, I discovered that in fact
> > "httpclient4.validate_after_inactivity" has no impact due to another bug
> > here I am also responsible of :-( .
> > validate() method always return true so JMeter always consider connection
> > as valid.
> > So only "httpclient4.time_to_live" does its job here.
> >
> >
> >>
> >> So I went on and tried these settings:
> >>
> >> 1) httpclient4.validate_after_inactivity=0 +
> >> httpclient4.time_to_live=2000
> >> -> CRASH
> >> 2) httpclient4.validate_after_inactivity=0 +
> >> httpclient4.time_to_live=1000
> >> -> CRASH
> >> 3) httpclient4.validate_after_inactivity=1 +
> >> httpclient4.time_to_live=2000
> >> -> CRASH
> >>
> >> 4) httpclient4.validate_after_inactivity=5000 +
> >> httpclient4.time_to_live=10000 -> SUCCESS
> >> 5) httpclient4.validate_after_inactivity=0 +
> >> httpclient4.time_to_live=10000
> >> -> SUCCESS
> >> 6) httpclient4.validate_after_inactivity=1 +
> >> httpclient4.time_to_live=10000
> >> -> SUCCESS
> >> 7) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=0
> >> ->
> >> SUCCESS
> >> 8) httpclient4.validate_after_inactivity=10000 +
> >> httpclient4.time_to_live=10000 -> SUCCESS
> >>
> >> Clearly increasing TTL makes the difference for me.
> >> validate_after_inactivity seems to have no effect. Although there is
> some
> >> packet loss, the connections themselves don't break.
> >>
> > Absolutely.
> >
> >>
> >> A) So... maybe my network chip configuration just cannot tolerate JMeter
> >> opening fresh connections every 2 seconds? That translates to 250
> >> connections/s with 500 users- maybe it's enough to bloat my network IO
> and
> >> hit some policy (and I'll then get "connect: Permission denied")?
> Sounds a
> >> bit weird.
> >>
> >
> > It's also your OS, you need to tune the TCP stack to work with port reuse
> > speed.
> > What's happening here is that you're most probably saturating the dynamic
> > port range
> >
> >>
> >> So I also tried:
> >>
> >> 9) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=1
> >> ->
> >> CRASH
> >>
> >> And it crashes quite instantly. I guess it's because JMeter is opening a
> >> new connection per request...
> >>
> > It's because TTL is > 0, so it will be included in the management of idle
> > connections.
> >
> >>
> >> B) ...but if I keep the default 2s settings and just disable Keep-Alive,
> >> then JMeter is opening new connection per request also, and it works.
> Why
> >> the difference?
> >>
> > Because JMeter will close connection immediately and inform server to do
> > the same.
> >
> >
> >>
> >> C) Then, I wonder if your "standard workstation" is supposed to tolerate
> >> these defaults? Can my network chip/driver/configuration just be so bad?
> >>
> >
> > Yes at least your OS Tuning defaults are not well suited for Load
> Testing,
> > you need to tune.
> >
> >>
> >> I see the *httpclient4.validate_after_inactivity* and
> >> *httpclient4.time_to_live
> >> default* properties mentioned in the JMeter 3 changelog:
> >>
> >> "Improved and better performing validation mechanism for Stale
> connections
> >> and Keep-Alive management, see properties
> >> httpclient4.validate_after_inactivity and httpclient4.time_to_live"
> >>
> >
> > Yes, it was the intention.
> >
> >>
> >> And when I monitored TCP connection in Wireshark, JMeter 2.13 indeed
> keeps
> >> connections open waaaay longer (seemed a bit random).
> >>
> >
> > Yes, at that time the only solution was to play with idletimeout or set
> > "http.connection.stalecheck$Boolean=true" which had important impact at
> > high throughput.
> >
> >>
> >> D) So I the new 2s default sounds a bit risky?
> >>
> > In your case yes, but you're in a "bad load testing config" :-)
> >
> >>
> >> In the end, increasing TTL seems to fix the issues for me and I can
> >> proceed
> >> upgrading to JMeter 3.
> >>
> >> Would love to hear your opinion/analysis on the open questions (A-D),
> but
> >> big thanks for your interest in my problem and the precious help already
> >> at
> >> this point!
> >>
> >
> > Thanks a lot for all your tests which helped us narrow this issue.
> > Besides Vladimir and Felix are working on improving performances in GUI
> > Mode but the team still advises against using it for Load Testing.
> >
> >
> >> Tuukka
> >>
> >>
> >>
> >> On Fri, Feb 3, 2017 at 9:40 PM, Philippe Mouawad <
> >> philippe.mouawad@gmail.com
> >> > wrote:
> >>
> >> > On Friday, February 3, 2017, Tuukka Mustonen <
> tuukka.mustonen@gmail.com
> >> >
> >> > wrote:
> >> >
> >> > > Sorry for the delay, got too busy yesterday.
> >> > >
> >> > > I have been running more tests now. I tried to a few variations to
> >> > pinpoint
> >> > > the issue, but first:
> >> > >
> >> > > @Philippe:
> >> > >
> >> > > 0) Response headers are like:
> >> > >
> >> > > Response headers:
> >> > > HTTP/1.1 200 OK
> >> > > Date: Fri, 03 Feb 2017 06:47:00 GMT
> >> > > Content-Type: application/json
> >> > > Content-Length: 374
> >> > > Connection: keep-alive
> >> > > Server: nginx/1.6.2
> >> > > ETag: 37926:1-37927:1
> >> > > Cache-Control: no-cache, no-store
> >> > > Pragma: no-cache
> >> > > Expires: 0
> >> > > X-Transaction: 0000-326srxia31tss3tc
> >> >
> >> >
> >> > As you pointed it later that issue is related to keep alive.
> >> > This is tunable , I 'll try to provide some advices
> >> >
> >> > But read:
> >> > http://jmeter.apache.org/usermanual/properties_reference.
> >> html#httpclient4
> >> >
> >> > And pay attention to:
> >> >
> >> > httpclient4.idletimeout
> >> > httpclient4.validate_after_inactivity
> >> > httpclient4.time_to_live
> >> >
> >> > It appears Default values do not suite your case.
> >> >
> >> >
> >> > > 1) Tried your test plan. Unfortunately, it does not make difference.
> >> >
> >> > it was not made to fix issue only to make it simpler and indépendant
> of
> >> any
> >> > 3rd party plugin
> >> >
> >> > >
> >> > > 2) I ran Wireshark during a couple of tests. You can find a few
> >> > screenshots
> >> > > in a later link. Note that my whole network kinda crashes when
> JMeter
> >> 3
> >> > > gets stuck (ping gets "General failure" and some active connections
> >> > crash,
> >> > > note that this seems a bit random).
> >> >
> >> > Thanks
> >> >
> >> > >
> >> > > 3) I gave it a few minutes to recover between the runs.
> >> > >
> >> > > @Felix:
> >> > >
> >> > > I ran all tests with loggerpanel.usejsyntaxtext=false.
> >> > >
> >> > > No, the amount of seems roughly the same.
> >> > >
> >> > >
> >> > >
> >> > > Then, about pinpointing the issue:
> >> > >
> >> > > I managed to reproduce this by installing nginx on a network
> computer
> >> > (sudo
> >> > > apt-get install nginx on debian 8). I did not configure anything
> and I
> >> > > simply bombarded the default nginx welcome page.
> >> > >
> >> > > I have attached Heap Dump, Thread Dumps and some other information
> in
> >> > these
> >> > > links:
> >> > >
> >> > > JAVA impl (that doesn't crash):
> >> > > https://drive.google.com/drive/folders/0B1uBdVuLED5NSFJvc2VSdE13a0k
> ?
> >> > > usp=sharing
> >> > >
> >> > > HttpCilent 4 impl (that crashes):
> >> > > https://drive.google.com/drive/folders/0B1uBdVuLED5NUGc1WWFYYkRncTQ
> ?
> >> > > usp=sharing
> >> > >
> >> > > I used Philippe's test plan and even simplified it a bit, the JMX
> >> file is
> >> > > in the latter link. Note that the errors occur (in my case) during
> the
> >> > > ramp-up so I also shortened to run just 30s after reaching 500
> users.
> >> > >
> >> > > With JAVA impl, I ran heap dump at ~450 users or so, and thread
> dumps
> >> > after
> >> > > that.
> >> > >
> >> > > With HttpClient 4 impl, I ran heap dump immediately after seeing the
> >> > first
> >> > > error (which was at ~430 users or so) and thread dumps right after
> >> that
> >> > > (the first two dumps I think took a bit longer than normally to
> >> create).
> >> > > JMeter goes unresponsive a second or two after seeing the first
> error,
> >> > so I
> >> > > just barely had time to click to take the dump.
> >> > >
> >> > > As said, there are some screenshots from Wireshark trace for
> >> HttpClient 4
> >> > > run. I am not network expert, so I don't dare to judge them, but I
> >> > included
> >> > > what I saw interesting. I think I can share some parts (privately)
> >> should
> >> > > you want to study the trace in detail.
> >> > >
> >> > > Finally, when I disabled keep-alive from nginx configuration, the
> >> problem
> >> > > was gone (as earlier). So, the problem is somehow related to
> >> keep-alive.
> >> > >
> >> > > Note that in all scenarios CPU goes to 100% quite immediately (also
> on
> >> > 2.13
> >> > > now, probably because of high throughput, I assume my workstation is
> >> the
> >> > > bottleneck here) but without keep-alive or with JAVA impl, things
> work
> >> > and
> >> > > with keep-alive enabled and HttpClient 4, things break apart. As
> >> before,
> >> > on
> >> > > JMeter 2.13 everything works.
> >> > >
> >> > > Tuukka
> >> > >
> >> > >
> >> > >
> >> > > On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <
> >> > > felix.schumacher@internetallee.de <javascript:;>> wrote:
> >> > >
> >> > > >
> >> > > >
> >> > > > Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
> >> > > > tuukka.mustonen@gmail.com <javascript:;>>:
> >> > > > >@Philippe:
> >> > > > >
> >> > > > >0. Alright, I will.
> >> > > > >1. Will do.
> >> > > > >2. Sure.
> >> > > > >3. To let the network connection state restore? Ok, will do.
> >> > > > >
> >> > > > >I am having some other stuff I need to attend today, but I will
> get
> >> > > > >back to
> >> > > > >this in the afternoon at latest.
> >> > > >
> >> > > > Could you also try to set loggerpanel.usejsyntaxtext to false in
> >> your
> >> > > > settings?
> >> > > >
> >> > > > Are there more log entries, when you use httpclient 4 instead of
> >> Java?
> >> > > >
> >> > > > Vladimir has found signs for a event congestion in the awt event
> >> queue,
> >> > > > that show in your threaddumps.
> >> > > >
> >> > > > Regards,
> >> > > >  Felix
> >> > > >
> >> > > > >
> >> > > > >Tuukka
> >> > > > >
> >> > > > >
> >> > > > >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
> >> > > > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> >> > > > >
> >> > > > >> Hello,
> >> > > > >> I've created this bugzilla for investigation and to share with
> >> you
> >> > > > >the plan
> >> > > > >> I have used to run my own tests:
> >> > > > >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
> >> > > > >>
> >> > > > >> I use a Tomcat 8.5.11 with a localhost and a certificate
> >> generated
> >> > > > >and auto
> >> > > > >> signed.
> >> > > > >>
> >> > > > >> So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan
> >> 2.5
> >> > > > >Ghz
> >> > > > >> Intel Core i7
> >> > > > >> Java 8 u121
> >> > > > >> -Xmx512m -Xms512m
> >> > > > >> Test in GUI mode (ANTI-PATTERN)
> >> > > > >>
> >> > > > >> 2.13:
> >> > > > >> - CPU is at 100%
> >> > > > >> - No error
> >> > > > >> - No leak
> >> > > > >>
> >> > > > >> 3.2 nightly with default settings :
> >> > > > >> - CPU is at 100%
> >> > > > >> - No error
> >> > > > >> - No leak
> >> > > > >>
> >> > > > >> 3.2 nightly with this in user.properties :
> >> > > > >> httpclient4.time_to_live=0
> >> > > > >> httpclient4.validate_after_inactivity=0
> >> > > > >>
> >> > > > >> - CPU is at 100%
> >> > > > >> - No error
> >> > > > >> - No leak
> >> > > > >>
> >> > > > >> I attached the Heap Graphs to bugzilla and the plan.
> >> > > > >>
> >> > > > >> So for now I don't  reproduce your issue but I am not in the
> same
> >> > > > >> configuration as I am not hitting the same website.
> >> > > > >>
> >> > > > >> So
> >> > > > >> 0/ Could you provide by running only 5 requests, the response
> >> header
> >> > > > >that
> >> > > > >> your website returns ?
> >> > > > >> 1/ Could you try my test plan by replacing with your target
> host
> >> and
> >> > > > >see if
> >> > > > >> behaviour is the same ?
> >> > > > >> 2/ Ensure nothing is running (No Spotify :-) ) except your
> JMeter
> >> > and
> >> > > > >> monitor network as per Felix note
> >> > > > >> 3/ Run 1 test after the other but ensure you wait a bit between
> >> each
> >> > > > >one
> >> > > > >>
> >> > > > >>
> >> > > > >> Thanks
> >> > > > >>
> >> > > > >>
> >> > > > >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
> >> > > > >> felix.schumacher@internetallee.de <javascript:;>> wrote:
> >> > > > >>
> >> > > > >> >
> >> > > > >> >
> >> > > > >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
> >> > > > >> > tuukka.mustonen@gmail.com <javascript:;>>:
> >> > > > >> > >Ok I re-ran the test.
> >> > > > >> > >
> >> > > > >> > >Took Heap Dump just before JMeter froze.
> >> > > > >> > >
> >> > > > >> > >Took Thread Dumps right after JMeter froze.
> >> > > > >> > >
> >> > > > >> >
> >> > > > >>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqal
> >> lGam13MTA
> >> > > > >> >
> >> > > > >> > That folder seems to be empty now.
> >> > > > >> >
> >> > > > >> > Are you running the tests with a security manager enabled?
> >> > > > >> >
> >> > > > >> > Can you share the test or at least the used requests?
> >> > > > >> >
> >> > > > >> > Are you using a proxy and if so, how is it configured?
> >> > > > >> >
> >> > > > >> > Is jmeter running locally, or over something like rdp?
> >> > > > >> >
> >> > > > >> > Can you look at the used network connections like netstat
> >> -etapn
> >> > > > >would do
> >> > > > >> > on Linux?
> >> > > > >> >
> >> > > > >> > Regards,
> >> > > > >> >  Felix
> >> > > > >> >
> >> > > > >> > >
> >> > > > >> > >...
> >> > > > >> > >
> >> > > > >> > >For the fun of it, I also tried upgrading to the latest
> >> > HttpClient
> >> > > > >> > >4.5.3
> >> > > > >> > >version. No effect.
> >> > > > >> > >
> >> > > > >> > >I also noticed that if I ping (google) during the perf
> >> testing,
> >> > > > >Windows
> >> > > > >> > >ping reports "General failure." when JMeter breaks. After
> >> killing
> >> > > > >it,
> >> > > > >> > >connectivity restores and there are replies from the server
> >> > again.
> >> > > > >It
> >> > > > >> > >seems
> >> > > > >> > >like JMeter/HttpClient kills my whole network connection...
> >> > > > >> > >
> >> > > > >> > >Tuukka
> >> > > > >> > >
> >> > > > >> > >
> >> > > > >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> >> > > > >> > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> >> > > > >> > >
> >> > > > >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> >> > > > >> > ><tuukka.mustonen@gmail.com <javascript:;>
> >> > > > >> > >> >
> >> > > > >> > >> wrote:
> >> > > > >> > >>
> >> > > > >> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I
> >> think
> >> > I
> >> > > > >saw
> >> > > > >> > >> slightly
> >> > > > >> > >> > different behavior in my Linux guest. I'll write more
> >> about
> >> > > > >that
> >> > > > >> > >later,
> >> > > > >> > >> but
> >> > > > >> > >> > for simplicity I'm testing only on Windows host now:
> >> > > > >> > >> >
> >> > > > >> > >> > Thread dumps are downloadable at:
> >> > > > >> > >> >
> >> > > > >https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEF
> >> uSS1iejA
> >> > > > >> > >> >
> >> > > > >> > >> > (Took them with 7-8s interval I think, hope it's ok).
> >> > > > >> > >> >
> >> > > > >> > >> > I took the thread dumps once JMeter started filling in
> >> > errors,
> >> > > > >> > >which is
> >> > > > >> > >> > somewhat after my CPU usage hit 100% (and music playback
> >> > > > >started to
> >> > > > >> > >> distort
> >> > > > >> > >> > etc.).
> >> > > > >> > >> >
> >> > > > >> > >> > I also created the head dump, note that I took it after
> >> > > > >running the
> >> > > > >> > >test
> >> > > > >> > >> > and after restarting JMeter, as JMeter froze during the
> >> test
> >> > > > >and I
> >> > > > >> > >had to
> >> > > > >> > >> > kill it. DId you mean I should take it *during* the
> test?
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >> YES.
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >> > Anyway, before
> >> > > > >> > >> > sharing it I want to make sure it can't contain anything
> >> > > > >> > >confidential?
> >> > > > >> > >> > (parts of my test plan, passwords, environment
> variables,
> >> > > > >> > >anything...)
> >> > > > >> > >> >
> >> > > > >> > >> > Tuukka
> >> > > > >> > >> >
> >> > > > >> > >> >
> >> > > > >> > >> >
> >> > > > >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support
> <
> >> > > > >> > >> > support@ubikloadpack.com <javascript:;>> wrote:
> >> > > > >> > >> >
> >> > > > >> > >> > > Hello,
> >> > > > >> > >> > > To investigate the problem:
> >> > > > >> > >> > > 1/ to create  thread dumps, install a JDK of same
> >> version
> >> > as
> >> > > > >the
> >> > > > >> > >one
> >> > > > >> > >> > > running your JMeter.
> >> > > > >> > >> > >
> >> > > > >> > >> > > Run until you get the problem, then call using same
> >> user as
> >> > > > >the
> >> > > > >> > >one
> >> > > > >> > >> > running
> >> > > > >> > >> > > JMeter 3 times at 5s interval:
> >> > > > >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >>
> >> > output_file.txt
> >> > > > >> > >> > >
> >> > > > >> > >> > > It will also help if you provide the structure of your
> >> Test
> >> > > > >Plan.
> >> > > > >> > >> > >
> >> > > > >> > >> > > 2/ It would also help if you can provide a Heap Dump.
> >> With
> >> > > > >JMeter
> >> > > > >> > >GUI,
> >> > > > >> > >> > use
> >> > > > >> > >> > > menu Help > Create a Heap Dump
> >> > > > >> > >> > > This will create a Heap Dump (rather big file) , it
> >> would
> >> > > > >help a
> >> > > > >> > >lot if
> >> > > > >> > >> > you
> >> > > > >> > >> > > can make it available.
> >> > > > >> > >> > >
> >> > > > >> > >> > > Regards
> >> > > > >> > >> > >
> >> > > > >> > >> > >
> >> > > > >> > >> > >
> >> > > > >> > >> > >
> >> > > > >> > >> > >
> >> > > > >> > >> > >
> >> > > > >> > >> > >
> >> > > > >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> >> > > > >> > >> > tuukka.mustonen@gmail.com <javascript:;>
> >> > > > >> > >> > > >
> >> > > > >> > >> > > wrote:
> >> > > > >> > >> > >
> >> > > > >> > >> > > > @Philippe:
> >> > > > >> > >> > > >
> >> > > > >> > >> > > > Yeah, I am normally running in non-GUI mode, but to
> >> > > > >> > >> > > > study/inspect/understand this problem I am running
> in
> >> GUI
> >> > > > >mode.
> >> > > > >> > >> JMeter
> >> > > > >> > >> > > 2.13
> >> > > > >> > >> > > > works just fine in GUI mode for these tests so
> JMeter
> >> 3.x
> >> > > > >> > >should too.
> >> > > > >> > >> > > >
> >> > > > >> > >> > > > I'll give the nightly build a try.
> >> > > > >> > >> > > >
> >> > > > >> > >> > > > Tuukka
> >> > > > >> > >> > > >
> >> > > > >> > >> > > >
> >> > > > >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> >> > > > >> > >> > > > philippe.mouawad@gmail.com <javascript:;>
> >> > > > >> > >> > > > > wrote:
> >> > > > >> > >> > > >
> >> > > > >> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> >> > > > >> > >> > > > tuukka.mustonen@gmail.com <javascript:;>
> >> > > > >> > >> > > > > >
> >> > > > >> > >> > > > > wrote:
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > > > @Philippe:
> >> > > > >> > >> > > > > >
> >> > > > >> > >> > > > > > 1. Yeah, I compared them with the same plan,
> same
> >> > > > >machine.
> >> > > > >> > >Should
> >> > > > >> > >> > be
> >> > > > >> > >> > > > > > everything equal.
> >> > > > >> > >> > > > > > 2. I tried both the defaults in jmeter.bat
> >> (-Xms512m
> >> > > > >> > >-Xmx512m)
> >> > > > >> > >> and
> >> > > > >> > >> > as
> >> > > > >> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I
> didn't
> >> > > > >tweak
> >> > > > >> > >them much
> >> > > > >> > >> > > > > because I
> >> > > > >> > >> > > > > > don't believe the issue lies there.
> >> > > > >> > >> > > > > > 3. No, I do just simple HTTP GETs (against RESTy
> >> > API).
> >> > > > >> > >> > > > > > 4. I am not familiar with that but I can try. If
> >> you
> >> > > > >have
> >> > > > >> > >> > > instructions
> >> > > > >> > >> > > > to
> >> > > > >> > >> > > > > > point to, please do.
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > > > 5. I am running with GUI.
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > > This is a known anti-pattern .
> >> > > > >> > >> > > > > switch to non gui mode if you want correct
> results.
> >> > > > >> > >> > > > > http://jmeter.apache.org/
> >> > usermanual/best-practices.html
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > > http://www.ubik-ingenierie.
> >> > com/blog/jmeter_performance_
> >> > > > >> > >> tuning_tips/
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > > > About file sending:
> >> > > > >> > >> > > > > >
> >> > > > >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > >  Could you give nightly build a try ?
> >> > > > >> > >> > > > > http://jmeter.apache.org/nightly.html
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > > > Tuukka
> >> > > > >> > >> > > > > >
> >> > > > >> > >> > > > > >
> >> > > > >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe
> Mouawad <
> >> > > > >> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> >> > > <javascript:;>
> >> > > > >> > >> > > > > > > wrote:
> >> > > > >> > >> > > > > >
> >> > > > >> > >> > > > > > > Also are you sending files in post body ?
> >> > > > >> > >> > > > > > >
> >> > > > >> > >> > > > > > > Thank you
> >> > > > >> > >> > > > > > >
> >> > > > >> > >> > > > > > > On Wednesday, February 1, 2017, Philippe
> >> Mouawad <
> >> > > > >> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>
> >> > > <javascript:;>>
> >> > > > >> > >> > > > > > > wrote:
> >> > > > >> > >> > > > > > >
> >> > > > >> > >> > > > > > > > Hello,
> >> > > > >> > >> > > > > > > > Few questions, please try to answer them
> all:
> >> > > > >> > >> > > > > > > >
> >> > > > >> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same
> >> > > > >plan/sape
> >> > > > >> > >> machine...
> >> > > > >> > >> > > > > > everything
> >> > > > >> > >> > > > > > > > equal ?
> >> > > > >> > >> > > > > > > > 2/ What are your Xmx settings
> >> > > > >> > >> > > > > > > > 3/ does your plan download embedded
> resources
> >> ?
> >> > > > >> > >> > > > > > > > 4/ when issue happens can you make 3 thread
> >> dumps
> >> > > > >at 5
> >> > > > >> > >> seconds
> >> > > > >> > >> > > > > interval
> >> > > > >> > >> > > > > > > > and send file ? (use jstack)
> >> > > > >> > >> > > > > > > > 5/ are you running gui or non gui ?
> >> > > > >> > >> > > > > > > >
> >> > > > >> > >> > > > > > > > Thanks
> >> > > > >> > >> > > > > > > > Regards
> >> > > > >> > >> > > > > > > >
> >> > > > >> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka
> >> Mustonen <
> >> > > > >> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> >> > > <javascript:;>
> >> > > > >> > >> > > > > > > >
> >> > > > ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> >> > <javascript:;>
> >> > > > >> > >> > > > > <javascript:;>');>>
> >> > > > >> > >> > > > > > wrote:
> >> > > > >> > >> > > > > > > >
> >> > > > >> > >> > > > > > > >> Hi,
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I
> >> > experience
> >> > > > >high
> >> > > > >> > >CPU
> >> > > > >> > >> > usage
> >> > > > >> > >> > > > > that
> >> > > > >> > >> > > > > > > >> renders JMeter unusable. CPU usage ramps
> >> quickly
> >> > > > >up
> >> > > > >> > >and
> >> > > > >> > >> after
> >> > > > >> > >> > > > having
> >> > > > >> > >> > > > > > > >> ~100-200 concurrent users or so (in ~30
> >> > seconds),
> >> > > > >it
> >> > > > >> > >hits
> >> > > > >> > >> 100%
> >> > > > >> > >> > > on
> >> > > > >> > >> > > > my
> >> > > > >> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter
> freezes
> >> and
> >> > > > >I
> >> > > > >> > >have to
> >> > > > >> > >> > kill
> >> > > > >> > >> > > > it.
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> On 2.13 everything goes smoothly (with
> >> default
> >> > > > >> > >settings).
> >> > > > >> > >> CPU
> >> > > > >> > >> > > > usage
> >> > > > >> > >> > > > > > > stays
> >> > > > >> > >> > > > > > > >> at ~30% even with 500 users.
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> The experience is similar on both Windows
> 10
> >> and
> >> > > > >Linux
> >> > > > >> > >> (Ubuntu
> >> > > > >> > >> > > > 14.04
> >> > > > >> > >> > > > > > LTS
> >> > > > >> > >> > > > > > > >> guest in Virtualbox).
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> The problems I saw in JMeter 3:
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> - java.net.SocketException: Permission
> >> denied:
> >> > > > >connect
> >> > > > >> > >> > > > > > > >> - After a while it runs out of heap space
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> For "Permission denied: connect" I tried to
> >> set
> >> > > > >> > >> > > > > > > >> java.net.preferIPv4Stack=true
> >> > > > >> > >> > > > > > > >> (in a few places to make sure it would
> >> really be
> >> > > > >in
> >> > > > >> > >use) and
> >> > > > >> > >> > > tried
> >> > > > >> > >> > > > > to
> >> > > > >> > >> > > > > > > run
> >> > > > >> > >> > > > > > > >> JMeter as administrator (in Win). No
> effect.
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> I also tried increasing heap size, but that
> >> > > > >didn't
> >> > > > >> > >help
> >> > > > >> > >> > either.
> >> > > > >> > >> > > > And
> >> > > > >> > >> > > > > I
> >> > > > >> > >> > > > > > > >> think
> >> > > > >> > >> > > > > > > >> memory size is not the root cause here -
> >> rather
> >> > > > >the
> >> > > > >> > >result.
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> I am normally connected to VPN but I also
> >> tried
> >> > > > >> > >disabling
> >> > > > >> > >> it.
> >> > > > >> > >> > No
> >> > > > >> > >> > > > > > effect.
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> I tried disabling with and without IPv6
> >> enabled
> >> > > > >in
> >> > > > >> > >Windows
> >> > > > >> > >> > > network
> >> > > > >> > >> > > > > > > >> connection settings. No effect.
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> I am running a few extra JMeter plugins.
> >> Tried
> >> > > > >cutting
> >> > > > >> > >my
> >> > > > >> > >> test
> >> > > > >> > >> > > > > > scenario
> >> > > > >> > >> > > > > > > to
> >> > > > >> > >> > > > > > > >> minimum. No effect.
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> I am running anti-virus software that I
> >> cannot
> >> > > > >> > >disable, due
> >> > > > >> > >> to
> >> > > > >> > >> > > > this
> >> > > > >> > >> > > > > > > being
> >> > > > >> > >> > > > > > > >> forced by IT department. However, it's not
> a
> >> > > > >problem
> >> > > > >> > >with
> >> > > > >> > >> 2.13
> >> > > > >> > >> > > so
> >> > > > >> > >> > > > I
> >> > > > >> > >> > > > > > > assume
> >> > > > >> > >> > > > > > > >> it shouldn't be a problem here either.
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> FINALLY, once I set HTTP request
> >> implementation
> >> > > > >from
> >> > > > >> > >Apache
> >> > > > >> > >> > > > > > HttpClient 4
> >> > > > >> > >> > > > > > > >> to
> >> > > > >> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem
> vanishes
> >> -
> >> > > > >CPU
> >> > > > >> > >stays
> >> > > > >> > >> where
> >> > > > >> > >> > > it
> >> > > > >> > >> > > > > did
> >> > > > >> > >> > > > > > > with
> >> > > > >> > >> > > > > > > >> JMeter 2.13.
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on
> Win,
> >> > > > >update
> >> > > > >> > >111 on
> >> > > > >> > >> > > Linux).
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> Why does (the recommended) HttpClient 4
> cause
> >> > > > >problems
> >> > > > >> > >here?
> >> > > > >> > >> > Is
> >> > > > >> > >> > > > > this a
> >> > > > >> > >> > > > > > > >> know
> >> > > > >> > >> > > > > > > >> bug/issue? JMeter 3.x is running newer
> >> version
> >> > > > >> > >compared to
> >> > > > >> > >> > 2.13,
> >> > > > >> > >> > > > but
> >> > > > >> > >> > > > > > > what
> >> > > > >> > >> > > > > > > >> change in HttpClient 4 might cause the
> >> problems
> >> > > > >> > >described?
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> Any ideas?
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >> Regards,
> >> > > > >> > >> > > > > > > >> Tuukka
> >> > > > >> > >> > > > > > > >>
> >> > > > >> > >> > > > > > > >
> >> > > > >> > >> > > > > > > >
> >> > > > >> > >> > > > > > > > --
> >> > > > >> > >> > > > > > > > Cordialement.
> >> > > > >> > >> > > > > > > > Philippe Mouawad.
> >> > > > >> > >> > > > > > > >
> >> > > > >> > >> > > > > > > >
> >> > > > >> > >> > > > > > > >
> >> > > > >> > >> > > > > > > >
> >> > > > >> > >> > > > > > >
> >> > > > >> > >> > > > > > > --
> >> > > > >> > >> > > > > > > Cordialement.
> >> > > > >> > >> > > > > > > Philippe Mouawad.
> >> > > > >> > >> > > > > > >
> >> > > > >> > >> > > > > >
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > > > --
> >> > > > >> > >> > > > > Cordialement.
> >> > > > >> > >> > > > > Philippe Mouawad.
> >> > > > >> > >> > > > >
> >> > > > >> > >> > > >
> >> > > > >> > >> > >
> >> > > > >> > >> >
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >>
> >> > > > >> > >> --
> >> > > > >> > >> Cordialement.
> >> > > > >> > >> Philippe Mouawad.
> >> > > > >> > >>
> >> > > > >> >
> >> > > > >> >
> >> > > > >-----------------------------------------------------------
> >> ----------
> >> > > > >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >> > > <javascript:;>
> >> > > > >> > For additional commands, e-mail: user-help@jmeter.apache.org
> >> > > <javascript:;>
> >> > > > >> >
> >> > > > >> >
> >> > > > >>
> >> > > > >>
> >> > > > >> --
> >> > > > >> Cordialement.
> >> > > > >> Philippe Mouawad.
> >> > > > >>
> >> > > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Cordialement.
> >> > Philippe Mouawad.
> >> >
> >>
> >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
> >
> >
>
>
> --
> Cordialement.
> Philippe Mouawad.
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

Hello Tukka,
I have commited :
https://bz.apache.org/bugzilla/show_bug.cgi?id=60690
https://bz.apache.org/bugzilla/show_bug.cgi?id=60689

If you used the nightly build (select Jenkins one) you should be able to
see that now :
httpclient4.validate_after_inactivity = 1700
and has an effect.

Still in your particular case as you've tested you'll need to increase
httpclient4.time_to_live to avoid too much connections creations that your
OS/Machine/Network does not correctly handle.

But beware that this may not be realistic as if you're server calls are
made from clients that do not use keep alive or persist their connections
for a short time.
To ensure your server correctly handles this , you should test from a fine
tuned machine that has a correct network with target server.


Regards
@philmdot

On Sat, Feb 4, 2017 at 3:16 PM, Philippe Mouawad <philippe.mouawad@gmail.com
> wrote:

>
>
> On Sat, Feb 4, 2017 at 12:58 PM, Tuukka Mustonen <
> tuukka.mustonen@gmail.com> wrote:
>
>> On point, Philippe!
>>
>> I think *httpclient4.idletimeout* is not of relevance here - my server
>> sends proper Keep-Alive headers so this option shouldn't have impact. Good
>> to know, for sure.
>>
> In the case of NGinx , it seems no header timeout is sent.
> So if you set a value > to 0, you can control the max idle duration of a
> connection from Client (JMeter) side to avoid storing too many connections
> that won't be reused and can impact JMeter heap.
> By default it's not set so no impact.
>
>>
>> *httpclient4.validate_after_inactivity* and *httpclient4.time_to_live
>> default* values (2 seconds) sound conflicting: TTL being the same as
>> validate_after_inactivity leads to connections never getting checked as
>> connections are discarded after 2 seconds anyway? So
>> validate_after_inactivity is not really in use by default if I understood
>> it right.
>>
> Good catch. It somehow went out of my radar, I remember it was something I
> had in mind but it seems I failed to change it.
> It's a bug or at least bad default value.
>
> But investigating your issue, I discovered that in fact
> "httpclient4.validate_after_inactivity" has no impact due to another bug
> here I am also responsible of :-( .
> validate() method always return true so JMeter always consider connection
> as valid.
> So only "httpclient4.time_to_live" does its job here.
>
>
>>
>> So I went on and tried these settings:
>>
>> 1) httpclient4.validate_after_inactivity=0 +
>> httpclient4.time_to_live=2000
>> -> CRASH
>> 2) httpclient4.validate_after_inactivity=0 +
>> httpclient4.time_to_live=1000
>> -> CRASH
>> 3) httpclient4.validate_after_inactivity=1 +
>> httpclient4.time_to_live=2000
>> -> CRASH
>>
>> 4) httpclient4.validate_after_inactivity=5000 +
>> httpclient4.time_to_live=10000 -> SUCCESS
>> 5) httpclient4.validate_after_inactivity=0 +
>> httpclient4.time_to_live=10000
>> -> SUCCESS
>> 6) httpclient4.validate_after_inactivity=1 +
>> httpclient4.time_to_live=10000
>> -> SUCCESS
>> 7) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=0
>> ->
>> SUCCESS
>> 8) httpclient4.validate_after_inactivity=10000 +
>> httpclient4.time_to_live=10000 -> SUCCESS
>>
>> Clearly increasing TTL makes the difference for me.
>> validate_after_inactivity seems to have no effect. Although there is some
>> packet loss, the connections themselves don't break.
>>
> Absolutely.
>
>>
>> A) So... maybe my network chip configuration just cannot tolerate JMeter
>> opening fresh connections every 2 seconds? That translates to 250
>> connections/s with 500 users- maybe it's enough to bloat my network IO and
>> hit some policy (and I'll then get "connect: Permission denied")? Sounds a
>> bit weird.
>>
>
> It's also your OS, you need to tune the TCP stack to work with port reuse
> speed.
> What's happening here is that you're most probably saturating the dynamic
> port range
>
>>
>> So I also tried:
>>
>> 9) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=1
>> ->
>> CRASH
>>
>> And it crashes quite instantly. I guess it's because JMeter is opening a
>> new connection per request...
>>
> It's because TTL is > 0, so it will be included in the management of idle
> connections.
>
>>
>> B) ...but if I keep the default 2s settings and just disable Keep-Alive,
>> then JMeter is opening new connection per request also, and it works. Why
>> the difference?
>>
> Because JMeter will close connection immediately and inform server to do
> the same.
>
>
>>
>> C) Then, I wonder if your "standard workstation" is supposed to tolerate
>> these defaults? Can my network chip/driver/configuration just be so bad?
>>
>
> Yes at least your OS Tuning defaults are not well suited for Load Testing,
> you need to tune.
>
>>
>> I see the *httpclient4.validate_after_inactivity* and
>> *httpclient4.time_to_live
>> default* properties mentioned in the JMeter 3 changelog:
>>
>> "Improved and better performing validation mechanism for Stale connections
>> and Keep-Alive management, see properties
>> httpclient4.validate_after_inactivity and httpclient4.time_to_live"
>>
>
> Yes, it was the intention.
>
>>
>> And when I monitored TCP connection in Wireshark, JMeter 2.13 indeed keeps
>> connections open waaaay longer (seemed a bit random).
>>
>
> Yes, at that time the only solution was to play with idletimeout or set
> "http.connection.stalecheck$Boolean=true" which had important impact at
> high throughput.
>
>>
>> D) So I the new 2s default sounds a bit risky?
>>
> In your case yes, but you're in a "bad load testing config" :-)
>
>>
>> In the end, increasing TTL seems to fix the issues for me and I can
>> proceed
>> upgrading to JMeter 3.
>>
>> Would love to hear your opinion/analysis on the open questions (A-D), but
>> big thanks for your interest in my problem and the precious help already
>> at
>> this point!
>>
>
> Thanks a lot for all your tests which helped us narrow this issue.
> Besides Vladimir and Felix are working on improving performances in GUI
> Mode but the team still advises against using it for Load Testing.
>
>
>> Tuukka
>>
>>
>>
>> On Fri, Feb 3, 2017 at 9:40 PM, Philippe Mouawad <
>> philippe.mouawad@gmail.com
>> > wrote:
>>
>> > On Friday, February 3, 2017, Tuukka Mustonen <tuukka.mustonen@gmail.com
>> >
>> > wrote:
>> >
>> > > Sorry for the delay, got too busy yesterday.
>> > >
>> > > I have been running more tests now. I tried to a few variations to
>> > pinpoint
>> > > the issue, but first:
>> > >
>> > > @Philippe:
>> > >
>> > > 0) Response headers are like:
>> > >
>> > > Response headers:
>> > > HTTP/1.1 200 OK
>> > > Date: Fri, 03 Feb 2017 06:47:00 GMT
>> > > Content-Type: application/json
>> > > Content-Length: 374
>> > > Connection: keep-alive
>> > > Server: nginx/1.6.2
>> > > ETag: 37926:1-37927:1
>> > > Cache-Control: no-cache, no-store
>> > > Pragma: no-cache
>> > > Expires: 0
>> > > X-Transaction: 0000-326srxia31tss3tc
>> >
>> >
>> > As you pointed it later that issue is related to keep alive.
>> > This is tunable , I 'll try to provide some advices
>> >
>> > But read:
>> > http://jmeter.apache.org/usermanual/properties_reference.
>> html#httpclient4
>> >
>> > And pay attention to:
>> >
>> > httpclient4.idletimeout
>> > httpclient4.validate_after_inactivity
>> > httpclient4.time_to_live
>> >
>> > It appears Default values do not suite your case.
>> >
>> >
>> > > 1) Tried your test plan. Unfortunately, it does not make difference.
>> >
>> > it was not made to fix issue only to make it simpler and indépendant of
>> any
>> > 3rd party plugin
>> >
>> > >
>> > > 2) I ran Wireshark during a couple of tests. You can find a few
>> > screenshots
>> > > in a later link. Note that my whole network kinda crashes when JMeter
>> 3
>> > > gets stuck (ping gets "General failure" and some active connections
>> > crash,
>> > > note that this seems a bit random).
>> >
>> > Thanks
>> >
>> > >
>> > > 3) I gave it a few minutes to recover between the runs.
>> > >
>> > > @Felix:
>> > >
>> > > I ran all tests with loggerpanel.usejsyntaxtext=false.
>> > >
>> > > No, the amount of seems roughly the same.
>> > >
>> > >
>> > >
>> > > Then, about pinpointing the issue:
>> > >
>> > > I managed to reproduce this by installing nginx on a network computer
>> > (sudo
>> > > apt-get install nginx on debian 8). I did not configure anything and I
>> > > simply bombarded the default nginx welcome page.
>> > >
>> > > I have attached Heap Dump, Thread Dumps and some other information in
>> > these
>> > > links:
>> > >
>> > > JAVA impl (that doesn't crash):
>> > > https://drive.google.com/drive/folders/0B1uBdVuLED5NSFJvc2VSdE13a0k?
>> > > usp=sharing
>> > >
>> > > HttpCilent 4 impl (that crashes):
>> > > https://drive.google.com/drive/folders/0B1uBdVuLED5NUGc1WWFYYkRncTQ?
>> > > usp=sharing
>> > >
>> > > I used Philippe's test plan and even simplified it a bit, the JMX
>> file is
>> > > in the latter link. Note that the errors occur (in my case) during the
>> > > ramp-up so I also shortened to run just 30s after reaching 500 users.
>> > >
>> > > With JAVA impl, I ran heap dump at ~450 users or so, and thread dumps
>> > after
>> > > that.
>> > >
>> > > With HttpClient 4 impl, I ran heap dump immediately after seeing the
>> > first
>> > > error (which was at ~430 users or so) and thread dumps right after
>> that
>> > > (the first two dumps I think took a bit longer than normally to
>> create).
>> > > JMeter goes unresponsive a second or two after seeing the first error,
>> > so I
>> > > just barely had time to click to take the dump.
>> > >
>> > > As said, there are some screenshots from Wireshark trace for
>> HttpClient 4
>> > > run. I am not network expert, so I don't dare to judge them, but I
>> > included
>> > > what I saw interesting. I think I can share some parts (privately)
>> should
>> > > you want to study the trace in detail.
>> > >
>> > > Finally, when I disabled keep-alive from nginx configuration, the
>> problem
>> > > was gone (as earlier). So, the problem is somehow related to
>> keep-alive.
>> > >
>> > > Note that in all scenarios CPU goes to 100% quite immediately (also on
>> > 2.13
>> > > now, probably because of high throughput, I assume my workstation is
>> the
>> > > bottleneck here) but without keep-alive or with JAVA impl, things work
>> > and
>> > > with keep-alive enabled and HttpClient 4, things break apart. As
>> before,
>> > on
>> > > JMeter 2.13 everything works.
>> > >
>> > > Tuukka
>> > >
>> > >
>> > >
>> > > On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <
>> > > felix.schumacher@internetallee.de <javascript:;>> wrote:
>> > >
>> > > >
>> > > >
>> > > > Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
>> > > > tuukka.mustonen@gmail.com <javascript:;>>:
>> > > > >@Philippe:
>> > > > >
>> > > > >0. Alright, I will.
>> > > > >1. Will do.
>> > > > >2. Sure.
>> > > > >3. To let the network connection state restore? Ok, will do.
>> > > > >
>> > > > >I am having some other stuff I need to attend today, but I will get
>> > > > >back to
>> > > > >this in the afternoon at latest.
>> > > >
>> > > > Could you also try to set loggerpanel.usejsyntaxtext to false in
>> your
>> > > > settings?
>> > > >
>> > > > Are there more log entries, when you use httpclient 4 instead of
>> Java?
>> > > >
>> > > > Vladimir has found signs for a event congestion in the awt event
>> queue,
>> > > > that show in your threaddumps.
>> > > >
>> > > > Regards,
>> > > >  Felix
>> > > >
>> > > > >
>> > > > >Tuukka
>> > > > >
>> > > > >
>> > > > >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
>> > > > >philippe.mouawad@gmail.com <javascript:;>> wrote:
>> > > > >
>> > > > >> Hello,
>> > > > >> I've created this bugzilla for investigation and to share with
>> you
>> > > > >the plan
>> > > > >> I have used to run my own tests:
>> > > > >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
>> > > > >>
>> > > > >> I use a Tomcat 8.5.11 with a localhost and a certificate
>> generated
>> > > > >and auto
>> > > > >> signed.
>> > > > >>
>> > > > >> So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan
>> 2.5
>> > > > >Ghz
>> > > > >> Intel Core i7
>> > > > >> Java 8 u121
>> > > > >> -Xmx512m -Xms512m
>> > > > >> Test in GUI mode (ANTI-PATTERN)
>> > > > >>
>> > > > >> 2.13:
>> > > > >> - CPU is at 100%
>> > > > >> - No error
>> > > > >> - No leak
>> > > > >>
>> > > > >> 3.2 nightly with default settings :
>> > > > >> - CPU is at 100%
>> > > > >> - No error
>> > > > >> - No leak
>> > > > >>
>> > > > >> 3.2 nightly with this in user.properties :
>> > > > >> httpclient4.time_to_live=0
>> > > > >> httpclient4.validate_after_inactivity=0
>> > > > >>
>> > > > >> - CPU is at 100%
>> > > > >> - No error
>> > > > >> - No leak
>> > > > >>
>> > > > >> I attached the Heap Graphs to bugzilla and the plan.
>> > > > >>
>> > > > >> So for now I don't  reproduce your issue but I am not in the same
>> > > > >> configuration as I am not hitting the same website.
>> > > > >>
>> > > > >> So
>> > > > >> 0/ Could you provide by running only 5 requests, the response
>> header
>> > > > >that
>> > > > >> your website returns ?
>> > > > >> 1/ Could you try my test plan by replacing with your target host
>> and
>> > > > >see if
>> > > > >> behaviour is the same ?
>> > > > >> 2/ Ensure nothing is running (No Spotify :-) ) except your JMeter
>> > and
>> > > > >> monitor network as per Felix note
>> > > > >> 3/ Run 1 test after the other but ensure you wait a bit between
>> each
>> > > > >one
>> > > > >>
>> > > > >>
>> > > > >> Thanks
>> > > > >>
>> > > > >>
>> > > > >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
>> > > > >> felix.schumacher@internetallee.de <javascript:;>> wrote:
>> > > > >>
>> > > > >> >
>> > > > >> >
>> > > > >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
>> > > > >> > tuukka.mustonen@gmail.com <javascript:;>>:
>> > > > >> > >Ok I re-ran the test.
>> > > > >> > >
>> > > > >> > >Took Heap Dump just before JMeter froze.
>> > > > >> > >
>> > > > >> > >Took Thread Dumps right after JMeter froze.
>> > > > >> > >
>> > > > >> >
>> > > > >>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqal
>> lGam13MTA
>> > > > >> >
>> > > > >> > That folder seems to be empty now.
>> > > > >> >
>> > > > >> > Are you running the tests with a security manager enabled?
>> > > > >> >
>> > > > >> > Can you share the test or at least the used requests?
>> > > > >> >
>> > > > >> > Are you using a proxy and if so, how is it configured?
>> > > > >> >
>> > > > >> > Is jmeter running locally, or over something like rdp?
>> > > > >> >
>> > > > >> > Can you look at the used network connections like netstat
>> -etapn
>> > > > >would do
>> > > > >> > on Linux?
>> > > > >> >
>> > > > >> > Regards,
>> > > > >> >  Felix
>> > > > >> >
>> > > > >> > >
>> > > > >> > >...
>> > > > >> > >
>> > > > >> > >For the fun of it, I also tried upgrading to the latest
>> > HttpClient
>> > > > >> > >4.5.3
>> > > > >> > >version. No effect.
>> > > > >> > >
>> > > > >> > >I also noticed that if I ping (google) during the perf
>> testing,
>> > > > >Windows
>> > > > >> > >ping reports "General failure." when JMeter breaks. After
>> killing
>> > > > >it,
>> > > > >> > >connectivity restores and there are replies from the server
>> > again.
>> > > > >It
>> > > > >> > >seems
>> > > > >> > >like JMeter/HttpClient kills my whole network connection...
>> > > > >> > >
>> > > > >> > >Tuukka
>> > > > >> > >
>> > > > >> > >
>> > > > >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
>> > > > >> > >philippe.mouawad@gmail.com <javascript:;>> wrote:
>> > > > >> > >
>> > > > >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
>> > > > >> > ><tuukka.mustonen@gmail.com <javascript:;>
>> > > > >> > >> >
>> > > > >> > >> wrote:
>> > > > >> > >>
>> > > > >> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I
>> think
>> > I
>> > > > >saw
>> > > > >> > >> slightly
>> > > > >> > >> > different behavior in my Linux guest. I'll write more
>> about
>> > > > >that
>> > > > >> > >later,
>> > > > >> > >> but
>> > > > >> > >> > for simplicity I'm testing only on Windows host now:
>> > > > >> > >> >
>> > > > >> > >> > Thread dumps are downloadable at:
>> > > > >> > >> >
>> > > > >https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEF
>> uSS1iejA
>> > > > >> > >> >
>> > > > >> > >> > (Took them with 7-8s interval I think, hope it's ok).
>> > > > >> > >> >
>> > > > >> > >> > I took the thread dumps once JMeter started filling in
>> > errors,
>> > > > >> > >which is
>> > > > >> > >> > somewhat after my CPU usage hit 100% (and music playback
>> > > > >started to
>> > > > >> > >> distort
>> > > > >> > >> > etc.).
>> > > > >> > >> >
>> > > > >> > >> > I also created the head dump, note that I took it after
>> > > > >running the
>> > > > >> > >test
>> > > > >> > >> > and after restarting JMeter, as JMeter froze during the
>> test
>> > > > >and I
>> > > > >> > >had to
>> > > > >> > >> > kill it. DId you mean I should take it *during* the test?
>> > > > >> > >>
>> > > > >> > >>
>> > > > >> > >> YES.
>> > > > >> > >>
>> > > > >> > >>
>> > > > >> > >> > Anyway, before
>> > > > >> > >> > sharing it I want to make sure it can't contain anything
>> > > > >> > >confidential?
>> > > > >> > >> > (parts of my test plan, passwords, environment variables,
>> > > > >> > >anything...)
>> > > > >> > >> >
>> > > > >> > >> > Tuukka
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> >
>> > > > >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
>> > > > >> > >> > support@ubikloadpack.com <javascript:;>> wrote:
>> > > > >> > >> >
>> > > > >> > >> > > Hello,
>> > > > >> > >> > > To investigate the problem:
>> > > > >> > >> > > 1/ to create  thread dumps, install a JDK of same
>> version
>> > as
>> > > > >the
>> > > > >> > >one
>> > > > >> > >> > > running your JMeter.
>> > > > >> > >> > >
>> > > > >> > >> > > Run until you get the problem, then call using same
>> user as
>> > > > >the
>> > > > >> > >one
>> > > > >> > >> > running
>> > > > >> > >> > > JMeter 3 times at 5s interval:
>> > > > >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >>
>> > output_file.txt
>> > > > >> > >> > >
>> > > > >> > >> > > It will also help if you provide the structure of your
>> Test
>> > > > >Plan.
>> > > > >> > >> > >
>> > > > >> > >> > > 2/ It would also help if you can provide a Heap Dump.
>> With
>> > > > >JMeter
>> > > > >> > >GUI,
>> > > > >> > >> > use
>> > > > >> > >> > > menu Help > Create a Heap Dump
>> > > > >> > >> > > This will create a Heap Dump (rather big file) , it
>> would
>> > > > >help a
>> > > > >> > >lot if
>> > > > >> > >> > you
>> > > > >> > >> > > can make it available.
>> > > > >> > >> > >
>> > > > >> > >> > > Regards
>> > > > >> > >> > >
>> > > > >> > >> > >
>> > > > >> > >> > >
>> > > > >> > >> > >
>> > > > >> > >> > >
>> > > > >> > >> > >
>> > > > >> > >> > >
>> > > > >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
>> > > > >> > >> > tuukka.mustonen@gmail.com <javascript:;>
>> > > > >> > >> > > >
>> > > > >> > >> > > wrote:
>> > > > >> > >> > >
>> > > > >> > >> > > > @Philippe:
>> > > > >> > >> > > >
>> > > > >> > >> > > > Yeah, I am normally running in non-GUI mode, but to
>> > > > >> > >> > > > study/inspect/understand this problem I am running in
>> GUI
>> > > > >mode.
>> > > > >> > >> JMeter
>> > > > >> > >> > > 2.13
>> > > > >> > >> > > > works just fine in GUI mode for these tests so JMeter
>> 3.x
>> > > > >> > >should too.
>> > > > >> > >> > > >
>> > > > >> > >> > > > I'll give the nightly build a try.
>> > > > >> > >> > > >
>> > > > >> > >> > > > Tuukka
>> > > > >> > >> > > >
>> > > > >> > >> > > >
>> > > > >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
>> > > > >> > >> > > > philippe.mouawad@gmail.com <javascript:;>
>> > > > >> > >> > > > > wrote:
>> > > > >> > >> > > >
>> > > > >> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> > > > >> > >> > > > tuukka.mustonen@gmail.com <javascript:;>
>> > > > >> > >> > > > > >
>> > > > >> > >> > > > > wrote:
>> > > > >> > >> > > > >
>> > > > >> > >> > > > > > @Philippe:
>> > > > >> > >> > > > > >
>> > > > >> > >> > > > > > 1. Yeah, I compared them with the same plan, same
>> > > > >machine.
>> > > > >> > >Should
>> > > > >> > >> > be
>> > > > >> > >> > > > > > everything equal.
>> > > > >> > >> > > > > > 2. I tried both the defaults in jmeter.bat
>> (-Xms512m
>> > > > >> > >-Xmx512m)
>> > > > >> > >> and
>> > > > >> > >> > as
>> > > > >> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't
>> > > > >tweak
>> > > > >> > >them much
>> > > > >> > >> > > > > because I
>> > > > >> > >> > > > > > don't believe the issue lies there.
>> > > > >> > >> > > > > > 3. No, I do just simple HTTP GETs (against RESTy
>> > API).
>> > > > >> > >> > > > > > 4. I am not familiar with that but I can try. If
>> you
>> > > > >have
>> > > > >> > >> > > instructions
>> > > > >> > >> > > > to
>> > > > >> > >> > > > > > point to, please do.
>> > > > >> > >> > > > >
>> > > > >> > >> > > > >
>> > > > >> > >> > > > >
>> > > > >> > >> > > > >
>> > > > >> > >> > > > > > 5. I am running with GUI.
>> > > > >> > >> > > > >
>> > > > >> > >> > > > >
>> > > > >> > >> > > > > This is a known anti-pattern .
>> > > > >> > >> > > > > switch to non gui mode if you want correct results.
>> > > > >> > >> > > > > http://jmeter.apache.org/
>> > usermanual/best-practices.html
>> > > > >> > >> > > > >
>> > > > >> > >> > > > > http://www.ubik-ingenierie.
>> > com/blog/jmeter_performance_
>> > > > >> > >> tuning_tips/
>> > > > >> > >> > > > >
>> > > > >> > >> > > > >
>> > > > >> > >> > > > > > About file sending:
>> > > > >> > >> > > > > >
>> > > > >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
>> > > > >> > >> > > > >
>> > > > >> > >> > > > >  Could you give nightly build a try ?
>> > > > >> > >> > > > > http://jmeter.apache.org/nightly.html
>> > > > >> > >> > > > >
>> > > > >> > >> > > > >
>> > > > >> > >> > > > > > Tuukka
>> > > > >> > >> > > > > >
>> > > > >> > >> > > > > >
>> > > > >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
>> > > > >> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
>> > > <javascript:;>
>> > > > >> > >> > > > > > > wrote:
>> > > > >> > >> > > > > >
>> > > > >> > >> > > > > > > Also are you sending files in post body ?
>> > > > >> > >> > > > > > >
>> > > > >> > >> > > > > > > Thank you
>> > > > >> > >> > > > > > >
>> > > > >> > >> > > > > > > On Wednesday, February 1, 2017, Philippe
>> Mouawad <
>> > > > >> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>
>> > > <javascript:;>>
>> > > > >> > >> > > > > > > wrote:
>> > > > >> > >> > > > > > >
>> > > > >> > >> > > > > > > > Hello,
>> > > > >> > >> > > > > > > > Few questions, please try to answer them all:
>> > > > >> > >> > > > > > > >
>> > > > >> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same
>> > > > >plan/sape
>> > > > >> > >> machine...
>> > > > >> > >> > > > > > everything
>> > > > >> > >> > > > > > > > equal ?
>> > > > >> > >> > > > > > > > 2/ What are your Xmx settings
>> > > > >> > >> > > > > > > > 3/ does your plan download embedded resources
>> ?
>> > > > >> > >> > > > > > > > 4/ when issue happens can you make 3 thread
>> dumps
>> > > > >at 5
>> > > > >> > >> seconds
>> > > > >> > >> > > > > interval
>> > > > >> > >> > > > > > > > and send file ? (use jstack)
>> > > > >> > >> > > > > > > > 5/ are you running gui or non gui ?
>> > > > >> > >> > > > > > > >
>> > > > >> > >> > > > > > > > Thanks
>> > > > >> > >> > > > > > > > Regards
>> > > > >> > >> > > > > > > >
>> > > > >> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka
>> Mustonen <
>> > > > >> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
>> > > <javascript:;>
>> > > > >> > >> > > > > > > >
>> > > > ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
>> > <javascript:;>
>> > > > >> > >> > > > > <javascript:;>');>>
>> > > > >> > >> > > > > > wrote:
>> > > > >> > >> > > > > > > >
>> > > > >> > >> > > > > > > >> Hi,
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I
>> > experience
>> > > > >high
>> > > > >> > >CPU
>> > > > >> > >> > usage
>> > > > >> > >> > > > > that
>> > > > >> > >> > > > > > > >> renders JMeter unusable. CPU usage ramps
>> quickly
>> > > > >up
>> > > > >> > >and
>> > > > >> > >> after
>> > > > >> > >> > > > having
>> > > > >> > >> > > > > > > >> ~100-200 concurrent users or so (in ~30
>> > seconds),
>> > > > >it
>> > > > >> > >hits
>> > > > >> > >> 100%
>> > > > >> > >> > > on
>> > > > >> > >> > > > my
>> > > > >> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes
>> and
>> > > > >I
>> > > > >> > >have to
>> > > > >> > >> > kill
>> > > > >> > >> > > > it.
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> On 2.13 everything goes smoothly (with
>> default
>> > > > >> > >settings).
>> > > > >> > >> CPU
>> > > > >> > >> > > > usage
>> > > > >> > >> > > > > > > stays
>> > > > >> > >> > > > > > > >> at ~30% even with 500 users.
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> The experience is similar on both Windows 10
>> and
>> > > > >Linux
>> > > > >> > >> (Ubuntu
>> > > > >> > >> > > > 14.04
>> > > > >> > >> > > > > > LTS
>> > > > >> > >> > > > > > > >> guest in Virtualbox).
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> The problems I saw in JMeter 3:
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> - java.net.SocketException: Permission
>> denied:
>> > > > >connect
>> > > > >> > >> > > > > > > >> - After a while it runs out of heap space
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> For "Permission denied: connect" I tried to
>> set
>> > > > >> > >> > > > > > > >> java.net.preferIPv4Stack=true
>> > > > >> > >> > > > > > > >> (in a few places to make sure it would
>> really be
>> > > > >in
>> > > > >> > >use) and
>> > > > >> > >> > > tried
>> > > > >> > >> > > > > to
>> > > > >> > >> > > > > > > run
>> > > > >> > >> > > > > > > >> JMeter as administrator (in Win). No effect.
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> I also tried increasing heap size, but that
>> > > > >didn't
>> > > > >> > >help
>> > > > >> > >> > either.
>> > > > >> > >> > > > And
>> > > > >> > >> > > > > I
>> > > > >> > >> > > > > > > >> think
>> > > > >> > >> > > > > > > >> memory size is not the root cause here -
>> rather
>> > > > >the
>> > > > >> > >result.
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> I am normally connected to VPN but I also
>> tried
>> > > > >> > >disabling
>> > > > >> > >> it.
>> > > > >> > >> > No
>> > > > >> > >> > > > > > effect.
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> I tried disabling with and without IPv6
>> enabled
>> > > > >in
>> > > > >> > >Windows
>> > > > >> > >> > > network
>> > > > >> > >> > > > > > > >> connection settings. No effect.
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> I am running a few extra JMeter plugins.
>> Tried
>> > > > >cutting
>> > > > >> > >my
>> > > > >> > >> test
>> > > > >> > >> > > > > > scenario
>> > > > >> > >> > > > > > > to
>> > > > >> > >> > > > > > > >> minimum. No effect.
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> I am running anti-virus software that I
>> cannot
>> > > > >> > >disable, due
>> > > > >> > >> to
>> > > > >> > >> > > > this
>> > > > >> > >> > > > > > > being
>> > > > >> > >> > > > > > > >> forced by IT department. However, it's not a
>> > > > >problem
>> > > > >> > >with
>> > > > >> > >> 2.13
>> > > > >> > >> > > so
>> > > > >> > >> > > > I
>> > > > >> > >> > > > > > > assume
>> > > > >> > >> > > > > > > >> it shouldn't be a problem here either.
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> FINALLY, once I set HTTP request
>> implementation
>> > > > >from
>> > > > >> > >Apache
>> > > > >> > >> > > > > > HttpClient 4
>> > > > >> > >> > > > > > > >> to
>> > > > >> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes
>> -
>> > > > >CPU
>> > > > >> > >stays
>> > > > >> > >> where
>> > > > >> > >> > > it
>> > > > >> > >> > > > > did
>> > > > >> > >> > > > > > > with
>> > > > >> > >> > > > > > > >> JMeter 2.13.
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win,
>> > > > >update
>> > > > >> > >111 on
>> > > > >> > >> > > Linux).
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> Why does (the recommended) HttpClient 4 cause
>> > > > >problems
>> > > > >> > >here?
>> > > > >> > >> > Is
>> > > > >> > >> > > > > this a
>> > > > >> > >> > > > > > > >> know
>> > > > >> > >> > > > > > > >> bug/issue? JMeter 3.x is running newer
>> version
>> > > > >> > >compared to
>> > > > >> > >> > 2.13,
>> > > > >> > >> > > > but
>> > > > >> > >> > > > > > > what
>> > > > >> > >> > > > > > > >> change in HttpClient 4 might cause the
>> problems
>> > > > >> > >described?
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> Any ideas?
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >> Regards,
>> > > > >> > >> > > > > > > >> Tuukka
>> > > > >> > >> > > > > > > >>
>> > > > >> > >> > > > > > > >
>> > > > >> > >> > > > > > > >
>> > > > >> > >> > > > > > > > --
>> > > > >> > >> > > > > > > > Cordialement.
>> > > > >> > >> > > > > > > > Philippe Mouawad.
>> > > > >> > >> > > > > > > >
>> > > > >> > >> > > > > > > >
>> > > > >> > >> > > > > > > >
>> > > > >> > >> > > > > > > >
>> > > > >> > >> > > > > > >
>> > > > >> > >> > > > > > > --
>> > > > >> > >> > > > > > > Cordialement.
>> > > > >> > >> > > > > > > Philippe Mouawad.
>> > > > >> > >> > > > > > >
>> > > > >> > >> > > > > >
>> > > > >> > >> > > > >
>> > > > >> > >> > > > >
>> > > > >> > >> > > > > --
>> > > > >> > >> > > > > Cordialement.
>> > > > >> > >> > > > > Philippe Mouawad.
>> > > > >> > >> > > > >
>> > > > >> > >> > > >
>> > > > >> > >> > >
>> > > > >> > >> >
>> > > > >> > >>
>> > > > >> > >>
>> > > > >> > >>
>> > > > >> > >> --
>> > > > >> > >> Cordialement.
>> > > > >> > >> Philippe Mouawad.
>> > > > >> > >>
>> > > > >> >
>> > > > >> >
>> > > > >-----------------------------------------------------------
>> ----------
>> > > > >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> > > <javascript:;>
>> > > > >> > For additional commands, e-mail: user-help@jmeter.apache.org
>> > > <javascript:;>
>> > > > >> >
>> > > > >> >
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >> Cordialement.
>> > > > >> Philippe Mouawad.
>> > > > >>
>> > > >
>> > >
>> >
>> >
>> > --
>> > Cordialement.
>> > Philippe Mouawad.
>> >
>>
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>


-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

On Sat, Feb 4, 2017 at 12:58 PM, Tuukka Mustonen <tu...@gmail.com>
wrote:

> On point, Philippe!
>
> I think *httpclient4.idletimeout* is not of relevance here - my server
> sends proper Keep-Alive headers so this option shouldn't have impact. Good
> to know, for sure.
>
In the case of NGinx , it seems no header timeout is sent.
So if you set a value > to 0, you can control the max idle duration of a
connection from Client (JMeter) side to avoid storing too many connections
that won't be reused and can impact JMeter heap.
By default it's not set so no impact.

>
> *httpclient4.validate_after_inactivity* and *httpclient4.time_to_live
> default* values (2 seconds) sound conflicting: TTL being the same as
> validate_after_inactivity leads to connections never getting checked as
> connections are discarded after 2 seconds anyway? So
> validate_after_inactivity is not really in use by default if I understood
> it right.
>
Good catch. It somehow went out of my radar, I remember it was something I
had in mind but it seems I failed to change it.
It's a bug or at least bad default value.

But investigating your issue, I discovered that in fact
"httpclient4.validate_after_inactivity" has no impact due to another bug
here I am also responsible of :-( .
validate() method always return true so JMeter always consider connection
as valid.
So only "httpclient4.time_to_live" does its job here.


>
> So I went on and tried these settings:
>
> 1) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=2000
> -> CRASH
> 2) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=1000
> -> CRASH
> 3) httpclient4.validate_after_inactivity=1 + httpclient4.time_to_live=2000
> -> CRASH
>
> 4) httpclient4.validate_after_inactivity=5000 +
> httpclient4.time_to_live=10000 -> SUCCESS
> 5) httpclient4.validate_after_inactivity=0 +
> httpclient4.time_to_live=10000
> -> SUCCESS
> 6) httpclient4.validate_after_inactivity=1 +
> httpclient4.time_to_live=10000
> -> SUCCESS
> 7) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=0 ->
> SUCCESS
> 8) httpclient4.validate_after_inactivity=10000 +
> httpclient4.time_to_live=10000 -> SUCCESS
>
> Clearly increasing TTL makes the difference for me.
> validate_after_inactivity seems to have no effect. Although there is some
> packet loss, the connections themselves don't break.
>
Absolutely.

>
> A) So... maybe my network chip configuration just cannot tolerate JMeter
> opening fresh connections every 2 seconds? That translates to 250
> connections/s with 500 users- maybe it's enough to bloat my network IO and
> hit some policy (and I'll then get "connect: Permission denied")? Sounds a
> bit weird.
>

It's also your OS, you need to tune the TCP stack to work with port reuse
speed.
What's happening here is that you're most probably saturating the dynamic
port range

>
> So I also tried:
>
> 9) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=1 ->
> CRASH
>
> And it crashes quite instantly. I guess it's because JMeter is opening a
> new connection per request...
>
It's because TTL is > 0, so it will be included in the management of idle
connections.

>
> B) ...but if I keep the default 2s settings and just disable Keep-Alive,
> then JMeter is opening new connection per request also, and it works. Why
> the difference?
>
Because JMeter will close connection immediately and inform server to do
the same.


>
> C) Then, I wonder if your "standard workstation" is supposed to tolerate
> these defaults? Can my network chip/driver/configuration just be so bad?
>

Yes at least your OS Tuning defaults are not well suited for Load Testing,
you need to tune.

>
> I see the *httpclient4.validate_after_inactivity* and
> *httpclient4.time_to_live
> default* properties mentioned in the JMeter 3 changelog:
>
> "Improved and better performing validation mechanism for Stale connections
> and Keep-Alive management, see properties
> httpclient4.validate_after_inactivity and httpclient4.time_to_live"
>

Yes, it was the intention.

>
> And when I monitored TCP connection in Wireshark, JMeter 2.13 indeed keeps
> connections open waaaay longer (seemed a bit random).
>

Yes, at that time the only solution was to play with idletimeout or set
"http.connection.stalecheck$Boolean=true" which had important impact at
high throughput.

>
> D) So I the new 2s default sounds a bit risky?
>
In your case yes, but you're in a "bad load testing config" :-)

>
> In the end, increasing TTL seems to fix the issues for me and I can proceed
> upgrading to JMeter 3.
>
> Would love to hear your opinion/analysis on the open questions (A-D), but
> big thanks for your interest in my problem and the precious help already at
> this point!
>

Thanks a lot for all your tests which helped us narrow this issue.
Besides Vladimir and Felix are working on improving performances in GUI
Mode but the team still advises against using it for Load Testing.


> Tuukka
>
>
>
> On Fri, Feb 3, 2017 at 9:40 PM, Philippe Mouawad <
> philippe.mouawad@gmail.com
> > wrote:
>
> > On Friday, February 3, 2017, Tuukka Mustonen <tu...@gmail.com>
> > wrote:
> >
> > > Sorry for the delay, got too busy yesterday.
> > >
> > > I have been running more tests now. I tried to a few variations to
> > pinpoint
> > > the issue, but first:
> > >
> > > @Philippe:
> > >
> > > 0) Response headers are like:
> > >
> > > Response headers:
> > > HTTP/1.1 200 OK
> > > Date: Fri, 03 Feb 2017 06:47:00 GMT
> > > Content-Type: application/json
> > > Content-Length: 374
> > > Connection: keep-alive
> > > Server: nginx/1.6.2
> > > ETag: 37926:1-37927:1
> > > Cache-Control: no-cache, no-store
> > > Pragma: no-cache
> > > Expires: 0
> > > X-Transaction: 0000-326srxia31tss3tc
> >
> >
> > As you pointed it later that issue is related to keep alive.
> > This is tunable , I 'll try to provide some advices
> >
> > But read:
> > http://jmeter.apache.org/usermanual/properties_
> reference.html#httpclient4
> >
> > And pay attention to:
> >
> > httpclient4.idletimeout
> > httpclient4.validate_after_inactivity
> > httpclient4.time_to_live
> >
> > It appears Default values do not suite your case.
> >
> >
> > > 1) Tried your test plan. Unfortunately, it does not make difference.
> >
> > it was not made to fix issue only to make it simpler and indépendant of
> any
> > 3rd party plugin
> >
> > >
> > > 2) I ran Wireshark during a couple of tests. You can find a few
> > screenshots
> > > in a later link. Note that my whole network kinda crashes when JMeter 3
> > > gets stuck (ping gets "General failure" and some active connections
> > crash,
> > > note that this seems a bit random).
> >
> > Thanks
> >
> > >
> > > 3) I gave it a few minutes to recover between the runs.
> > >
> > > @Felix:
> > >
> > > I ran all tests with loggerpanel.usejsyntaxtext=false.
> > >
> > > No, the amount of seems roughly the same.
> > >
> > >
> > >
> > > Then, about pinpointing the issue:
> > >
> > > I managed to reproduce this by installing nginx on a network computer
> > (sudo
> > > apt-get install nginx on debian 8). I did not configure anything and I
> > > simply bombarded the default nginx welcome page.
> > >
> > > I have attached Heap Dump, Thread Dumps and some other information in
> > these
> > > links:
> > >
> > > JAVA impl (that doesn't crash):
> > > https://drive.google.com/drive/folders/0B1uBdVuLED5NSFJvc2VSdE13a0k?
> > > usp=sharing
> > >
> > > HttpCilent 4 impl (that crashes):
> > > https://drive.google.com/drive/folders/0B1uBdVuLED5NUGc1WWFYYkRncTQ?
> > > usp=sharing
> > >
> > > I used Philippe's test plan and even simplified it a bit, the JMX file
> is
> > > in the latter link. Note that the errors occur (in my case) during the
> > > ramp-up so I also shortened to run just 30s after reaching 500 users.
> > >
> > > With JAVA impl, I ran heap dump at ~450 users or so, and thread dumps
> > after
> > > that.
> > >
> > > With HttpClient 4 impl, I ran heap dump immediately after seeing the
> > first
> > > error (which was at ~430 users or so) and thread dumps right after that
> > > (the first two dumps I think took a bit longer than normally to
> create).
> > > JMeter goes unresponsive a second or two after seeing the first error,
> > so I
> > > just barely had time to click to take the dump.
> > >
> > > As said, there are some screenshots from Wireshark trace for
> HttpClient 4
> > > run. I am not network expert, so I don't dare to judge them, but I
> > included
> > > what I saw interesting. I think I can share some parts (privately)
> should
> > > you want to study the trace in detail.
> > >
> > > Finally, when I disabled keep-alive from nginx configuration, the
> problem
> > > was gone (as earlier). So, the problem is somehow related to
> keep-alive.
> > >
> > > Note that in all scenarios CPU goes to 100% quite immediately (also on
> > 2.13
> > > now, probably because of high throughput, I assume my workstation is
> the
> > > bottleneck here) but without keep-alive or with JAVA impl, things work
> > and
> > > with keep-alive enabled and HttpClient 4, things break apart. As
> before,
> > on
> > > JMeter 2.13 everything works.
> > >
> > > Tuukka
> > >
> > >
> > >
> > > On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <
> > > felix.schumacher@internetallee.de <javascript:;>> wrote:
> > >
> > > >
> > > >
> > > > Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
> > > > tuukka.mustonen@gmail.com <javascript:;>>:
> > > > >@Philippe:
> > > > >
> > > > >0. Alright, I will.
> > > > >1. Will do.
> > > > >2. Sure.
> > > > >3. To let the network connection state restore? Ok, will do.
> > > > >
> > > > >I am having some other stuff I need to attend today, but I will get
> > > > >back to
> > > > >this in the afternoon at latest.
> > > >
> > > > Could you also try to set loggerpanel.usejsyntaxtext to false in your
> > > > settings?
> > > >
> > > > Are there more log entries, when you use httpclient 4 instead of
> Java?
> > > >
> > > > Vladimir has found signs for a event congestion in the awt event
> queue,
> > > > that show in your threaddumps.
> > > >
> > > > Regards,
> > > >  Felix
> > > >
> > > > >
> > > > >Tuukka
> > > > >
> > > > >
> > > > >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
> > > > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> > > > >
> > > > >> Hello,
> > > > >> I've created this bugzilla for investigation and to share with you
> > > > >the plan
> > > > >> I have used to run my own tests:
> > > > >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
> > > > >>
> > > > >> I use a Tomcat 8.5.11 with a localhost and a certificate generated
> > > > >and auto
> > > > >> signed.
> > > > >>
> > > > >> So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan
> 2.5
> > > > >Ghz
> > > > >> Intel Core i7
> > > > >> Java 8 u121
> > > > >> -Xmx512m -Xms512m
> > > > >> Test in GUI mode (ANTI-PATTERN)
> > > > >>
> > > > >> 2.13:
> > > > >> - CPU is at 100%
> > > > >> - No error
> > > > >> - No leak
> > > > >>
> > > > >> 3.2 nightly with default settings :
> > > > >> - CPU is at 100%
> > > > >> - No error
> > > > >> - No leak
> > > > >>
> > > > >> 3.2 nightly with this in user.properties :
> > > > >> httpclient4.time_to_live=0
> > > > >> httpclient4.validate_after_inactivity=0
> > > > >>
> > > > >> - CPU is at 100%
> > > > >> - No error
> > > > >> - No leak
> > > > >>
> > > > >> I attached the Heap Graphs to bugzilla and the plan.
> > > > >>
> > > > >> So for now I don't  reproduce your issue but I am not in the same
> > > > >> configuration as I am not hitting the same website.
> > > > >>
> > > > >> So
> > > > >> 0/ Could you provide by running only 5 requests, the response
> header
> > > > >that
> > > > >> your website returns ?
> > > > >> 1/ Could you try my test plan by replacing with your target host
> and
> > > > >see if
> > > > >> behaviour is the same ?
> > > > >> 2/ Ensure nothing is running (No Spotify :-) ) except your JMeter
> > and
> > > > >> monitor network as per Felix note
> > > > >> 3/ Run 1 test after the other but ensure you wait a bit between
> each
> > > > >one
> > > > >>
> > > > >>
> > > > >> Thanks
> > > > >>
> > > > >>
> > > > >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
> > > > >> felix.schumacher@internetallee.de <javascript:;>> wrote:
> > > > >>
> > > > >> >
> > > > >> >
> > > > >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
> > > > >> > tuukka.mustonen@gmail.com <javascript:;>>:
> > > > >> > >Ok I re-ran the test.
> > > > >> > >
> > > > >> > >Took Heap Dump just before JMeter froze.
> > > > >> > >
> > > > >> > >Took Thread Dumps right after JMeter froze.
> > > > >> > >
> > > > >> >
> > > > >>https://drive.google.com/drive/folders/
> 0B1uBdVuLED5NNFNqallGam13MTA
> > > > >> >
> > > > >> > That folder seems to be empty now.
> > > > >> >
> > > > >> > Are you running the tests with a security manager enabled?
> > > > >> >
> > > > >> > Can you share the test or at least the used requests?
> > > > >> >
> > > > >> > Are you using a proxy and if so, how is it configured?
> > > > >> >
> > > > >> > Is jmeter running locally, or over something like rdp?
> > > > >> >
> > > > >> > Can you look at the used network connections like netstat -etapn
> > > > >would do
> > > > >> > on Linux?
> > > > >> >
> > > > >> > Regards,
> > > > >> >  Felix
> > > > >> >
> > > > >> > >
> > > > >> > >...
> > > > >> > >
> > > > >> > >For the fun of it, I also tried upgrading to the latest
> > HttpClient
> > > > >> > >4.5.3
> > > > >> > >version. No effect.
> > > > >> > >
> > > > >> > >I also noticed that if I ping (google) during the perf testing,
> > > > >Windows
> > > > >> > >ping reports "General failure." when JMeter breaks. After
> killing
> > > > >it,
> > > > >> > >connectivity restores and there are replies from the server
> > again.
> > > > >It
> > > > >> > >seems
> > > > >> > >like JMeter/HttpClient kills my whole network connection...
> > > > >> > >
> > > > >> > >Tuukka
> > > > >> > >
> > > > >> > >
> > > > >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > > > >> > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> > > > >> > >
> > > > >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> > > > >> > ><tuukka.mustonen@gmail.com <javascript:;>
> > > > >> > >> >
> > > > >> > >> wrote:
> > > > >> > >>
> > > > >> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I
> think
> > I
> > > > >saw
> > > > >> > >> slightly
> > > > >> > >> > different behavior in my Linux guest. I'll write more about
> > > > >that
> > > > >> > >later,
> > > > >> > >> but
> > > > >> > >> > for simplicity I'm testing only on Windows host now:
> > > > >> > >> >
> > > > >> > >> > Thread dumps are downloadable at:
> > > > >> > >> >
> > > > >https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> > > > >> > >> >
> > > > >> > >> > (Took them with 7-8s interval I think, hope it's ok).
> > > > >> > >> >
> > > > >> > >> > I took the thread dumps once JMeter started filling in
> > errors,
> > > > >> > >which is
> > > > >> > >> > somewhat after my CPU usage hit 100% (and music playback
> > > > >started to
> > > > >> > >> distort
> > > > >> > >> > etc.).
> > > > >> > >> >
> > > > >> > >> > I also created the head dump, note that I took it after
> > > > >running the
> > > > >> > >test
> > > > >> > >> > and after restarting JMeter, as JMeter froze during the
> test
> > > > >and I
> > > > >> > >had to
> > > > >> > >> > kill it. DId you mean I should take it *during* the test?
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> YES.
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> > Anyway, before
> > > > >> > >> > sharing it I want to make sure it can't contain anything
> > > > >> > >confidential?
> > > > >> > >> > (parts of my test plan, passwords, environment variables,
> > > > >> > >anything...)
> > > > >> > >> >
> > > > >> > >> > Tuukka
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> >
> > > > >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > > > >> > >> > support@ubikloadpack.com <javascript:;>> wrote:
> > > > >> > >> >
> > > > >> > >> > > Hello,
> > > > >> > >> > > To investigate the problem:
> > > > >> > >> > > 1/ to create  thread dumps, install a JDK of same version
> > as
> > > > >the
> > > > >> > >one
> > > > >> > >> > > running your JMeter.
> > > > >> > >> > >
> > > > >> > >> > > Run until you get the problem, then call using same user
> as
> > > > >the
> > > > >> > >one
> > > > >> > >> > running
> > > > >> > >> > > JMeter 3 times at 5s interval:
> > > > >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >>
> > output_file.txt
> > > > >> > >> > >
> > > > >> > >> > > It will also help if you provide the structure of your
> Test
> > > > >Plan.
> > > > >> > >> > >
> > > > >> > >> > > 2/ It would also help if you can provide a Heap Dump.
> With
> > > > >JMeter
> > > > >> > >GUI,
> > > > >> > >> > use
> > > > >> > >> > > menu Help > Create a Heap Dump
> > > > >> > >> > > This will create a Heap Dump (rather big file) , it would
> > > > >help a
> > > > >> > >lot if
> > > > >> > >> > you
> > > > >> > >> > > can make it available.
> > > > >> > >> > >
> > > > >> > >> > > Regards
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > >
> > > > >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > > > >> > >> > tuukka.mustonen@gmail.com <javascript:;>
> > > > >> > >> > > >
> > > > >> > >> > > wrote:
> > > > >> > >> > >
> > > > >> > >> > > > @Philippe:
> > > > >> > >> > > >
> > > > >> > >> > > > Yeah, I am normally running in non-GUI mode, but to
> > > > >> > >> > > > study/inspect/understand this problem I am running in
> GUI
> > > > >mode.
> > > > >> > >> JMeter
> > > > >> > >> > > 2.13
> > > > >> > >> > > > works just fine in GUI mode for these tests so JMeter
> 3.x
> > > > >> > >should too.
> > > > >> > >> > > >
> > > > >> > >> > > > I'll give the nightly build a try.
> > > > >> > >> > > >
> > > > >> > >> > > > Tuukka
> > > > >> > >> > > >
> > > > >> > >> > > >
> > > > >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > > > >> > >> > > > philippe.mouawad@gmail.com <javascript:;>
> > > > >> > >> > > > > wrote:
> > > > >> > >> > > >
> > > > >> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > >> > >> > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > >> > >> > > > > >
> > > > >> > >> > > > > wrote:
> > > > >> > >> > > > >
> > > > >> > >> > > > > > @Philippe:
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > 1. Yeah, I compared them with the same plan, same
> > > > >machine.
> > > > >> > >Should
> > > > >> > >> > be
> > > > >> > >> > > > > > everything equal.
> > > > >> > >> > > > > > 2. I tried both the defaults in jmeter.bat
> (-Xms512m
> > > > >> > >-Xmx512m)
> > > > >> > >> and
> > > > >> > >> > as
> > > > >> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't
> > > > >tweak
> > > > >> > >them much
> > > > >> > >> > > > > because I
> > > > >> > >> > > > > > don't believe the issue lies there.
> > > > >> > >> > > > > > 3. No, I do just simple HTTP GETs (against RESTy
> > API).
> > > > >> > >> > > > > > 4. I am not familiar with that but I can try. If
> you
> > > > >have
> > > > >> > >> > > instructions
> > > > >> > >> > > > to
> > > > >> > >> > > > > > point to, please do.
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > > > 5. I am running with GUI.
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > > This is a known anti-pattern .
> > > > >> > >> > > > > switch to non gui mode if you want correct results.
> > > > >> > >> > > > > http://jmeter.apache.org/
> > usermanual/best-practices.html
> > > > >> > >> > > > >
> > > > >> > >> > > > > http://www.ubik-ingenierie.
> > com/blog/jmeter_performance_
> > > > >> > >> tuning_tips/
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > > > About file sending:
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
> > > > >> > >> > > > >
> > > > >> > >> > > > >  Could you give nightly build a try ?
> > > > >> > >> > > > > http://jmeter.apache.org/nightly.html
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > > > Tuukka
> > > > >> > >> > > > > >
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > > >> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> > > <javascript:;>
> > > > >> > >> > > > > > > wrote:
> > > > >> > >> > > > > >
> > > > >> > >> > > > > > > Also are you sending files in post body ?
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > > Thank you
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad
> <
> > > > >> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>
> > > <javascript:;>>
> > > > >> > >> > > > > > > wrote:
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > > > Hello,
> > > > >> > >> > > > > > > > Few questions, please try to answer them all:
> > > > >> > >> > > > > > > >
> > > > >> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same
> > > > >plan/sape
> > > > >> > >> machine...
> > > > >> > >> > > > > > everything
> > > > >> > >> > > > > > > > equal ?
> > > > >> > >> > > > > > > > 2/ What are your Xmx settings
> > > > >> > >> > > > > > > > 3/ does your plan download embedded resources ?
> > > > >> > >> > > > > > > > 4/ when issue happens can you make 3 thread
> dumps
> > > > >at 5
> > > > >> > >> seconds
> > > > >> > >> > > > > interval
> > > > >> > >> > > > > > > > and send file ? (use jstack)
> > > > >> > >> > > > > > > > 5/ are you running gui or non gui ?
> > > > >> > >> > > > > > > >
> > > > >> > >> > > > > > > > Thanks
> > > > >> > >> > > > > > > > Regards
> > > > >> > >> > > > > > > >
> > > > >> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka
> Mustonen <
> > > > >> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > <javascript:;>
> > > > >> > >> > > > > > > >
> > > > ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > <javascript:;>
> > > > >> > >> > > > > <javascript:;>');>>
> > > > >> > >> > > > > > wrote:
> > > > >> > >> > > > > > > >
> > > > >> > >> > > > > > > >> Hi,
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I
> > experience
> > > > >high
> > > > >> > >CPU
> > > > >> > >> > usage
> > > > >> > >> > > > > that
> > > > >> > >> > > > > > > >> renders JMeter unusable. CPU usage ramps
> quickly
> > > > >up
> > > > >> > >and
> > > > >> > >> after
> > > > >> > >> > > > having
> > > > >> > >> > > > > > > >> ~100-200 concurrent users or so (in ~30
> > seconds),
> > > > >it
> > > > >> > >hits
> > > > >> > >> 100%
> > > > >> > >> > > on
> > > > >> > >> > > > my
> > > > >> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes
> and
> > > > >I
> > > > >> > >have to
> > > > >> > >> > kill
> > > > >> > >> > > > it.
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> On 2.13 everything goes smoothly (with default
> > > > >> > >settings).
> > > > >> > >> CPU
> > > > >> > >> > > > usage
> > > > >> > >> > > > > > > stays
> > > > >> > >> > > > > > > >> at ~30% even with 500 users.
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> The experience is similar on both Windows 10
> and
> > > > >Linux
> > > > >> > >> (Ubuntu
> > > > >> > >> > > > 14.04
> > > > >> > >> > > > > > LTS
> > > > >> > >> > > > > > > >> guest in Virtualbox).
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> The problems I saw in JMeter 3:
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> - java.net.SocketException: Permission denied:
> > > > >connect
> > > > >> > >> > > > > > > >> - After a while it runs out of heap space
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> For "Permission denied: connect" I tried to
> set
> > > > >> > >> > > > > > > >> java.net.preferIPv4Stack=true
> > > > >> > >> > > > > > > >> (in a few places to make sure it would really
> be
> > > > >in
> > > > >> > >use) and
> > > > >> > >> > > tried
> > > > >> > >> > > > > to
> > > > >> > >> > > > > > > run
> > > > >> > >> > > > > > > >> JMeter as administrator (in Win). No effect.
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> I also tried increasing heap size, but that
> > > > >didn't
> > > > >> > >help
> > > > >> > >> > either.
> > > > >> > >> > > > And
> > > > >> > >> > > > > I
> > > > >> > >> > > > > > > >> think
> > > > >> > >> > > > > > > >> memory size is not the root cause here -
> rather
> > > > >the
> > > > >> > >result.
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> I am normally connected to VPN but I also
> tried
> > > > >> > >disabling
> > > > >> > >> it.
> > > > >> > >> > No
> > > > >> > >> > > > > > effect.
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> I tried disabling with and without IPv6
> enabled
> > > > >in
> > > > >> > >Windows
> > > > >> > >> > > network
> > > > >> > >> > > > > > > >> connection settings. No effect.
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> I am running a few extra JMeter plugins. Tried
> > > > >cutting
> > > > >> > >my
> > > > >> > >> test
> > > > >> > >> > > > > > scenario
> > > > >> > >> > > > > > > to
> > > > >> > >> > > > > > > >> minimum. No effect.
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> I am running anti-virus software that I cannot
> > > > >> > >disable, due
> > > > >> > >> to
> > > > >> > >> > > > this
> > > > >> > >> > > > > > > being
> > > > >> > >> > > > > > > >> forced by IT department. However, it's not a
> > > > >problem
> > > > >> > >with
> > > > >> > >> 2.13
> > > > >> > >> > > so
> > > > >> > >> > > > I
> > > > >> > >> > > > > > > assume
> > > > >> > >> > > > > > > >> it shouldn't be a problem here either.
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> FINALLY, once I set HTTP request
> implementation
> > > > >from
> > > > >> > >Apache
> > > > >> > >> > > > > > HttpClient 4
> > > > >> > >> > > > > > > >> to
> > > > >> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes -
> > > > >CPU
> > > > >> > >stays
> > > > >> > >> where
> > > > >> > >> > > it
> > > > >> > >> > > > > did
> > > > >> > >> > > > > > > with
> > > > >> > >> > > > > > > >> JMeter 2.13.
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win,
> > > > >update
> > > > >> > >111 on
> > > > >> > >> > > Linux).
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> Why does (the recommended) HttpClient 4 cause
> > > > >problems
> > > > >> > >here?
> > > > >> > >> > Is
> > > > >> > >> > > > > this a
> > > > >> > >> > > > > > > >> know
> > > > >> > >> > > > > > > >> bug/issue? JMeter 3.x is running newer version
> > > > >> > >compared to
> > > > >> > >> > 2.13,
> > > > >> > >> > > > but
> > > > >> > >> > > > > > > what
> > > > >> > >> > > > > > > >> change in HttpClient 4 might cause the
> problems
> > > > >> > >described?
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> Any ideas?
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >> Regards,
> > > > >> > >> > > > > > > >> Tuukka
> > > > >> > >> > > > > > > >>
> > > > >> > >> > > > > > > >
> > > > >> > >> > > > > > > >
> > > > >> > >> > > > > > > > --
> > > > >> > >> > > > > > > > Cordialement.
> > > > >> > >> > > > > > > > Philippe Mouawad.
> > > > >> > >> > > > > > > >
> > > > >> > >> > > > > > > >
> > > > >> > >> > > > > > > >
> > > > >> > >> > > > > > > >
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > > > --
> > > > >> > >> > > > > > > Cordialement.
> > > > >> > >> > > > > > > Philippe Mouawad.
> > > > >> > >> > > > > > >
> > > > >> > >> > > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > >
> > > > >> > >> > > > > --
> > > > >> > >> > > > > Cordialement.
> > > > >> > >> > > > > Philippe Mouawad.
> > > > >> > >> > > > >
> > > > >> > >> > > >
> > > > >> > >> > >
> > > > >> > >> >
> > > > >> > >>
> > > > >> > >>
> > > > >> > >>
> > > > >> > >> --
> > > > >> > >> Cordialement.
> > > > >> > >> Philippe Mouawad.
> > > > >> > >>
> > > > >> >
> > > > >> >
> > > > >-----------------------------------------------------------
> ----------
> > > > >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > > <javascript:;>
> > > > >> > For additional commands, e-mail: user-help@jmeter.apache.org
> > > <javascript:;>
> > > > >> >
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Cordialement.
> > > > >> Philippe Mouawad.
> > > > >>
> > > >
> > >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
>



-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

On point, Philippe!

I think *httpclient4.idletimeout* is not of relevance here - my server
sends proper Keep-Alive headers so this option shouldn't have impact. Good
to know, for sure.

*httpclient4.validate_after_inactivity* and *httpclient4.time_to_live
default* values (2 seconds) sound conflicting: TTL being the same as
validate_after_inactivity leads to connections never getting checked as
connections are discarded after 2 seconds anyway? So
validate_after_inactivity is not really in use by default if I understood
it right.

So I went on and tried these settings:

1) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=2000
-> CRASH
2) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=1000
-> CRASH
3) httpclient4.validate_after_inactivity=1 + httpclient4.time_to_live=2000
-> CRASH

4) httpclient4.validate_after_inactivity=5000 +
httpclient4.time_to_live=10000 -> SUCCESS
5) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=10000
-> SUCCESS
6) httpclient4.validate_after_inactivity=1 + httpclient4.time_to_live=10000
-> SUCCESS
7) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=0 ->
SUCCESS
8) httpclient4.validate_after_inactivity=10000 +
httpclient4.time_to_live=10000 -> SUCCESS

Clearly increasing TTL makes the difference for me.
validate_after_inactivity seems to have no effect. Although there is some
packet loss, the connections themselves don't break.

A) So... maybe my network chip configuration just cannot tolerate JMeter
opening fresh connections every 2 seconds? That translates to 250
connections/s with 500 users- maybe it's enough to bloat my network IO and
hit some policy (and I'll then get "connect: Permission denied")? Sounds a
bit weird.

So I also tried:

9) httpclient4.validate_after_inactivity=0 + httpclient4.time_to_live=1 ->
CRASH

And it crashes quite instantly. I guess it's because JMeter is opening a
new connection per request...

B) ...but if I keep the default 2s settings and just disable Keep-Alive,
then JMeter is opening new connection per request also, and it works. Why
the difference?

C) Then, I wonder if your "standard workstation" is supposed to tolerate
these defaults? Can my network chip/driver/configuration just be so bad?

I see the *httpclient4.validate_after_inactivity* and *httpclient4.time_to_live
default* properties mentioned in the JMeter 3 changelog:

"Improved and better performing validation mechanism for Stale connections
and Keep-Alive management, see properties
httpclient4.validate_after_inactivity and httpclient4.time_to_live"

And when I monitored TCP connection in Wireshark, JMeter 2.13 indeed keeps
connections open waaaay longer (seemed a bit random).

D) So I the new 2s default sounds a bit risky?

In the end, increasing TTL seems to fix the issues for me and I can proceed
upgrading to JMeter 3.

Would love to hear your opinion/analysis on the open questions (A-D), but
big thanks for your interest in my problem and the precious help already at
this point!

Tuukka



On Fri, Feb 3, 2017 at 9:40 PM, Philippe Mouawad <philippe.mouawad@gmail.com
> wrote:

> On Friday, February 3, 2017, Tuukka Mustonen <tu...@gmail.com>
> wrote:
>
> > Sorry for the delay, got too busy yesterday.
> >
> > I have been running more tests now. I tried to a few variations to
> pinpoint
> > the issue, but first:
> >
> > @Philippe:
> >
> > 0) Response headers are like:
> >
> > Response headers:
> > HTTP/1.1 200 OK
> > Date: Fri, 03 Feb 2017 06:47:00 GMT
> > Content-Type: application/json
> > Content-Length: 374
> > Connection: keep-alive
> > Server: nginx/1.6.2
> > ETag: 37926:1-37927:1
> > Cache-Control: no-cache, no-store
> > Pragma: no-cache
> > Expires: 0
> > X-Transaction: 0000-326srxia31tss3tc
>
>
> As you pointed it later that issue is related to keep alive.
> This is tunable , I 'll try to provide some advices
>
> But read:
> http://jmeter.apache.org/usermanual/properties_reference.html#httpclient4
>
> And pay attention to:
>
> httpclient4.idletimeout
> httpclient4.validate_after_inactivity
> httpclient4.time_to_live
>
> It appears Default values do not suite your case.
>
>
> > 1) Tried your test plan. Unfortunately, it does not make difference.
>
> it was not made to fix issue only to make it simpler and indépendant of any
> 3rd party plugin
>
> >
> > 2) I ran Wireshark during a couple of tests. You can find a few
> screenshots
> > in a later link. Note that my whole network kinda crashes when JMeter 3
> > gets stuck (ping gets "General failure" and some active connections
> crash,
> > note that this seems a bit random).
>
> Thanks
>
> >
> > 3) I gave it a few minutes to recover between the runs.
> >
> > @Felix:
> >
> > I ran all tests with loggerpanel.usejsyntaxtext=false.
> >
> > No, the amount of seems roughly the same.
> >
> >
> >
> > Then, about pinpointing the issue:
> >
> > I managed to reproduce this by installing nginx on a network computer
> (sudo
> > apt-get install nginx on debian 8). I did not configure anything and I
> > simply bombarded the default nginx welcome page.
> >
> > I have attached Heap Dump, Thread Dumps and some other information in
> these
> > links:
> >
> > JAVA impl (that doesn't crash):
> > https://drive.google.com/drive/folders/0B1uBdVuLED5NSFJvc2VSdE13a0k?
> > usp=sharing
> >
> > HttpCilent 4 impl (that crashes):
> > https://drive.google.com/drive/folders/0B1uBdVuLED5NUGc1WWFYYkRncTQ?
> > usp=sharing
> >
> > I used Philippe's test plan and even simplified it a bit, the JMX file is
> > in the latter link. Note that the errors occur (in my case) during the
> > ramp-up so I also shortened to run just 30s after reaching 500 users.
> >
> > With JAVA impl, I ran heap dump at ~450 users or so, and thread dumps
> after
> > that.
> >
> > With HttpClient 4 impl, I ran heap dump immediately after seeing the
> first
> > error (which was at ~430 users or so) and thread dumps right after that
> > (the first two dumps I think took a bit longer than normally to create).
> > JMeter goes unresponsive a second or two after seeing the first error,
> so I
> > just barely had time to click to take the dump.
> >
> > As said, there are some screenshots from Wireshark trace for HttpClient 4
> > run. I am not network expert, so I don't dare to judge them, but I
> included
> > what I saw interesting. I think I can share some parts (privately) should
> > you want to study the trace in detail.
> >
> > Finally, when I disabled keep-alive from nginx configuration, the problem
> > was gone (as earlier). So, the problem is somehow related to keep-alive.
> >
> > Note that in all scenarios CPU goes to 100% quite immediately (also on
> 2.13
> > now, probably because of high throughput, I assume my workstation is the
> > bottleneck here) but without keep-alive or with JAVA impl, things work
> and
> > with keep-alive enabled and HttpClient 4, things break apart. As before,
> on
> > JMeter 2.13 everything works.
> >
> > Tuukka
> >
> >
> >
> > On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <
> > felix.schumacher@internetallee.de <javascript:;>> wrote:
> >
> > >
> > >
> > > Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
> > > tuukka.mustonen@gmail.com <javascript:;>>:
> > > >@Philippe:
> > > >
> > > >0. Alright, I will.
> > > >1. Will do.
> > > >2. Sure.
> > > >3. To let the network connection state restore? Ok, will do.
> > > >
> > > >I am having some other stuff I need to attend today, but I will get
> > > >back to
> > > >this in the afternoon at latest.
> > >
> > > Could you also try to set loggerpanel.usejsyntaxtext to false in your
> > > settings?
> > >
> > > Are there more log entries, when you use httpclient 4 instead of Java?
> > >
> > > Vladimir has found signs for a event congestion in the awt event queue,
> > > that show in your threaddumps.
> > >
> > > Regards,
> > >  Felix
> > >
> > > >
> > > >Tuukka
> > > >
> > > >
> > > >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
> > > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> > > >
> > > >> Hello,
> > > >> I've created this bugzilla for investigation and to share with you
> > > >the plan
> > > >> I have used to run my own tests:
> > > >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
> > > >>
> > > >> I use a Tomcat 8.5.11 with a localhost and a certificate generated
> > > >and auto
> > > >> signed.
> > > >>
> > > >> So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan 2.5
> > > >Ghz
> > > >> Intel Core i7
> > > >> Java 8 u121
> > > >> -Xmx512m -Xms512m
> > > >> Test in GUI mode (ANTI-PATTERN)
> > > >>
> > > >> 2.13:
> > > >> - CPU is at 100%
> > > >> - No error
> > > >> - No leak
> > > >>
> > > >> 3.2 nightly with default settings :
> > > >> - CPU is at 100%
> > > >> - No error
> > > >> - No leak
> > > >>
> > > >> 3.2 nightly with this in user.properties :
> > > >> httpclient4.time_to_live=0
> > > >> httpclient4.validate_after_inactivity=0
> > > >>
> > > >> - CPU is at 100%
> > > >> - No error
> > > >> - No leak
> > > >>
> > > >> I attached the Heap Graphs to bugzilla and the plan.
> > > >>
> > > >> So for now I don't  reproduce your issue but I am not in the same
> > > >> configuration as I am not hitting the same website.
> > > >>
> > > >> So
> > > >> 0/ Could you provide by running only 5 requests, the response header
> > > >that
> > > >> your website returns ?
> > > >> 1/ Could you try my test plan by replacing with your target host and
> > > >see if
> > > >> behaviour is the same ?
> > > >> 2/ Ensure nothing is running (No Spotify :-) ) except your JMeter
> and
> > > >> monitor network as per Felix note
> > > >> 3/ Run 1 test after the other but ensure you wait a bit between each
> > > >one
> > > >>
> > > >>
> > > >> Thanks
> > > >>
> > > >>
> > > >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
> > > >> felix.schumacher@internetallee.de <javascript:;>> wrote:
> > > >>
> > > >> >
> > > >> >
> > > >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
> > > >> > tuukka.mustonen@gmail.com <javascript:;>>:
> > > >> > >Ok I re-ran the test.
> > > >> > >
> > > >> > >Took Heap Dump just before JMeter froze.
> > > >> > >
> > > >> > >Took Thread Dumps right after JMeter froze.
> > > >> > >
> > > >> >
> > > >>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
> > > >> >
> > > >> > That folder seems to be empty now.
> > > >> >
> > > >> > Are you running the tests with a security manager enabled?
> > > >> >
> > > >> > Can you share the test or at least the used requests?
> > > >> >
> > > >> > Are you using a proxy and if so, how is it configured?
> > > >> >
> > > >> > Is jmeter running locally, or over something like rdp?
> > > >> >
> > > >> > Can you look at the used network connections like netstat -etapn
> > > >would do
> > > >> > on Linux?
> > > >> >
> > > >> > Regards,
> > > >> >  Felix
> > > >> >
> > > >> > >
> > > >> > >...
> > > >> > >
> > > >> > >For the fun of it, I also tried upgrading to the latest
> HttpClient
> > > >> > >4.5.3
> > > >> > >version. No effect.
> > > >> > >
> > > >> > >I also noticed that if I ping (google) during the perf testing,
> > > >Windows
> > > >> > >ping reports "General failure." when JMeter breaks. After killing
> > > >it,
> > > >> > >connectivity restores and there are replies from the server
> again.
> > > >It
> > > >> > >seems
> > > >> > >like JMeter/HttpClient kills my whole network connection...
> > > >> > >
> > > >> > >Tuukka
> > > >> > >
> > > >> > >
> > > >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > > >> > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> > > >> > >
> > > >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> > > >> > ><tuukka.mustonen@gmail.com <javascript:;>
> > > >> > >> >
> > > >> > >> wrote:
> > > >> > >>
> > > >> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I think
> I
> > > >saw
> > > >> > >> slightly
> > > >> > >> > different behavior in my Linux guest. I'll write more about
> > > >that
> > > >> > >later,
> > > >> > >> but
> > > >> > >> > for simplicity I'm testing only on Windows host now:
> > > >> > >> >
> > > >> > >> > Thread dumps are downloadable at:
> > > >> > >> >
> > > >https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> > > >> > >> >
> > > >> > >> > (Took them with 7-8s interval I think, hope it's ok).
> > > >> > >> >
> > > >> > >> > I took the thread dumps once JMeter started filling in
> errors,
> > > >> > >which is
> > > >> > >> > somewhat after my CPU usage hit 100% (and music playback
> > > >started to
> > > >> > >> distort
> > > >> > >> > etc.).
> > > >> > >> >
> > > >> > >> > I also created the head dump, note that I took it after
> > > >running the
> > > >> > >test
> > > >> > >> > and after restarting JMeter, as JMeter froze during the test
> > > >and I
> > > >> > >had to
> > > >> > >> > kill it. DId you mean I should take it *during* the test?
> > > >> > >>
> > > >> > >>
> > > >> > >> YES.
> > > >> > >>
> > > >> > >>
> > > >> > >> > Anyway, before
> > > >> > >> > sharing it I want to make sure it can't contain anything
> > > >> > >confidential?
> > > >> > >> > (parts of my test plan, passwords, environment variables,
> > > >> > >anything...)
> > > >> > >> >
> > > >> > >> > Tuukka
> > > >> > >> >
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > > >> > >> > support@ubikloadpack.com <javascript:;>> wrote:
> > > >> > >> >
> > > >> > >> > > Hello,
> > > >> > >> > > To investigate the problem:
> > > >> > >> > > 1/ to create  thread dumps, install a JDK of same version
> as
> > > >the
> > > >> > >one
> > > >> > >> > > running your JMeter.
> > > >> > >> > >
> > > >> > >> > > Run until you get the problem, then call using same user as
> > > >the
> > > >> > >one
> > > >> > >> > running
> > > >> > >> > > JMeter 3 times at 5s interval:
> > > >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >>
> output_file.txt
> > > >> > >> > >
> > > >> > >> > > It will also help if you provide the structure of your Test
> > > >Plan.
> > > >> > >> > >
> > > >> > >> > > 2/ It would also help if you can provide a Heap Dump. With
> > > >JMeter
> > > >> > >GUI,
> > > >> > >> > use
> > > >> > >> > > menu Help > Create a Heap Dump
> > > >> > >> > > This will create a Heap Dump (rather big file) , it would
> > > >help a
> > > >> > >lot if
> > > >> > >> > you
> > > >> > >> > > can make it available.
> > > >> > >> > >
> > > >> > >> > > Regards
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > >
> > > >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > > >> > >> > tuukka.mustonen@gmail.com <javascript:;>
> > > >> > >> > > >
> > > >> > >> > > wrote:
> > > >> > >> > >
> > > >> > >> > > > @Philippe:
> > > >> > >> > > >
> > > >> > >> > > > Yeah, I am normally running in non-GUI mode, but to
> > > >> > >> > > > study/inspect/understand this problem I am running in GUI
> > > >mode.
> > > >> > >> JMeter
> > > >> > >> > > 2.13
> > > >> > >> > > > works just fine in GUI mode for these tests so JMeter 3.x
> > > >> > >should too.
> > > >> > >> > > >
> > > >> > >> > > > I'll give the nightly build a try.
> > > >> > >> > > >
> > > >> > >> > > > Tuukka
> > > >> > >> > > >
> > > >> > >> > > >
> > > >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > > >> > >> > > > philippe.mouawad@gmail.com <javascript:;>
> > > >> > >> > > > > wrote:
> > > >> > >> > > >
> > > >> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > >> > >> > > > tuukka.mustonen@gmail.com <javascript:;>
> > > >> > >> > > > > >
> > > >> > >> > > > > wrote:
> > > >> > >> > > > >
> > > >> > >> > > > > > @Philippe:
> > > >> > >> > > > > >
> > > >> > >> > > > > > 1. Yeah, I compared them with the same plan, same
> > > >machine.
> > > >> > >Should
> > > >> > >> > be
> > > >> > >> > > > > > everything equal.
> > > >> > >> > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
> > > >> > >-Xmx512m)
> > > >> > >> and
> > > >> > >> > as
> > > >> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't
> > > >tweak
> > > >> > >them much
> > > >> > >> > > > > because I
> > > >> > >> > > > > > don't believe the issue lies there.
> > > >> > >> > > > > > 3. No, I do just simple HTTP GETs (against RESTy
> API).
> > > >> > >> > > > > > 4. I am not familiar with that but I can try. If you
> > > >have
> > > >> > >> > > instructions
> > > >> > >> > > > to
> > > >> > >> > > > > > point to, please do.
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > > > > 5. I am running with GUI.
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > > > This is a known anti-pattern .
> > > >> > >> > > > > switch to non gui mode if you want correct results.
> > > >> > >> > > > > http://jmeter.apache.org/
> usermanual/best-practices.html
> > > >> > >> > > > >
> > > >> > >> > > > > http://www.ubik-ingenierie.
> com/blog/jmeter_performance_
> > > >> > >> tuning_tips/
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > > > > About file sending:
> > > >> > >> > > > > >
> > > >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
> > > >> > >> > > > >
> > > >> > >> > > > >  Could you give nightly build a try ?
> > > >> > >> > > > > http://jmeter.apache.org/nightly.html
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > > > > Tuukka
> > > >> > >> > > > > >
> > > >> > >> > > > > >
> > > >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > >> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> > <javascript:;>
> > > >> > >> > > > > > > wrote:
> > > >> > >> > > > > >
> > > >> > >> > > > > > > Also are you sending files in post body ?
> > > >> > >> > > > > > >
> > > >> > >> > > > > > > Thank you
> > > >> > >> > > > > > >
> > > >> > >> > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > >> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>
> > <javascript:;>>
> > > >> > >> > > > > > > wrote:
> > > >> > >> > > > > > >
> > > >> > >> > > > > > > > Hello,
> > > >> > >> > > > > > > > Few questions, please try to answer them all:
> > > >> > >> > > > > > > >
> > > >> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same
> > > >plan/sape
> > > >> > >> machine...
> > > >> > >> > > > > > everything
> > > >> > >> > > > > > > > equal ?
> > > >> > >> > > > > > > > 2/ What are your Xmx settings
> > > >> > >> > > > > > > > 3/ does your plan download embedded resources ?
> > > >> > >> > > > > > > > 4/ when issue happens can you make 3 thread dumps
> > > >at 5
> > > >> > >> seconds
> > > >> > >> > > > > interval
> > > >> > >> > > > > > > > and send file ? (use jstack)
> > > >> > >> > > > > > > > 5/ are you running gui or non gui ?
> > > >> > >> > > > > > > >
> > > >> > >> > > > > > > > Thanks
> > > >> > >> > > > > > > > Regards
> > > >> > >> > > > > > > >
> > > >> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > >> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > <javascript:;>
> > > >> > >> > > > > > > >
> > > ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> <javascript:;>
> > > >> > >> > > > > <javascript:;>');>>
> > > >> > >> > > > > > wrote:
> > > >> > >> > > > > > > >
> > > >> > >> > > > > > > >> Hi,
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I
> experience
> > > >high
> > > >> > >CPU
> > > >> > >> > usage
> > > >> > >> > > > > that
> > > >> > >> > > > > > > >> renders JMeter unusable. CPU usage ramps quickly
> > > >up
> > > >> > >and
> > > >> > >> after
> > > >> > >> > > > having
> > > >> > >> > > > > > > >> ~100-200 concurrent users or so (in ~30
> seconds),
> > > >it
> > > >> > >hits
> > > >> > >> 100%
> > > >> > >> > > on
> > > >> > >> > > > my
> > > >> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and
> > > >I
> > > >> > >have to
> > > >> > >> > kill
> > > >> > >> > > > it.
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> On 2.13 everything goes smoothly (with default
> > > >> > >settings).
> > > >> > >> CPU
> > > >> > >> > > > usage
> > > >> > >> > > > > > > stays
> > > >> > >> > > > > > > >> at ~30% even with 500 users.
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> The experience is similar on both Windows 10 and
> > > >Linux
> > > >> > >> (Ubuntu
> > > >> > >> > > > 14.04
> > > >> > >> > > > > > LTS
> > > >> > >> > > > > > > >> guest in Virtualbox).
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> The problems I saw in JMeter 3:
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> - java.net.SocketException: Permission denied:
> > > >connect
> > > >> > >> > > > > > > >> - After a while it runs out of heap space
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> For "Permission denied: connect" I tried to set
> > > >> > >> > > > > > > >> java.net.preferIPv4Stack=true
> > > >> > >> > > > > > > >> (in a few places to make sure it would really be
> > > >in
> > > >> > >use) and
> > > >> > >> > > tried
> > > >> > >> > > > > to
> > > >> > >> > > > > > > run
> > > >> > >> > > > > > > >> JMeter as administrator (in Win). No effect.
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> I also tried increasing heap size, but that
> > > >didn't
> > > >> > >help
> > > >> > >> > either.
> > > >> > >> > > > And
> > > >> > >> > > > > I
> > > >> > >> > > > > > > >> think
> > > >> > >> > > > > > > >> memory size is not the root cause here - rather
> > > >the
> > > >> > >result.
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> I am normally connected to VPN but I also tried
> > > >> > >disabling
> > > >> > >> it.
> > > >> > >> > No
> > > >> > >> > > > > > effect.
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> I tried disabling with and without IPv6 enabled
> > > >in
> > > >> > >Windows
> > > >> > >> > > network
> > > >> > >> > > > > > > >> connection settings. No effect.
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> I am running a few extra JMeter plugins. Tried
> > > >cutting
> > > >> > >my
> > > >> > >> test
> > > >> > >> > > > > > scenario
> > > >> > >> > > > > > > to
> > > >> > >> > > > > > > >> minimum. No effect.
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> I am running anti-virus software that I cannot
> > > >> > >disable, due
> > > >> > >> to
> > > >> > >> > > > this
> > > >> > >> > > > > > > being
> > > >> > >> > > > > > > >> forced by IT department. However, it's not a
> > > >problem
> > > >> > >with
> > > >> > >> 2.13
> > > >> > >> > > so
> > > >> > >> > > > I
> > > >> > >> > > > > > > assume
> > > >> > >> > > > > > > >> it shouldn't be a problem here either.
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> FINALLY, once I set HTTP request implementation
> > > >from
> > > >> > >Apache
> > > >> > >> > > > > > HttpClient 4
> > > >> > >> > > > > > > >> to
> > > >> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes -
> > > >CPU
> > > >> > >stays
> > > >> > >> where
> > > >> > >> > > it
> > > >> > >> > > > > did
> > > >> > >> > > > > > > with
> > > >> > >> > > > > > > >> JMeter 2.13.
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win,
> > > >update
> > > >> > >111 on
> > > >> > >> > > Linux).
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> Why does (the recommended) HttpClient 4 cause
> > > >problems
> > > >> > >here?
> > > >> > >> > Is
> > > >> > >> > > > > this a
> > > >> > >> > > > > > > >> know
> > > >> > >> > > > > > > >> bug/issue? JMeter 3.x is running newer version
> > > >> > >compared to
> > > >> > >> > 2.13,
> > > >> > >> > > > but
> > > >> > >> > > > > > > what
> > > >> > >> > > > > > > >> change in HttpClient 4 might cause the problems
> > > >> > >described?
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> Any ideas?
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >> Regards,
> > > >> > >> > > > > > > >> Tuukka
> > > >> > >> > > > > > > >>
> > > >> > >> > > > > > > >
> > > >> > >> > > > > > > >
> > > >> > >> > > > > > > > --
> > > >> > >> > > > > > > > Cordialement.
> > > >> > >> > > > > > > > Philippe Mouawad.
> > > >> > >> > > > > > > >
> > > >> > >> > > > > > > >
> > > >> > >> > > > > > > >
> > > >> > >> > > > > > > >
> > > >> > >> > > > > > >
> > > >> > >> > > > > > > --
> > > >> > >> > > > > > > Cordialement.
> > > >> > >> > > > > > > Philippe Mouawad.
> > > >> > >> > > > > > >
> > > >> > >> > > > > >
> > > >> > >> > > > >
> > > >> > >> > > > >
> > > >> > >> > > > > --
> > > >> > >> > > > > Cordialement.
> > > >> > >> > > > > Philippe Mouawad.
> > > >> > >> > > > >
> > > >> > >> > > >
> > > >> > >> > >
> > > >> > >> >
> > > >> > >>
> > > >> > >>
> > > >> > >>
> > > >> > >> --
> > > >> > >> Cordialement.
> > > >> > >> Philippe Mouawad.
> > > >> > >>
> > > >> >
> > > >> >
> > > >---------------------------------------------------------------------
> > > >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > <javascript:;>
> > > >> > For additional commands, e-mail: user-help@jmeter.apache.org
> > <javascript:;>
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> Cordialement.
> > > >> Philippe Mouawad.
> > > >>
> > >
> >
>
>
> --
> Cordialement.
> Philippe Mouawad.
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

On Friday, February 3, 2017, Tuukka Mustonen <tu...@gmail.com>
wrote:

> Sorry for the delay, got too busy yesterday.
>
> I have been running more tests now. I tried to a few variations to pinpoint
> the issue, but first:
>
> @Philippe:
>
> 0) Response headers are like:
>
> Response headers:
> HTTP/1.1 200 OK
> Date: Fri, 03 Feb 2017 06:47:00 GMT
> Content-Type: application/json
> Content-Length: 374
> Connection: keep-alive
> Server: nginx/1.6.2
> ETag: 37926:1-37927:1
> Cache-Control: no-cache, no-store
> Pragma: no-cache
> Expires: 0
> X-Transaction: 0000-326srxia31tss3tc


As you pointed it later that issue is related to keep alive.
This is tunable , I 'll try to provide some advices

But read:
http://jmeter.apache.org/usermanual/properties_reference.html#httpclient4

And pay attention to:

httpclient4.idletimeout
httpclient4.validate_after_inactivity
httpclient4.time_to_live

It appears Default values do not suite your case.


> 1) Tried your test plan. Unfortunately, it does not make difference.

it was not made to fix issue only to make it simpler and indépendant of any
3rd party plugin

>
> 2) I ran Wireshark during a couple of tests. You can find a few screenshots
> in a later link. Note that my whole network kinda crashes when JMeter 3
> gets stuck (ping gets "General failure" and some active connections crash,
> note that this seems a bit random).

Thanks

>
> 3) I gave it a few minutes to recover between the runs.
>
> @Felix:
>
> I ran all tests with loggerpanel.usejsyntaxtext=false.
>
> No, the amount of seems roughly the same.
>
>
>
> Then, about pinpointing the issue:
>
> I managed to reproduce this by installing nginx on a network computer (sudo
> apt-get install nginx on debian 8). I did not configure anything and I
> simply bombarded the default nginx welcome page.
>
> I have attached Heap Dump, Thread Dumps and some other information in these
> links:
>
> JAVA impl (that doesn't crash):
> https://drive.google.com/drive/folders/0B1uBdVuLED5NSFJvc2VSdE13a0k?
> usp=sharing
>
> HttpCilent 4 impl (that crashes):
> https://drive.google.com/drive/folders/0B1uBdVuLED5NUGc1WWFYYkRncTQ?
> usp=sharing
>
> I used Philippe's test plan and even simplified it a bit, the JMX file is
> in the latter link. Note that the errors occur (in my case) during the
> ramp-up so I also shortened to run just 30s after reaching 500 users.
>
> With JAVA impl, I ran heap dump at ~450 users or so, and thread dumps after
> that.
>
> With HttpClient 4 impl, I ran heap dump immediately after seeing the first
> error (which was at ~430 users or so) and thread dumps right after that
> (the first two dumps I think took a bit longer than normally to create).
> JMeter goes unresponsive a second or two after seeing the first error, so I
> just barely had time to click to take the dump.
>
> As said, there are some screenshots from Wireshark trace for HttpClient 4
> run. I am not network expert, so I don't dare to judge them, but I included
> what I saw interesting. I think I can share some parts (privately) should
> you want to study the trace in detail.
>
> Finally, when I disabled keep-alive from nginx configuration, the problem
> was gone (as earlier). So, the problem is somehow related to keep-alive.
>
> Note that in all scenarios CPU goes to 100% quite immediately (also on 2.13
> now, probably because of high throughput, I assume my workstation is the
> bottleneck here) but without keep-alive or with JAVA impl, things work and
> with keep-alive enabled and HttpClient 4, things break apart. As before, on
> JMeter 2.13 everything works.
>
> Tuukka
>
>
>
> On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <
> felix.schumacher@internetallee.de <javascript:;>> wrote:
>
> >
> >
> > Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
> > tuukka.mustonen@gmail.com <javascript:;>>:
> > >@Philippe:
> > >
> > >0. Alright, I will.
> > >1. Will do.
> > >2. Sure.
> > >3. To let the network connection state restore? Ok, will do.
> > >
> > >I am having some other stuff I need to attend today, but I will get
> > >back to
> > >this in the afternoon at latest.
> >
> > Could you also try to set loggerpanel.usejsyntaxtext to false in your
> > settings?
> >
> > Are there more log entries, when you use httpclient 4 instead of Java?
> >
> > Vladimir has found signs for a event congestion in the awt event queue,
> > that show in your threaddumps.
> >
> > Regards,
> >  Felix
> >
> > >
> > >Tuukka
> > >
> > >
> > >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
> > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> > >
> > >> Hello,
> > >> I've created this bugzilla for investigation and to share with you
> > >the plan
> > >> I have used to run my own tests:
> > >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
> > >>
> > >> I use a Tomcat 8.5.11 with a localhost and a certificate generated
> > >and auto
> > >> signed.
> > >>
> > >> So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan 2.5
> > >Ghz
> > >> Intel Core i7
> > >> Java 8 u121
> > >> -Xmx512m -Xms512m
> > >> Test in GUI mode (ANTI-PATTERN)
> > >>
> > >> 2.13:
> > >> - CPU is at 100%
> > >> - No error
> > >> - No leak
> > >>
> > >> 3.2 nightly with default settings :
> > >> - CPU is at 100%
> > >> - No error
> > >> - No leak
> > >>
> > >> 3.2 nightly with this in user.properties :
> > >> httpclient4.time_to_live=0
> > >> httpclient4.validate_after_inactivity=0
> > >>
> > >> - CPU is at 100%
> > >> - No error
> > >> - No leak
> > >>
> > >> I attached the Heap Graphs to bugzilla and the plan.
> > >>
> > >> So for now I don't  reproduce your issue but I am not in the same
> > >> configuration as I am not hitting the same website.
> > >>
> > >> So
> > >> 0/ Could you provide by running only 5 requests, the response header
> > >that
> > >> your website returns ?
> > >> 1/ Could you try my test plan by replacing with your target host and
> > >see if
> > >> behaviour is the same ?
> > >> 2/ Ensure nothing is running (No Spotify :-) ) except your JMeter and
> > >> monitor network as per Felix note
> > >> 3/ Run 1 test after the other but ensure you wait a bit between each
> > >one
> > >>
> > >>
> > >> Thanks
> > >>
> > >>
> > >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
> > >> felix.schumacher@internetallee.de <javascript:;>> wrote:
> > >>
> > >> >
> > >> >
> > >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
> > >> > tuukka.mustonen@gmail.com <javascript:;>>:
> > >> > >Ok I re-ran the test.
> > >> > >
> > >> > >Took Heap Dump just before JMeter froze.
> > >> > >
> > >> > >Took Thread Dumps right after JMeter froze.
> > >> > >
> > >> >
> > >>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
> > >> >
> > >> > That folder seems to be empty now.
> > >> >
> > >> > Are you running the tests with a security manager enabled?
> > >> >
> > >> > Can you share the test or at least the used requests?
> > >> >
> > >> > Are you using a proxy and if so, how is it configured?
> > >> >
> > >> > Is jmeter running locally, or over something like rdp?
> > >> >
> > >> > Can you look at the used network connections like netstat -etapn
> > >would do
> > >> > on Linux?
> > >> >
> > >> > Regards,
> > >> >  Felix
> > >> >
> > >> > >
> > >> > >...
> > >> > >
> > >> > >For the fun of it, I also tried upgrading to the latest HttpClient
> > >> > >4.5.3
> > >> > >version. No effect.
> > >> > >
> > >> > >I also noticed that if I ping (google) during the perf testing,
> > >Windows
> > >> > >ping reports "General failure." when JMeter breaks. After killing
> > >it,
> > >> > >connectivity restores and there are replies from the server again.
> > >It
> > >> > >seems
> > >> > >like JMeter/HttpClient kills my whole network connection...
> > >> > >
> > >> > >Tuukka
> > >> > >
> > >> > >
> > >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > >> > >philippe.mouawad@gmail.com <javascript:;>> wrote:
> > >> > >
> > >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> > >> > ><tuukka.mustonen@gmail.com <javascript:;>
> > >> > >> >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I think I
> > >saw
> > >> > >> slightly
> > >> > >> > different behavior in my Linux guest. I'll write more about
> > >that
> > >> > >later,
> > >> > >> but
> > >> > >> > for simplicity I'm testing only on Windows host now:
> > >> > >> >
> > >> > >> > Thread dumps are downloadable at:
> > >> > >> >
> > >https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> > >> > >> >
> > >> > >> > (Took them with 7-8s interval I think, hope it's ok).
> > >> > >> >
> > >> > >> > I took the thread dumps once JMeter started filling in errors,
> > >> > >which is
> > >> > >> > somewhat after my CPU usage hit 100% (and music playback
> > >started to
> > >> > >> distort
> > >> > >> > etc.).
> > >> > >> >
> > >> > >> > I also created the head dump, note that I took it after
> > >running the
> > >> > >test
> > >> > >> > and after restarting JMeter, as JMeter froze during the test
> > >and I
> > >> > >had to
> > >> > >> > kill it. DId you mean I should take it *during* the test?
> > >> > >>
> > >> > >>
> > >> > >> YES.
> > >> > >>
> > >> > >>
> > >> > >> > Anyway, before
> > >> > >> > sharing it I want to make sure it can't contain anything
> > >> > >confidential?
> > >> > >> > (parts of my test plan, passwords, environment variables,
> > >> > >anything...)
> > >> > >> >
> > >> > >> > Tuukka
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > >> > >> > support@ubikloadpack.com <javascript:;>> wrote:
> > >> > >> >
> > >> > >> > > Hello,
> > >> > >> > > To investigate the problem:
> > >> > >> > > 1/ to create  thread dumps, install a JDK of same version as
> > >the
> > >> > >one
> > >> > >> > > running your JMeter.
> > >> > >> > >
> > >> > >> > > Run until you get the problem, then call using same user as
> > >the
> > >> > >one
> > >> > >> > running
> > >> > >> > > JMeter 3 times at 5s interval:
> > >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> > >> > >> > >
> > >> > >> > > It will also help if you provide the structure of your Test
> > >Plan.
> > >> > >> > >
> > >> > >> > > 2/ It would also help if you can provide a Heap Dump. With
> > >JMeter
> > >> > >GUI,
> > >> > >> > use
> > >> > >> > > menu Help > Create a Heap Dump
> > >> > >> > > This will create a Heap Dump (rather big file) , it would
> > >help a
> > >> > >lot if
> > >> > >> > you
> > >> > >> > > can make it available.
> > >> > >> > >
> > >> > >> > > Regards
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > >
> > >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > >> > >> > tuukka.mustonen@gmail.com <javascript:;>
> > >> > >> > > >
> > >> > >> > > wrote:
> > >> > >> > >
> > >> > >> > > > @Philippe:
> > >> > >> > > >
> > >> > >> > > > Yeah, I am normally running in non-GUI mode, but to
> > >> > >> > > > study/inspect/understand this problem I am running in GUI
> > >mode.
> > >> > >> JMeter
> > >> > >> > > 2.13
> > >> > >> > > > works just fine in GUI mode for these tests so JMeter 3.x
> > >> > >should too.
> > >> > >> > > >
> > >> > >> > > > I'll give the nightly build a try.
> > >> > >> > > >
> > >> > >> > > > Tuukka
> > >> > >> > > >
> > >> > >> > > >
> > >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > >> > >> > > > philippe.mouawad@gmail.com <javascript:;>
> > >> > >> > > > > wrote:
> > >> > >> > > >
> > >> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > >> > >> > > > tuukka.mustonen@gmail.com <javascript:;>
> > >> > >> > > > > >
> > >> > >> > > > > wrote:
> > >> > >> > > > >
> > >> > >> > > > > > @Philippe:
> > >> > >> > > > > >
> > >> > >> > > > > > 1. Yeah, I compared them with the same plan, same
> > >machine.
> > >> > >Should
> > >> > >> > be
> > >> > >> > > > > > everything equal.
> > >> > >> > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
> > >> > >-Xmx512m)
> > >> > >> and
> > >> > >> > as
> > >> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't
> > >tweak
> > >> > >them much
> > >> > >> > > > > because I
> > >> > >> > > > > > don't believe the issue lies there.
> > >> > >> > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > >> > >> > > > > > 4. I am not familiar with that but I can try. If you
> > >have
> > >> > >> > > instructions
> > >> > >> > > > to
> > >> > >> > > > > > point to, please do.
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > > 5. I am running with GUI.
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > This is a known anti-pattern .
> > >> > >> > > > > switch to non gui mode if you want correct results.
> > >> > >> > > > > http://jmeter.apache.org/usermanual/best-practices.html
> > >> > >> > > > >
> > >> > >> > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> > >> > >> tuning_tips/
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > > About file sending:
> > >> > >> > > > > >
> > >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
> > >> > >> > > > >
> > >> > >> > > > >  Could you give nightly build a try ?
> > >> > >> > > > > http://jmeter.apache.org/nightly.html
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > > Tuukka
> > >> > >> > > > > >
> > >> > >> > > > > >
> > >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > >> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> <javascript:;>
> > >> > >> > > > > > > wrote:
> > >> > >> > > > > >
> > >> > >> > > > > > > Also are you sending files in post body ?
> > >> > >> > > > > > >
> > >> > >> > > > > > > Thank you
> > >> > >> > > > > > >
> > >> > >> > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > >> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>
> <javascript:;>>
> > >> > >> > > > > > > wrote:
> > >> > >> > > > > > >
> > >> > >> > > > > > > > Hello,
> > >> > >> > > > > > > > Few questions, please try to answer them all:
> > >> > >> > > > > > > >
> > >> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same
> > >plan/sape
> > >> > >> machine...
> > >> > >> > > > > > everything
> > >> > >> > > > > > > > equal ?
> > >> > >> > > > > > > > 2/ What are your Xmx settings
> > >> > >> > > > > > > > 3/ does your plan download embedded resources ?
> > >> > >> > > > > > > > 4/ when issue happens can you make 3 thread dumps
> > >at 5
> > >> > >> seconds
> > >> > >> > > > > interval
> > >> > >> > > > > > > > and send file ? (use jstack)
> > >> > >> > > > > > > > 5/ are you running gui or non gui ?
> > >> > >> > > > > > > >
> > >> > >> > > > > > > > Thanks
> > >> > >> > > > > > > > Regards
> > >> > >> > > > > > > >
> > >> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > >> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> <javascript:;>
> > >> > >> > > > > > > >
> > ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com <javascript:;>
> > >> > >> > > > > <javascript:;>');>>
> > >> > >> > > > > > wrote:
> > >> > >> > > > > > > >
> > >> > >> > > > > > > >> Hi,
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience
> > >high
> > >> > >CPU
> > >> > >> > usage
> > >> > >> > > > > that
> > >> > >> > > > > > > >> renders JMeter unusable. CPU usage ramps quickly
> > >up
> > >> > >and
> > >> > >> after
> > >> > >> > > > having
> > >> > >> > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds),
> > >it
> > >> > >hits
> > >> > >> 100%
> > >> > >> > > on
> > >> > >> > > > my
> > >> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and
> > >I
> > >> > >have to
> > >> > >> > kill
> > >> > >> > > > it.
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> On 2.13 everything goes smoothly (with default
> > >> > >settings).
> > >> > >> CPU
> > >> > >> > > > usage
> > >> > >> > > > > > > stays
> > >> > >> > > > > > > >> at ~30% even with 500 users.
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> The experience is similar on both Windows 10 and
> > >Linux
> > >> > >> (Ubuntu
> > >> > >> > > > 14.04
> > >> > >> > > > > > LTS
> > >> > >> > > > > > > >> guest in Virtualbox).
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> The problems I saw in JMeter 3:
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> - java.net.SocketException: Permission denied:
> > >connect
> > >> > >> > > > > > > >> - After a while it runs out of heap space
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> For "Permission denied: connect" I tried to set
> > >> > >> > > > > > > >> java.net.preferIPv4Stack=true
> > >> > >> > > > > > > >> (in a few places to make sure it would really be
> > >in
> > >> > >use) and
> > >> > >> > > tried
> > >> > >> > > > > to
> > >> > >> > > > > > > run
> > >> > >> > > > > > > >> JMeter as administrator (in Win). No effect.
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> I also tried increasing heap size, but that
> > >didn't
> > >> > >help
> > >> > >> > either.
> > >> > >> > > > And
> > >> > >> > > > > I
> > >> > >> > > > > > > >> think
> > >> > >> > > > > > > >> memory size is not the root cause here - rather
> > >the
> > >> > >result.
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> I am normally connected to VPN but I also tried
> > >> > >disabling
> > >> > >> it.
> > >> > >> > No
> > >> > >> > > > > > effect.
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> I tried disabling with and without IPv6 enabled
> > >in
> > >> > >Windows
> > >> > >> > > network
> > >> > >> > > > > > > >> connection settings. No effect.
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> I am running a few extra JMeter plugins. Tried
> > >cutting
> > >> > >my
> > >> > >> test
> > >> > >> > > > > > scenario
> > >> > >> > > > > > > to
> > >> > >> > > > > > > >> minimum. No effect.
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> I am running anti-virus software that I cannot
> > >> > >disable, due
> > >> > >> to
> > >> > >> > > > this
> > >> > >> > > > > > > being
> > >> > >> > > > > > > >> forced by IT department. However, it's not a
> > >problem
> > >> > >with
> > >> > >> 2.13
> > >> > >> > > so
> > >> > >> > > > I
> > >> > >> > > > > > > assume
> > >> > >> > > > > > > >> it shouldn't be a problem here either.
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> FINALLY, once I set HTTP request implementation
> > >from
> > >> > >Apache
> > >> > >> > > > > > HttpClient 4
> > >> > >> > > > > > > >> to
> > >> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes -
> > >CPU
> > >> > >stays
> > >> > >> where
> > >> > >> > > it
> > >> > >> > > > > did
> > >> > >> > > > > > > with
> > >> > >> > > > > > > >> JMeter 2.13.
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win,
> > >update
> > >> > >111 on
> > >> > >> > > Linux).
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> Why does (the recommended) HttpClient 4 cause
> > >problems
> > >> > >here?
> > >> > >> > Is
> > >> > >> > > > > this a
> > >> > >> > > > > > > >> know
> > >> > >> > > > > > > >> bug/issue? JMeter 3.x is running newer version
> > >> > >compared to
> > >> > >> > 2.13,
> > >> > >> > > > but
> > >> > >> > > > > > > what
> > >> > >> > > > > > > >> change in HttpClient 4 might cause the problems
> > >> > >described?
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> Any ideas?
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >> Regards,
> > >> > >> > > > > > > >> Tuukka
> > >> > >> > > > > > > >>
> > >> > >> > > > > > > >
> > >> > >> > > > > > > >
> > >> > >> > > > > > > > --
> > >> > >> > > > > > > > Cordialement.
> > >> > >> > > > > > > > Philippe Mouawad.
> > >> > >> > > > > > > >
> > >> > >> > > > > > > >
> > >> > >> > > > > > > >
> > >> > >> > > > > > > >
> > >> > >> > > > > > >
> > >> > >> > > > > > > --
> > >> > >> > > > > > > Cordialement.
> > >> > >> > > > > > > Philippe Mouawad.
> > >> > >> > > > > > >
> > >> > >> > > > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > --
> > >> > >> > > > > Cordialement.
> > >> > >> > > > > Philippe Mouawad.
> > >> > >> > > > >
> > >> > >> > > >
> > >> > >> > >
> > >> > >> >
> > >> > >>
> > >> > >>
> > >> > >>
> > >> > >> --
> > >> > >> Cordialement.
> > >> > >> Philippe Mouawad.
> > >> > >>
> > >> >
> > >> >
> > >---------------------------------------------------------------------
> > >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> <javascript:;>
> > >> > For additional commands, e-mail: user-help@jmeter.apache.org
> <javascript:;>
> > >> >
> > >> >
> > >>
> > >>
> > >> --
> > >> Cordialement.
> > >> Philippe Mouawad.
> > >>
> >
>


-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

Well, to maybe elaborate the Wireshark findings a bit:

- At 5s, there are no errors (they start to appear at 30-40seconds or so),
but there was a stream (that I took a look at by random) where JMeter seems
to close a connection that could still be re-used (the server sends
Connection: keep-alive header in the reponse). The TCP stream was used to
do ~33 requests or so... I don't understand why JMeter would suddenly close
it? In any case, this did not produce error in the logs, but it still seems
a bit weird?

- At 45s, there's a "retransmission" of a package or something, and again
client closes connection even though server send Connection: keep-alive?

- At 50s, a bit different retransmission case?

These are just few examples from the trace.

Tuukka


On Fri, Feb 3, 2017 at 3:40 PM, Tuukka Mustonen <tu...@gmail.com>
wrote:

> Sorry for the delay, got too busy yesterday.
>
> I have been running more tests now. I tried to a few variations to
> pinpoint the issue, but first:
>
> @Philippe:
>
> 0) Response headers are like:
>
> Response headers:
> HTTP/1.1 200 OK
> Date: Fri, 03 Feb 2017 06:47:00 GMT
> Content-Type: application/json
> Content-Length: 374
> Connection: keep-alive
> Server: nginx/1.6.2
> ETag: 37926:1-37927:1
> Cache-Control: no-cache, no-store
> Pragma: no-cache
> Expires: 0
> X-Transaction: 0000-326srxia31tss3tc
>
> 1) Tried your test plan. Unfortunately, it does not make difference.
>
> 2) I ran Wireshark during a couple of tests. You can find a few
> screenshots in a later link. Note that my whole network kinda crashes when
> JMeter 3 gets stuck (ping gets "General failure" and some active
> connections crash, note that this seems a bit random).
>
> 3) I gave it a few minutes to recover between the runs.
>
> @Felix:
>
> I ran all tests with loggerpanel.usejsyntaxtext=false.
>
> No, the amount of seems roughly the same.
>
>
>
> Then, about pinpointing the issue:
>
> I managed to reproduce this by installing nginx on a network computer
> (sudo apt-get install nginx on debian 8). I did not configure anything and
> I simply bombarded the default nginx welcome page.
>
> I have attached Heap Dump, Thread Dumps and some other information in
> these links:
>
> JAVA impl (that doesn't crash): https://drive.google.com/drive/folders/
> 0B1uBdVuLED5NSFJvc2VSdE13a0k?usp=sharing
>
> HttpCilent 4 impl (that crashes): https://drive.google.com/drive/folders/
> 0B1uBdVuLED5NUGc1WWFYYkRncTQ?usp=sharing
>
> I used Philippe's test plan and even simplified it a bit, the JMX file is
> in the latter link. Note that the errors occur (in my case) during the
> ramp-up so I also shortened to run just 30s after reaching 500 users.
>
> With JAVA impl, I ran heap dump at ~450 users or so, and thread dumps
> after that.
>
> With HttpClient 4 impl, I ran heap dump immediately after seeing the first
> error (which was at ~430 users or so) and thread dumps right after that
> (the first two dumps I think took a bit longer than normally to create).
> JMeter goes unresponsive a second or two after seeing the first error, so I
> just barely had time to click to take the dump.
>
> As said, there are some screenshots from Wireshark trace for HttpClient 4
> run. I am not network expert, so I don't dare to judge them, but I included
> what I saw interesting. I think I can share some parts (privately) should
> you want to study the trace in detail.
>
> Finally, when I disabled keep-alive from nginx configuration, the problem
> was gone (as earlier). So, the problem is somehow related to keep-alive.
>
> Note that in all scenarios CPU goes to 100% quite immediately (also on
> 2.13 now, probably because of high throughput, I assume my workstation is
> the bottleneck here) but without keep-alive or with JAVA impl, things work
> and with keep-alive enabled and HttpClient 4, things break apart. As
> before, on JMeter 2.13 everything works.
>
> Tuukka
>
>
>
> On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <felix.schumacher@
> internetallee.de> wrote:
>
>>
>>
>> Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
>> tuukka.mustonen@gmail.com>:
>> >@Philippe:
>> >
>> >0. Alright, I will.
>> >1. Will do.
>> >2. Sure.
>> >3. To let the network connection state restore? Ok, will do.
>> >
>> >I am having some other stuff I need to attend today, but I will get
>> >back to
>> >this in the afternoon at latest.
>>
>> Could you also try to set loggerpanel.usejsyntaxtext to false in your
>> settings?
>>
>> Are there more log entries, when you use httpclient 4 instead of Java?
>>
>> Vladimir has found signs for a event congestion in the awt event queue,
>> that show in your threaddumps.
>>
>> Regards,
>>  Felix
>>
>> >
>> >Tuukka
>> >
>> >
>> >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
>> >philippe.mouawad@gmail.com> wrote:
>> >
>> >> Hello,
>> >> I've created this bugzilla for investigation and to share with you
>> >the plan
>> >> I have used to run my own tests:
>> >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
>> >>
>> >> I use a Tomcat 8.5.11 with a localhost and a certificate generated
>> >and auto
>> >> signed.
>> >>
>> >> So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan 2.5
>> >Ghz
>> >> Intel Core i7
>> >> Java 8 u121
>> >> -Xmx512m -Xms512m
>> >> Test in GUI mode (ANTI-PATTERN)
>> >>
>> >> 2.13:
>> >> - CPU is at 100%
>> >> - No error
>> >> - No leak
>> >>
>> >> 3.2 nightly with default settings :
>> >> - CPU is at 100%
>> >> - No error
>> >> - No leak
>> >>
>> >> 3.2 nightly with this in user.properties :
>> >> httpclient4.time_to_live=0
>> >> httpclient4.validate_after_inactivity=0
>> >>
>> >> - CPU is at 100%
>> >> - No error
>> >> - No leak
>> >>
>> >> I attached the Heap Graphs to bugzilla and the plan.
>> >>
>> >> So for now I don't  reproduce your issue but I am not in the same
>> >> configuration as I am not hitting the same website.
>> >>
>> >> So
>> >> 0/ Could you provide by running only 5 requests, the response header
>> >that
>> >> your website returns ?
>> >> 1/ Could you try my test plan by replacing with your target host and
>> >see if
>> >> behaviour is the same ?
>> >> 2/ Ensure nothing is running (No Spotify :-) ) except your JMeter and
>> >> monitor network as per Felix note
>> >> 3/ Run 1 test after the other but ensure you wait a bit between each
>> >one
>> >>
>> >>
>> >> Thanks
>> >>
>> >>
>> >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
>> >> felix.schumacher@internetallee.de> wrote:
>> >>
>> >> >
>> >> >
>> >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
>> >> > tuukka.mustonen@gmail.com>:
>> >> > >Ok I re-ran the test.
>> >> > >
>> >> > >Took Heap Dump just before JMeter froze.
>> >> > >
>> >> > >Took Thread Dumps right after JMeter froze.
>> >> > >
>> >> >
>> >>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
>> >> >
>> >> > That folder seems to be empty now.
>> >> >
>> >> > Are you running the tests with a security manager enabled?
>> >> >
>> >> > Can you share the test or at least the used requests?
>> >> >
>> >> > Are you using a proxy and if so, how is it configured?
>> >> >
>> >> > Is jmeter running locally, or over something like rdp?
>> >> >
>> >> > Can you look at the used network connections like netstat -etapn
>> >would do
>> >> > on Linux?
>> >> >
>> >> > Regards,
>> >> >  Felix
>> >> >
>> >> > >
>> >> > >...
>> >> > >
>> >> > >For the fun of it, I also tried upgrading to the latest HttpClient
>> >> > >4.5.3
>> >> > >version. No effect.
>> >> > >
>> >> > >I also noticed that if I ping (google) during the perf testing,
>> >Windows
>> >> > >ping reports "General failure." when JMeter breaks. After killing
>> >it,
>> >> > >connectivity restores and there are replies from the server again.
>> >It
>> >> > >seems
>> >> > >like JMeter/HttpClient kills my whole network connection...
>> >> > >
>> >> > >Tuukka
>> >> > >
>> >> > >
>> >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
>> >> > >philippe.mouawad@gmail.com> wrote:
>> >> > >
>> >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
>> >> > ><tuukka.mustonen@gmail.com
>> >> > >> >
>> >> > >> wrote:
>> >> > >>
>> >> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I think I
>> >saw
>> >> > >> slightly
>> >> > >> > different behavior in my Linux guest. I'll write more about
>> >that
>> >> > >later,
>> >> > >> but
>> >> > >> > for simplicity I'm testing only on Windows host now:
>> >> > >> >
>> >> > >> > Thread dumps are downloadable at:
>> >> > >> >
>> >https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
>> >> > >> >
>> >> > >> > (Took them with 7-8s interval I think, hope it's ok).
>> >> > >> >
>> >> > >> > I took the thread dumps once JMeter started filling in errors,
>> >> > >which is
>> >> > >> > somewhat after my CPU usage hit 100% (and music playback
>> >started to
>> >> > >> distort
>> >> > >> > etc.).
>> >> > >> >
>> >> > >> > I also created the head dump, note that I took it after
>> >running the
>> >> > >test
>> >> > >> > and after restarting JMeter, as JMeter froze during the test
>> >and I
>> >> > >had to
>> >> > >> > kill it. DId you mean I should take it *during* the test?
>> >> > >>
>> >> > >>
>> >> > >> YES.
>> >> > >>
>> >> > >>
>> >> > >> > Anyway, before
>> >> > >> > sharing it I want to make sure it can't contain anything
>> >> > >confidential?
>> >> > >> > (parts of my test plan, passwords, environment variables,
>> >> > >anything...)
>> >> > >> >
>> >> > >> > Tuukka
>> >> > >> >
>> >> > >> >
>> >> > >> >
>> >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
>> >> > >> > support@ubikloadpack.com> wrote:
>> >> > >> >
>> >> > >> > > Hello,
>> >> > >> > > To investigate the problem:
>> >> > >> > > 1/ to create  thread dumps, install a JDK of same version as
>> >the
>> >> > >one
>> >> > >> > > running your JMeter.
>> >> > >> > >
>> >> > >> > > Run until you get the problem, then call using same user as
>> >the
>> >> > >one
>> >> > >> > running
>> >> > >> > > JMeter 3 times at 5s interval:
>> >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
>> >> > >> > >
>> >> > >> > > It will also help if you provide the structure of your Test
>> >Plan.
>> >> > >> > >
>> >> > >> > > 2/ It would also help if you can provide a Heap Dump. With
>> >JMeter
>> >> > >GUI,
>> >> > >> > use
>> >> > >> > > menu Help > Create a Heap Dump
>> >> > >> > > This will create a Heap Dump (rather big file) , it would
>> >help a
>> >> > >lot if
>> >> > >> > you
>> >> > >> > > can make it available.
>> >> > >> > >
>> >> > >> > > Regards
>> >> > >> > >
>> >> > >> > >
>> >> > >> > >
>> >> > >> > >
>> >> > >> > >
>> >> > >> > >
>> >> > >> > >
>> >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
>> >> > >> > tuukka.mustonen@gmail.com
>> >> > >> > > >
>> >> > >> > > wrote:
>> >> > >> > >
>> >> > >> > > > @Philippe:
>> >> > >> > > >
>> >> > >> > > > Yeah, I am normally running in non-GUI mode, but to
>> >> > >> > > > study/inspect/understand this problem I am running in GUI
>> >mode.
>> >> > >> JMeter
>> >> > >> > > 2.13
>> >> > >> > > > works just fine in GUI mode for these tests so JMeter 3.x
>> >> > >should too.
>> >> > >> > > >
>> >> > >> > > > I'll give the nightly build a try.
>> >> > >> > > >
>> >> > >> > > > Tuukka
>> >> > >> > > >
>> >> > >> > > >
>> >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
>> >> > >> > > > philippe.mouawad@gmail.com
>> >> > >> > > > > wrote:
>> >> > >> > > >
>> >> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> >> > >> > > > tuukka.mustonen@gmail.com
>> >> > >> > > > > >
>> >> > >> > > > > wrote:
>> >> > >> > > > >
>> >> > >> > > > > > @Philippe:
>> >> > >> > > > > >
>> >> > >> > > > > > 1. Yeah, I compared them with the same plan, same
>> >machine.
>> >> > >Should
>> >> > >> > be
>> >> > >> > > > > > everything equal.
>> >> > >> > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
>> >> > >-Xmx512m)
>> >> > >> and
>> >> > >> > as
>> >> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't
>> >tweak
>> >> > >them much
>> >> > >> > > > > because I
>> >> > >> > > > > > don't believe the issue lies there.
>> >> > >> > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
>> >> > >> > > > > > 4. I am not familiar with that but I can try. If you
>> >have
>> >> > >> > > instructions
>> >> > >> > > > to
>> >> > >> > > > > > point to, please do.
>> >> > >> > > > >
>> >> > >> > > > >
>> >> > >> > > > >
>> >> > >> > > > >
>> >> > >> > > > > > 5. I am running with GUI.
>> >> > >> > > > >
>> >> > >> > > > >
>> >> > >> > > > > This is a known anti-pattern .
>> >> > >> > > > > switch to non gui mode if you want correct results.
>> >> > >> > > > > http://jmeter.apache.org/usermanual/best-practices.html
>> >> > >> > > > >
>> >> > >> > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
>> >> > >> tuning_tips/
>> >> > >> > > > >
>> >> > >> > > > >
>> >> > >> > > > > > About file sending:
>> >> > >> > > > > >
>> >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
>> >> > >> > > > >
>> >> > >> > > > >  Could you give nightly build a try ?
>> >> > >> > > > > http://jmeter.apache.org/nightly.html
>> >> > >> > > > >
>> >> > >> > > > >
>> >> > >> > > > > > Tuukka
>> >> > >> > > > > >
>> >> > >> > > > > >
>> >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
>> >> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
>> >> > >> > > > > > > wrote:
>> >> > >> > > > > >
>> >> > >> > > > > > > Also are you sending files in post body ?
>> >> > >> > > > > > >
>> >> > >> > > > > > > Thank you
>> >> > >> > > > > > >
>> >> > >> > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
>> >> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>>
>> >> > >> > > > > > > wrote:
>> >> > >> > > > > > >
>> >> > >> > > > > > > > Hello,
>> >> > >> > > > > > > > Few questions, please try to answer them all:
>> >> > >> > > > > > > >
>> >> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same
>> >plan/sape
>> >> > >> machine...
>> >> > >> > > > > > everything
>> >> > >> > > > > > > > equal ?
>> >> > >> > > > > > > > 2/ What are your Xmx settings
>> >> > >> > > > > > > > 3/ does your plan download embedded resources ?
>> >> > >> > > > > > > > 4/ when issue happens can you make 3 thread dumps
>> >at 5
>> >> > >> seconds
>> >> > >> > > > > interval
>> >> > >> > > > > > > > and send file ? (use jstack)
>> >> > >> > > > > > > > 5/ are you running gui or non gui ?
>> >> > >> > > > > > > >
>> >> > >> > > > > > > > Thanks
>> >> > >> > > > > > > > Regards
>> >> > >> > > > > > > >
>> >> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> >> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
>> >> > >> > > > > > > >
>> ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
>> >> > >> > > > > <javascript:;>');>>
>> >> > >> > > > > > wrote:
>> >> > >> > > > > > > >
>> >> > >> > > > > > > >> Hi,
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience
>> >high
>> >> > >CPU
>> >> > >> > usage
>> >> > >> > > > > that
>> >> > >> > > > > > > >> renders JMeter unusable. CPU usage ramps quickly
>> >up
>> >> > >and
>> >> > >> after
>> >> > >> > > > having
>> >> > >> > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds),
>> >it
>> >> > >hits
>> >> > >> 100%
>> >> > >> > > on
>> >> > >> > > > my
>> >> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and
>> >I
>> >> > >have to
>> >> > >> > kill
>> >> > >> > > > it.
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> On 2.13 everything goes smoothly (with default
>> >> > >settings).
>> >> > >> CPU
>> >> > >> > > > usage
>> >> > >> > > > > > > stays
>> >> > >> > > > > > > >> at ~30% even with 500 users.
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> The experience is similar on both Windows 10 and
>> >Linux
>> >> > >> (Ubuntu
>> >> > >> > > > 14.04
>> >> > >> > > > > > LTS
>> >> > >> > > > > > > >> guest in Virtualbox).
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> The problems I saw in JMeter 3:
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> - java.net.SocketException: Permission denied:
>> >connect
>> >> > >> > > > > > > >> - After a while it runs out of heap space
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> For "Permission denied: connect" I tried to set
>> >> > >> > > > > > > >> java.net.preferIPv4Stack=true
>> >> > >> > > > > > > >> (in a few places to make sure it would really be
>> >in
>> >> > >use) and
>> >> > >> > > tried
>> >> > >> > > > > to
>> >> > >> > > > > > > run
>> >> > >> > > > > > > >> JMeter as administrator (in Win). No effect.
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> I also tried increasing heap size, but that
>> >didn't
>> >> > >help
>> >> > >> > either.
>> >> > >> > > > And
>> >> > >> > > > > I
>> >> > >> > > > > > > >> think
>> >> > >> > > > > > > >> memory size is not the root cause here - rather
>> >the
>> >> > >result.
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> I am normally connected to VPN but I also tried
>> >> > >disabling
>> >> > >> it.
>> >> > >> > No
>> >> > >> > > > > > effect.
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> I tried disabling with and without IPv6 enabled
>> >in
>> >> > >Windows
>> >> > >> > > network
>> >> > >> > > > > > > >> connection settings. No effect.
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> I am running a few extra JMeter plugins. Tried
>> >cutting
>> >> > >my
>> >> > >> test
>> >> > >> > > > > > scenario
>> >> > >> > > > > > > to
>> >> > >> > > > > > > >> minimum. No effect.
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> I am running anti-virus software that I cannot
>> >> > >disable, due
>> >> > >> to
>> >> > >> > > > this
>> >> > >> > > > > > > being
>> >> > >> > > > > > > >> forced by IT department. However, it's not a
>> >problem
>> >> > >with
>> >> > >> 2.13
>> >> > >> > > so
>> >> > >> > > > I
>> >> > >> > > > > > > assume
>> >> > >> > > > > > > >> it shouldn't be a problem here either.
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> FINALLY, once I set HTTP request implementation
>> >from
>> >> > >Apache
>> >> > >> > > > > > HttpClient 4
>> >> > >> > > > > > > >> to
>> >> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes -
>> >CPU
>> >> > >stays
>> >> > >> where
>> >> > >> > > it
>> >> > >> > > > > did
>> >> > >> > > > > > > with
>> >> > >> > > > > > > >> JMeter 2.13.
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win,
>> >update
>> >> > >111 on
>> >> > >> > > Linux).
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> Why does (the recommended) HttpClient 4 cause
>> >problems
>> >> > >here?
>> >> > >> > Is
>> >> > >> > > > > this a
>> >> > >> > > > > > > >> know
>> >> > >> > > > > > > >> bug/issue? JMeter 3.x is running newer version
>> >> > >compared to
>> >> > >> > 2.13,
>> >> > >> > > > but
>> >> > >> > > > > > > what
>> >> > >> > > > > > > >> change in HttpClient 4 might cause the problems
>> >> > >described?
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> Any ideas?
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >> Regards,
>> >> > >> > > > > > > >> Tuukka
>> >> > >> > > > > > > >>
>> >> > >> > > > > > > >
>> >> > >> > > > > > > >
>> >> > >> > > > > > > > --
>> >> > >> > > > > > > > Cordialement.
>> >> > >> > > > > > > > Philippe Mouawad.
>> >> > >> > > > > > > >
>> >> > >> > > > > > > >
>> >> > >> > > > > > > >
>> >> > >> > > > > > > >
>> >> > >> > > > > > >
>> >> > >> > > > > > > --
>> >> > >> > > > > > > Cordialement.
>> >> > >> > > > > > > Philippe Mouawad.
>> >> > >> > > > > > >
>> >> > >> > > > > >
>> >> > >> > > > >
>> >> > >> > > > >
>> >> > >> > > > > --
>> >> > >> > > > > Cordialement.
>> >> > >> > > > > Philippe Mouawad.
>> >> > >> > > > >
>> >> > >> > > >
>> >> > >> > >
>> >> > >> >
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >> --
>> >> > >> Cordialement.
>> >> > >> Philippe Mouawad.
>> >> > >>
>> >> >
>> >> >
>> >---------------------------------------------------------------------
>> >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> >> > For additional commands, e-mail: user-help@jmeter.apache.org
>> >> >
>> >> >
>> >>
>> >>
>> >> --
>> >> Cordialement.
>> >> Philippe Mouawad.
>> >>
>>
>
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

Sorry for the delay, got too busy yesterday.

I have been running more tests now. I tried to a few variations to pinpoint
the issue, but first:

@Philippe:

0) Response headers are like:

Response headers:
HTTP/1.1 200 OK
Date: Fri, 03 Feb 2017 06:47:00 GMT
Content-Type: application/json
Content-Length: 374
Connection: keep-alive
Server: nginx/1.6.2
ETag: 37926:1-37927:1
Cache-Control: no-cache, no-store
Pragma: no-cache
Expires: 0
X-Transaction: 0000-326srxia31tss3tc

1) Tried your test plan. Unfortunately, it does not make difference.

2) I ran Wireshark during a couple of tests. You can find a few screenshots
in a later link. Note that my whole network kinda crashes when JMeter 3
gets stuck (ping gets "General failure" and some active connections crash,
note that this seems a bit random).

3) I gave it a few minutes to recover between the runs.

@Felix:

I ran all tests with loggerpanel.usejsyntaxtext=false.

No, the amount of seems roughly the same.



Then, about pinpointing the issue:

I managed to reproduce this by installing nginx on a network computer (sudo
apt-get install nginx on debian 8). I did not configure anything and I
simply bombarded the default nginx welcome page.

I have attached Heap Dump, Thread Dumps and some other information in these
links:

JAVA impl (that doesn't crash):
https://drive.google.com/drive/folders/0B1uBdVuLED5NSFJvc2VSdE13a0k?usp=sharing

HttpCilent 4 impl (that crashes):
https://drive.google.com/drive/folders/0B1uBdVuLED5NUGc1WWFYYkRncTQ?usp=sharing

I used Philippe's test plan and even simplified it a bit, the JMX file is
in the latter link. Note that the errors occur (in my case) during the
ramp-up so I also shortened to run just 30s after reaching 500 users.

With JAVA impl, I ran heap dump at ~450 users or so, and thread dumps after
that.

With HttpClient 4 impl, I ran heap dump immediately after seeing the first
error (which was at ~430 users or so) and thread dumps right after that
(the first two dumps I think took a bit longer than normally to create).
JMeter goes unresponsive a second or two after seeing the first error, so I
just barely had time to click to take the dump.

As said, there are some screenshots from Wireshark trace for HttpClient 4
run. I am not network expert, so I don't dare to judge them, but I included
what I saw interesting. I think I can share some parts (privately) should
you want to study the trace in detail.

Finally, when I disabled keep-alive from nginx configuration, the problem
was gone (as earlier). So, the problem is somehow related to keep-alive.

Note that in all scenarios CPU goes to 100% quite immediately (also on 2.13
now, probably because of high throughput, I assume my workstation is the
bottleneck here) but without keep-alive or with JAVA impl, things work and
with keep-alive enabled and HttpClient 4, things break apart. As before, on
JMeter 2.13 everything works.

Tuukka



On Fri, Feb 3, 2017 at 7:57 AM, Felix Schumacher <
felix.schumacher@internetallee.de> wrote:

>
>
> Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <
> tuukka.mustonen@gmail.com>:
> >@Philippe:
> >
> >0. Alright, I will.
> >1. Will do.
> >2. Sure.
> >3. To let the network connection state restore? Ok, will do.
> >
> >I am having some other stuff I need to attend today, but I will get
> >back to
> >this in the afternoon at latest.
>
> Could you also try to set loggerpanel.usejsyntaxtext to false in your
> settings?
>
> Are there more log entries, when you use httpclient 4 instead of Java?
>
> Vladimir has found signs for a event congestion in the awt event queue,
> that show in your threaddumps.
>
> Regards,
>  Felix
>
> >
> >Tuukka
> >
> >
> >On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
> >philippe.mouawad@gmail.com> wrote:
> >
> >> Hello,
> >> I've created this bugzilla for investigation and to share with you
> >the plan
> >> I have used to run my own tests:
> >> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
> >>
> >> I use a Tomcat 8.5.11 with a localhost and a certificate generated
> >and auto
> >> signed.
> >>
> >> So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan 2.5
> >Ghz
> >> Intel Core i7
> >> Java 8 u121
> >> -Xmx512m -Xms512m
> >> Test in GUI mode (ANTI-PATTERN)
> >>
> >> 2.13:
> >> - CPU is at 100%
> >> - No error
> >> - No leak
> >>
> >> 3.2 nightly with default settings :
> >> - CPU is at 100%
> >> - No error
> >> - No leak
> >>
> >> 3.2 nightly with this in user.properties :
> >> httpclient4.time_to_live=0
> >> httpclient4.validate_after_inactivity=0
> >>
> >> - CPU is at 100%
> >> - No error
> >> - No leak
> >>
> >> I attached the Heap Graphs to bugzilla and the plan.
> >>
> >> So for now I don't  reproduce your issue but I am not in the same
> >> configuration as I am not hitting the same website.
> >>
> >> So
> >> 0/ Could you provide by running only 5 requests, the response header
> >that
> >> your website returns ?
> >> 1/ Could you try my test plan by replacing with your target host and
> >see if
> >> behaviour is the same ?
> >> 2/ Ensure nothing is running (No Spotify :-) ) except your JMeter and
> >> monitor network as per Felix note
> >> 3/ Run 1 test after the other but ensure you wait a bit between each
> >one
> >>
> >>
> >> Thanks
> >>
> >>
> >> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
> >> felix.schumacher@internetallee.de> wrote:
> >>
> >> >
> >> >
> >> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
> >> > tuukka.mustonen@gmail.com>:
> >> > >Ok I re-ran the test.
> >> > >
> >> > >Took Heap Dump just before JMeter froze.
> >> > >
> >> > >Took Thread Dumps right after JMeter froze.
> >> > >
> >> >
> >>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
> >> >
> >> > That folder seems to be empty now.
> >> >
> >> > Are you running the tests with a security manager enabled?
> >> >
> >> > Can you share the test or at least the used requests?
> >> >
> >> > Are you using a proxy and if so, how is it configured?
> >> >
> >> > Is jmeter running locally, or over something like rdp?
> >> >
> >> > Can you look at the used network connections like netstat -etapn
> >would do
> >> > on Linux?
> >> >
> >> > Regards,
> >> >  Felix
> >> >
> >> > >
> >> > >...
> >> > >
> >> > >For the fun of it, I also tried upgrading to the latest HttpClient
> >> > >4.5.3
> >> > >version. No effect.
> >> > >
> >> > >I also noticed that if I ping (google) during the perf testing,
> >Windows
> >> > >ping reports "General failure." when JMeter breaks. After killing
> >it,
> >> > >connectivity restores and there are replies from the server again.
> >It
> >> > >seems
> >> > >like JMeter/HttpClient kills my whole network connection...
> >> > >
> >> > >Tuukka
> >> > >
> >> > >
> >> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> >> > >philippe.mouawad@gmail.com> wrote:
> >> > >
> >> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> >> > ><tuukka.mustonen@gmail.com
> >> > >> >
> >> > >> wrote:
> >> > >>
> >> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I think I
> >saw
> >> > >> slightly
> >> > >> > different behavior in my Linux guest. I'll write more about
> >that
> >> > >later,
> >> > >> but
> >> > >> > for simplicity I'm testing only on Windows host now:
> >> > >> >
> >> > >> > Thread dumps are downloadable at:
> >> > >> >
> >https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> >> > >> >
> >> > >> > (Took them with 7-8s interval I think, hope it's ok).
> >> > >> >
> >> > >> > I took the thread dumps once JMeter started filling in errors,
> >> > >which is
> >> > >> > somewhat after my CPU usage hit 100% (and music playback
> >started to
> >> > >> distort
> >> > >> > etc.).
> >> > >> >
> >> > >> > I also created the head dump, note that I took it after
> >running the
> >> > >test
> >> > >> > and after restarting JMeter, as JMeter froze during the test
> >and I
> >> > >had to
> >> > >> > kill it. DId you mean I should take it *during* the test?
> >> > >>
> >> > >>
> >> > >> YES.
> >> > >>
> >> > >>
> >> > >> > Anyway, before
> >> > >> > sharing it I want to make sure it can't contain anything
> >> > >confidential?
> >> > >> > (parts of my test plan, passwords, environment variables,
> >> > >anything...)
> >> > >> >
> >> > >> > Tuukka
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> >> > >> > support@ubikloadpack.com> wrote:
> >> > >> >
> >> > >> > > Hello,
> >> > >> > > To investigate the problem:
> >> > >> > > 1/ to create  thread dumps, install a JDK of same version as
> >the
> >> > >one
> >> > >> > > running your JMeter.
> >> > >> > >
> >> > >> > > Run until you get the problem, then call using same user as
> >the
> >> > >one
> >> > >> > running
> >> > >> > > JMeter 3 times at 5s interval:
> >> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> >> > >> > >
> >> > >> > > It will also help if you provide the structure of your Test
> >Plan.
> >> > >> > >
> >> > >> > > 2/ It would also help if you can provide a Heap Dump. With
> >JMeter
> >> > >GUI,
> >> > >> > use
> >> > >> > > menu Help > Create a Heap Dump
> >> > >> > > This will create a Heap Dump (rather big file) , it would
> >help a
> >> > >lot if
> >> > >> > you
> >> > >> > > can make it available.
> >> > >> > >
> >> > >> > > Regards
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> >> > >> > tuukka.mustonen@gmail.com
> >> > >> > > >
> >> > >> > > wrote:
> >> > >> > >
> >> > >> > > > @Philippe:
> >> > >> > > >
> >> > >> > > > Yeah, I am normally running in non-GUI mode, but to
> >> > >> > > > study/inspect/understand this problem I am running in GUI
> >mode.
> >> > >> JMeter
> >> > >> > > 2.13
> >> > >> > > > works just fine in GUI mode for these tests so JMeter 3.x
> >> > >should too.
> >> > >> > > >
> >> > >> > > > I'll give the nightly build a try.
> >> > >> > > >
> >> > >> > > > Tuukka
> >> > >> > > >
> >> > >> > > >
> >> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> >> > >> > > > philippe.mouawad@gmail.com
> >> > >> > > > > wrote:
> >> > >> > > >
> >> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> >> > >> > > > tuukka.mustonen@gmail.com
> >> > >> > > > > >
> >> > >> > > > > wrote:
> >> > >> > > > >
> >> > >> > > > > > @Philippe:
> >> > >> > > > > >
> >> > >> > > > > > 1. Yeah, I compared them with the same plan, same
> >machine.
> >> > >Should
> >> > >> > be
> >> > >> > > > > > everything equal.
> >> > >> > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
> >> > >-Xmx512m)
> >> > >> and
> >> > >> > as
> >> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't
> >tweak
> >> > >them much
> >> > >> > > > > because I
> >> > >> > > > > > don't believe the issue lies there.
> >> > >> > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> >> > >> > > > > > 4. I am not familiar with that but I can try. If you
> >have
> >> > >> > > instructions
> >> > >> > > > to
> >> > >> > > > > > point to, please do.
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > > 5. I am running with GUI.
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > This is a known anti-pattern .
> >> > >> > > > > switch to non gui mode if you want correct results.
> >> > >> > > > > http://jmeter.apache.org/usermanual/best-practices.html
> >> > >> > > > >
> >> > >> > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> >> > >> tuning_tips/
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > > About file sending:
> >> > >> > > > > >
> >> > >> > > > > > 6. Nope. I'm just doing simple GETs.
> >> > >> > > > >
> >> > >> > > > >  Could you give nightly build a try ?
> >> > >> > > > > http://jmeter.apache.org/nightly.html
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > > Tuukka
> >> > >> > > > > >
> >> > >> > > > > >
> >> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> >> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> >> > >> > > > > > > wrote:
> >> > >> > > > > >
> >> > >> > > > > > > Also are you sending files in post body ?
> >> > >> > > > > > >
> >> > >> > > > > > > Thank you
> >> > >> > > > > > >
> >> > >> > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> >> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> >> > >> > > > > > > wrote:
> >> > >> > > > > > >
> >> > >> > > > > > > > Hello,
> >> > >> > > > > > > > Few questions, please try to answer them all:
> >> > >> > > > > > > >
> >> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same
> >plan/sape
> >> > >> machine...
> >> > >> > > > > > everything
> >> > >> > > > > > > > equal ?
> >> > >> > > > > > > > 2/ What are your Xmx settings
> >> > >> > > > > > > > 3/ does your plan download embedded resources ?
> >> > >> > > > > > > > 4/ when issue happens can you make 3 thread dumps
> >at 5
> >> > >> seconds
> >> > >> > > > > interval
> >> > >> > > > > > > > and send file ? (use jstack)
> >> > >> > > > > > > > 5/ are you running gui or non gui ?
> >> > >> > > > > > > >
> >> > >> > > > > > > > Thanks
> >> > >> > > > > > > > Regards
> >> > >> > > > > > > >
> >> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> >> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> >> > >> > > > > > > >
> ><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> >> > >> > > > > <javascript:;>');>>
> >> > >> > > > > > wrote:
> >> > >> > > > > > > >
> >> > >> > > > > > > >> Hi,
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience
> >high
> >> > >CPU
> >> > >> > usage
> >> > >> > > > > that
> >> > >> > > > > > > >> renders JMeter unusable. CPU usage ramps quickly
> >up
> >> > >and
> >> > >> after
> >> > >> > > > having
> >> > >> > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds),
> >it
> >> > >hits
> >> > >> 100%
> >> > >> > > on
> >> > >> > > > my
> >> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and
> >I
> >> > >have to
> >> > >> > kill
> >> > >> > > > it.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> On 2.13 everything goes smoothly (with default
> >> > >settings).
> >> > >> CPU
> >> > >> > > > usage
> >> > >> > > > > > > stays
> >> > >> > > > > > > >> at ~30% even with 500 users.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> The experience is similar on both Windows 10 and
> >Linux
> >> > >> (Ubuntu
> >> > >> > > > 14.04
> >> > >> > > > > > LTS
> >> > >> > > > > > > >> guest in Virtualbox).
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> The problems I saw in JMeter 3:
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> - java.net.SocketException: Permission denied:
> >connect
> >> > >> > > > > > > >> - After a while it runs out of heap space
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> For "Permission denied: connect" I tried to set
> >> > >> > > > > > > >> java.net.preferIPv4Stack=true
> >> > >> > > > > > > >> (in a few places to make sure it would really be
> >in
> >> > >use) and
> >> > >> > > tried
> >> > >> > > > > to
> >> > >> > > > > > > run
> >> > >> > > > > > > >> JMeter as administrator (in Win). No effect.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I also tried increasing heap size, but that
> >didn't
> >> > >help
> >> > >> > either.
> >> > >> > > > And
> >> > >> > > > > I
> >> > >> > > > > > > >> think
> >> > >> > > > > > > >> memory size is not the root cause here - rather
> >the
> >> > >result.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I am normally connected to VPN but I also tried
> >> > >disabling
> >> > >> it.
> >> > >> > No
> >> > >> > > > > > effect.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I tried disabling with and without IPv6 enabled
> >in
> >> > >Windows
> >> > >> > > network
> >> > >> > > > > > > >> connection settings. No effect.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I am running a few extra JMeter plugins. Tried
> >cutting
> >> > >my
> >> > >> test
> >> > >> > > > > > scenario
> >> > >> > > > > > > to
> >> > >> > > > > > > >> minimum. No effect.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I am running anti-virus software that I cannot
> >> > >disable, due
> >> > >> to
> >> > >> > > > this
> >> > >> > > > > > > being
> >> > >> > > > > > > >> forced by IT department. However, it's not a
> >problem
> >> > >with
> >> > >> 2.13
> >> > >> > > so
> >> > >> > > > I
> >> > >> > > > > > > assume
> >> > >> > > > > > > >> it shouldn't be a problem here either.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> FINALLY, once I set HTTP request implementation
> >from
> >> > >Apache
> >> > >> > > > > > HttpClient 4
> >> > >> > > > > > > >> to
> >> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes -
> >CPU
> >> > >stays
> >> > >> where
> >> > >> > > it
> >> > >> > > > > did
> >> > >> > > > > > > with
> >> > >> > > > > > > >> JMeter 2.13.
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win,
> >update
> >> > >111 on
> >> > >> > > Linux).
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> Why does (the recommended) HttpClient 4 cause
> >problems
> >> > >here?
> >> > >> > Is
> >> > >> > > > > this a
> >> > >> > > > > > > >> know
> >> > >> > > > > > > >> bug/issue? JMeter 3.x is running newer version
> >> > >compared to
> >> > >> > 2.13,
> >> > >> > > > but
> >> > >> > > > > > > what
> >> > >> > > > > > > >> change in HttpClient 4 might cause the problems
> >> > >described?
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> Any ideas?
> >> > >> > > > > > > >>
> >> > >> > > > > > > >> Regards,
> >> > >> > > > > > > >> Tuukka
> >> > >> > > > > > > >>
> >> > >> > > > > > > >
> >> > >> > > > > > > >
> >> > >> > > > > > > > --
> >> > >> > > > > > > > Cordialement.
> >> > >> > > > > > > > Philippe Mouawad.
> >> > >> > > > > > > >
> >> > >> > > > > > > >
> >> > >> > > > > > > >
> >> > >> > > > > > > >
> >> > >> > > > > > >
> >> > >> > > > > > > --
> >> > >> > > > > > > Cordialement.
> >> > >> > > > > > > Philippe Mouawad.
> >> > >> > > > > > >
> >> > >> > > > > >
> >> > >> > > > >
> >> > >> > > > >
> >> > >> > > > > --
> >> > >> > > > > Cordialement.
> >> > >> > > > > Philippe Mouawad.
> >> > >> > > > >
> >> > >> > > >
> >> > >> > >
> >> > >> >
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Cordialement.
> >> > >> Philippe Mouawad.
> >> > >>
> >> >
> >> >
> >---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >> > For additional commands, e-mail: user-help@jmeter.apache.org
> >> >
> >> >
> >>
> >>
> >> --
> >> Cordialement.
> >> Philippe Mouawad.
> >>
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Felix Schumacher <fe...@internetallee.de>.


Am 2. Februar 2017 06:52:32 MEZ schrieb Tuukka Mustonen <tu...@gmail.com>:
>@Philippe:
>
>0. Alright, I will.
>1. Will do.
>2. Sure.
>3. To let the network connection state restore? Ok, will do.
>
>I am having some other stuff I need to attend today, but I will get
>back to
>this in the afternoon at latest.

Could you also try to set loggerpanel.usejsyntaxtext to false in your settings? 

Are there more log entries, when you use httpclient 4 instead of Java?

Vladimir has found signs for a event congestion in the awt event queue, that show in your threaddumps.

Regards,
 Felix

>
>Tuukka
>
>
>On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
>philippe.mouawad@gmail.com> wrote:
>
>> Hello,
>> I've created this bugzilla for investigation and to share with you
>the plan
>> I have used to run my own tests:
>> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
>>
>> I use a Tomcat 8.5.11 with a localhost and a certificate generated
>and auto
>> signed.
>>
>> So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan 2.5
>Ghz
>> Intel Core i7
>> Java 8 u121
>> -Xmx512m -Xms512m
>> Test in GUI mode (ANTI-PATTERN)
>>
>> 2.13:
>> - CPU is at 100%
>> - No error
>> - No leak
>>
>> 3.2 nightly with default settings :
>> - CPU is at 100%
>> - No error
>> - No leak
>>
>> 3.2 nightly with this in user.properties :
>> httpclient4.time_to_live=0
>> httpclient4.validate_after_inactivity=0
>>
>> - CPU is at 100%
>> - No error
>> - No leak
>>
>> I attached the Heap Graphs to bugzilla and the plan.
>>
>> So for now I don't  reproduce your issue but I am not in the same
>> configuration as I am not hitting the same website.
>>
>> So
>> 0/ Could you provide by running only 5 requests, the response header
>that
>> your website returns ?
>> 1/ Could you try my test plan by replacing with your target host and
>see if
>> behaviour is the same ?
>> 2/ Ensure nothing is running (No Spotify :-) ) except your JMeter and
>> monitor network as per Felix note
>> 3/ Run 1 test after the other but ensure you wait a bit between each
>one
>>
>>
>> Thanks
>>
>>
>> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
>> felix.schumacher@internetallee.de> wrote:
>>
>> >
>> >
>> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
>> > tuukka.mustonen@gmail.com>:
>> > >Ok I re-ran the test.
>> > >
>> > >Took Heap Dump just before JMeter froze.
>> > >
>> > >Took Thread Dumps right after JMeter froze.
>> > >
>> >
>>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
>> >
>> > That folder seems to be empty now.
>> >
>> > Are you running the tests with a security manager enabled?
>> >
>> > Can you share the test or at least the used requests?
>> >
>> > Are you using a proxy and if so, how is it configured?
>> >
>> > Is jmeter running locally, or over something like rdp?
>> >
>> > Can you look at the used network connections like netstat -etapn
>would do
>> > on Linux?
>> >
>> > Regards,
>> >  Felix
>> >
>> > >
>> > >...
>> > >
>> > >For the fun of it, I also tried upgrading to the latest HttpClient
>> > >4.5.3
>> > >version. No effect.
>> > >
>> > >I also noticed that if I ping (google) during the perf testing,
>Windows
>> > >ping reports "General failure." when JMeter breaks. After killing
>it,
>> > >connectivity restores and there are replies from the server again.
>It
>> > >seems
>> > >like JMeter/HttpClient kills my whole network connection...
>> > >
>> > >Tuukka
>> > >
>> > >
>> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
>> > >philippe.mouawad@gmail.com> wrote:
>> > >
>> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
>> > ><tuukka.mustonen@gmail.com
>> > >> >
>> > >> wrote:
>> > >>
>> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I think I
>saw
>> > >> slightly
>> > >> > different behavior in my Linux guest. I'll write more about
>that
>> > >later,
>> > >> but
>> > >> > for simplicity I'm testing only on Windows host now:
>> > >> >
>> > >> > Thread dumps are downloadable at:
>> > >> >
>https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
>> > >> >
>> > >> > (Took them with 7-8s interval I think, hope it's ok).
>> > >> >
>> > >> > I took the thread dumps once JMeter started filling in errors,
>> > >which is
>> > >> > somewhat after my CPU usage hit 100% (and music playback
>started to
>> > >> distort
>> > >> > etc.).
>> > >> >
>> > >> > I also created the head dump, note that I took it after
>running the
>> > >test
>> > >> > and after restarting JMeter, as JMeter froze during the test
>and I
>> > >had to
>> > >> > kill it. DId you mean I should take it *during* the test?
>> > >>
>> > >>
>> > >> YES.
>> > >>
>> > >>
>> > >> > Anyway, before
>> > >> > sharing it I want to make sure it can't contain anything
>> > >confidential?
>> > >> > (parts of my test plan, passwords, environment variables,
>> > >anything...)
>> > >> >
>> > >> > Tuukka
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
>> > >> > support@ubikloadpack.com> wrote:
>> > >> >
>> > >> > > Hello,
>> > >> > > To investigate the problem:
>> > >> > > 1/ to create  thread dumps, install a JDK of same version as
>the
>> > >one
>> > >> > > running your JMeter.
>> > >> > >
>> > >> > > Run until you get the problem, then call using same user as
>the
>> > >one
>> > >> > running
>> > >> > > JMeter 3 times at 5s interval:
>> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
>> > >> > >
>> > >> > > It will also help if you provide the structure of your Test
>Plan.
>> > >> > >
>> > >> > > 2/ It would also help if you can provide a Heap Dump. With
>JMeter
>> > >GUI,
>> > >> > use
>> > >> > > menu Help > Create a Heap Dump
>> > >> > > This will create a Heap Dump (rather big file) , it would
>help a
>> > >lot if
>> > >> > you
>> > >> > > can make it available.
>> > >> > >
>> > >> > > Regards
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
>> > >> > tuukka.mustonen@gmail.com
>> > >> > > >
>> > >> > > wrote:
>> > >> > >
>> > >> > > > @Philippe:
>> > >> > > >
>> > >> > > > Yeah, I am normally running in non-GUI mode, but to
>> > >> > > > study/inspect/understand this problem I am running in GUI
>mode.
>> > >> JMeter
>> > >> > > 2.13
>> > >> > > > works just fine in GUI mode for these tests so JMeter 3.x
>> > >should too.
>> > >> > > >
>> > >> > > > I'll give the nightly build a try.
>> > >> > > >
>> > >> > > > Tuukka
>> > >> > > >
>> > >> > > >
>> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
>> > >> > > > philippe.mouawad@gmail.com
>> > >> > > > > wrote:
>> > >> > > >
>> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> > >> > > > tuukka.mustonen@gmail.com
>> > >> > > > > >
>> > >> > > > > wrote:
>> > >> > > > >
>> > >> > > > > > @Philippe:
>> > >> > > > > >
>> > >> > > > > > 1. Yeah, I compared them with the same plan, same
>machine.
>> > >Should
>> > >> > be
>> > >> > > > > > everything equal.
>> > >> > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
>> > >-Xmx512m)
>> > >> and
>> > >> > as
>> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't
>tweak
>> > >them much
>> > >> > > > > because I
>> > >> > > > > > don't believe the issue lies there.
>> > >> > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
>> > >> > > > > > 4. I am not familiar with that but I can try. If you
>have
>> > >> > > instructions
>> > >> > > > to
>> > >> > > > > > point to, please do.
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > > 5. I am running with GUI.
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > This is a known anti-pattern .
>> > >> > > > > switch to non gui mode if you want correct results.
>> > >> > > > > http://jmeter.apache.org/usermanual/best-practices.html
>> > >> > > > >
>> > >> > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
>> > >> tuning_tips/
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > > About file sending:
>> > >> > > > > >
>> > >> > > > > > 6. Nope. I'm just doing simple GETs.
>> > >> > > > >
>> > >> > > > >  Could you give nightly build a try ?
>> > >> > > > > http://jmeter.apache.org/nightly.html
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > > Tuukka
>> > >> > > > > >
>> > >> > > > > >
>> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
>> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
>> > >> > > > > > > wrote:
>> > >> > > > > >
>> > >> > > > > > > Also are you sending files in post body ?
>> > >> > > > > > >
>> > >> > > > > > > Thank you
>> > >> > > > > > >
>> > >> > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
>> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>>
>> > >> > > > > > > wrote:
>> > >> > > > > > >
>> > >> > > > > > > > Hello,
>> > >> > > > > > > > Few questions, please try to answer them all:
>> > >> > > > > > > >
>> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same
>plan/sape
>> > >> machine...
>> > >> > > > > > everything
>> > >> > > > > > > > equal ?
>> > >> > > > > > > > 2/ What are your Xmx settings
>> > >> > > > > > > > 3/ does your plan download embedded resources ?
>> > >> > > > > > > > 4/ when issue happens can you make 3 thread dumps
>at 5
>> > >> seconds
>> > >> > > > > interval
>> > >> > > > > > > > and send file ? (use jstack)
>> > >> > > > > > > > 5/ are you running gui or non gui ?
>> > >> > > > > > > >
>> > >> > > > > > > > Thanks
>> > >> > > > > > > > Regards
>> > >> > > > > > > >
>> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
>> > >> > > > > > > >
><javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
>> > >> > > > > <javascript:;>');>>
>> > >> > > > > > wrote:
>> > >> > > > > > > >
>> > >> > > > > > > >> Hi,
>> > >> > > > > > > >>
>> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience
>high
>> > >CPU
>> > >> > usage
>> > >> > > > > that
>> > >> > > > > > > >> renders JMeter unusable. CPU usage ramps quickly
>up
>> > >and
>> > >> after
>> > >> > > > having
>> > >> > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds),
>it
>> > >hits
>> > >> 100%
>> > >> > > on
>> > >> > > > my
>> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and
>I
>> > >have to
>> > >> > kill
>> > >> > > > it.
>> > >> > > > > > > >>
>> > >> > > > > > > >> On 2.13 everything goes smoothly (with default
>> > >settings).
>> > >> CPU
>> > >> > > > usage
>> > >> > > > > > > stays
>> > >> > > > > > > >> at ~30% even with 500 users.
>> > >> > > > > > > >>
>> > >> > > > > > > >> The experience is similar on both Windows 10 and
>Linux
>> > >> (Ubuntu
>> > >> > > > 14.04
>> > >> > > > > > LTS
>> > >> > > > > > > >> guest in Virtualbox).
>> > >> > > > > > > >>
>> > >> > > > > > > >> The problems I saw in JMeter 3:
>> > >> > > > > > > >>
>> > >> > > > > > > >> - java.net.SocketException: Permission denied:
>connect
>> > >> > > > > > > >> - After a while it runs out of heap space
>> > >> > > > > > > >>
>> > >> > > > > > > >> For "Permission denied: connect" I tried to set
>> > >> > > > > > > >> java.net.preferIPv4Stack=true
>> > >> > > > > > > >> (in a few places to make sure it would really be
>in
>> > >use) and
>> > >> > > tried
>> > >> > > > > to
>> > >> > > > > > > run
>> > >> > > > > > > >> JMeter as administrator (in Win). No effect.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I also tried increasing heap size, but that
>didn't
>> > >help
>> > >> > either.
>> > >> > > > And
>> > >> > > > > I
>> > >> > > > > > > >> think
>> > >> > > > > > > >> memory size is not the root cause here - rather
>the
>> > >result.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I am normally connected to VPN but I also tried
>> > >disabling
>> > >> it.
>> > >> > No
>> > >> > > > > > effect.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I tried disabling with and without IPv6 enabled
>in
>> > >Windows
>> > >> > > network
>> > >> > > > > > > >> connection settings. No effect.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I am running a few extra JMeter plugins. Tried
>cutting
>> > >my
>> > >> test
>> > >> > > > > > scenario
>> > >> > > > > > > to
>> > >> > > > > > > >> minimum. No effect.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I am running anti-virus software that I cannot
>> > >disable, due
>> > >> to
>> > >> > > > this
>> > >> > > > > > > being
>> > >> > > > > > > >> forced by IT department. However, it's not a
>problem
>> > >with
>> > >> 2.13
>> > >> > > so
>> > >> > > > I
>> > >> > > > > > > assume
>> > >> > > > > > > >> it shouldn't be a problem here either.
>> > >> > > > > > > >>
>> > >> > > > > > > >> FINALLY, once I set HTTP request implementation
>from
>> > >Apache
>> > >> > > > > > HttpClient 4
>> > >> > > > > > > >> to
>> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes -
>CPU
>> > >stays
>> > >> where
>> > >> > > it
>> > >> > > > > did
>> > >> > > > > > > with
>> > >> > > > > > > >> JMeter 2.13.
>> > >> > > > > > > >>
>> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win,
>update
>> > >111 on
>> > >> > > Linux).
>> > >> > > > > > > >>
>> > >> > > > > > > >> Why does (the recommended) HttpClient 4 cause
>problems
>> > >here?
>> > >> > Is
>> > >> > > > > this a
>> > >> > > > > > > >> know
>> > >> > > > > > > >> bug/issue? JMeter 3.x is running newer version
>> > >compared to
>> > >> > 2.13,
>> > >> > > > but
>> > >> > > > > > > what
>> > >> > > > > > > >> change in HttpClient 4 might cause the problems
>> > >described?
>> > >> > > > > > > >>
>> > >> > > > > > > >> Any ideas?
>> > >> > > > > > > >>
>> > >> > > > > > > >> Regards,
>> > >> > > > > > > >> Tuukka
>> > >> > > > > > > >>
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > > > --
>> > >> > > > > > > > Cordialement.
>> > >> > > > > > > > Philippe Mouawad.
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > > >
>> > >> > > > > > >
>> > >> > > > > > > --
>> > >> > > > > > > Cordialement.
>> > >> > > > > > > Philippe Mouawad.
>> > >> > > > > > >
>> > >> > > > > >
>> > >> > > > >
>> > >> > > > >
>> > >> > > > > --
>> > >> > > > > Cordialement.
>> > >> > > > > Philippe Mouawad.
>> > >> > > > >
>> > >> > > >
>> > >> > >
>> > >> >
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Cordialement.
>> > >> Philippe Mouawad.
>> > >>
>> >
>> >
>---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>> > For additional commands, e-mail: user-help@jmeter.apache.org
>> >
>> >
>>
>>
>> --
>> Cordialement.
>> Philippe Mouawad.
>>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

@Philippe:

0. Alright, I will.
1. Will do.
2. Sure.
3. To let the network connection state restore? Ok, will do.

I am having some other stuff I need to attend today, but I will get back to
this in the afternoon at latest.

Tuukka


On Wed, Feb 1, 2017 at 11:54 PM, Philippe Mouawad <
philippe.mouawad@gmail.com> wrote:

> Hello,
> I've created this bugzilla for investigation and to share with you the plan
> I have used to run my own tests:
> - https://bz.apache.org/bugzilla/show_bug.cgi?id=60676
>
> I use a Tomcat 8.5.11 with a localhost and a certificate generated and auto
> signed.
>
> So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan 2.5 Ghz
> Intel Core i7
> Java 8 u121
> -Xmx512m -Xms512m
> Test in GUI mode (ANTI-PATTERN)
>
> 2.13:
> - CPU is at 100%
> - No error
> - No leak
>
> 3.2 nightly with default settings :
> - CPU is at 100%
> - No error
> - No leak
>
> 3.2 nightly with this in user.properties :
> httpclient4.time_to_live=0
> httpclient4.validate_after_inactivity=0
>
> - CPU is at 100%
> - No error
> - No leak
>
> I attached the Heap Graphs to bugzilla and the plan.
>
> So for now I don't  reproduce your issue but I am not in the same
> configuration as I am not hitting the same website.
>
> So
> 0/ Could you provide by running only 5 requests, the response header that
> your website returns ?
> 1/ Could you try my test plan by replacing with your target host and see if
> behaviour is the same ?
> 2/ Ensure nothing is running (No Spotify :-) ) except your JMeter and
> monitor network as per Felix note
> 3/ Run 1 test after the other but ensure you wait a bit between each one
>
>
> Thanks
>
>
> On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
> felix.schumacher@internetallee.de> wrote:
>
> >
> >
> > Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
> > tuukka.mustonen@gmail.com>:
> > >Ok I re-ran the test.
> > >
> > >Took Heap Dump just before JMeter froze.
> > >
> > >Took Thread Dumps right after JMeter froze.
> > >
> > >https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
> >
> > That folder seems to be empty now.
> >
> > Are you running the tests with a security manager enabled?
> >
> > Can you share the test or at least the used requests?
> >
> > Are you using a proxy and if so, how is it configured?
> >
> > Is jmeter running locally, or over something like rdp?
> >
> > Can you look at the used network connections like netstat -etapn would do
> > on Linux?
> >
> > Regards,
> >  Felix
> >
> > >
> > >...
> > >
> > >For the fun of it, I also tried upgrading to the latest HttpClient
> > >4.5.3
> > >version. No effect.
> > >
> > >I also noticed that if I ping (google) during the perf testing, Windows
> > >ping reports "General failure." when JMeter breaks. After killing it,
> > >connectivity restores and there are replies from the server again. It
> > >seems
> > >like JMeter/HttpClient kills my whole network connection...
> > >
> > >Tuukka
> > >
> > >
> > >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> > >philippe.mouawad@gmail.com> wrote:
> > >
> > >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> > ><tuukka.mustonen@gmail.com
> > >> >
> > >> wrote:
> > >>
> > >> > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
> > >> slightly
> > >> > different behavior in my Linux guest. I'll write more about that
> > >later,
> > >> but
> > >> > for simplicity I'm testing only on Windows host now:
> > >> >
> > >> > Thread dumps are downloadable at:
> > >> > https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> > >> >
> > >> > (Took them with 7-8s interval I think, hope it's ok).
> > >> >
> > >> > I took the thread dumps once JMeter started filling in errors,
> > >which is
> > >> > somewhat after my CPU usage hit 100% (and music playback started to
> > >> distort
> > >> > etc.).
> > >> >
> > >> > I also created the head dump, note that I took it after running the
> > >test
> > >> > and after restarting JMeter, as JMeter froze during the test and I
> > >had to
> > >> > kill it. DId you mean I should take it *during* the test?
> > >>
> > >>
> > >> YES.
> > >>
> > >>
> > >> > Anyway, before
> > >> > sharing it I want to make sure it can't contain anything
> > >confidential?
> > >> > (parts of my test plan, passwords, environment variables,
> > >anything...)
> > >> >
> > >> > Tuukka
> > >> >
> > >> >
> > >> >
> > >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > >> > support@ubikloadpack.com> wrote:
> > >> >
> > >> > > Hello,
> > >> > > To investigate the problem:
> > >> > > 1/ to create  thread dumps, install a JDK of same version as the
> > >one
> > >> > > running your JMeter.
> > >> > >
> > >> > > Run until you get the problem, then call using same user as the
> > >one
> > >> > running
> > >> > > JMeter 3 times at 5s interval:
> > >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> > >> > >
> > >> > > It will also help if you provide the structure of your Test Plan.
> > >> > >
> > >> > > 2/ It would also help if you can provide a Heap Dump. With JMeter
> > >GUI,
> > >> > use
> > >> > > menu Help > Create a Heap Dump
> > >> > > This will create a Heap Dump (rather big file) , it would help a
> > >lot if
> > >> > you
> > >> > > can make it available.
> > >> > >
> > >> > > Regards
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > >> > tuukka.mustonen@gmail.com
> > >> > > >
> > >> > > wrote:
> > >> > >
> > >> > > > @Philippe:
> > >> > > >
> > >> > > > Yeah, I am normally running in non-GUI mode, but to
> > >> > > > study/inspect/understand this problem I am running in GUI mode.
> > >> JMeter
> > >> > > 2.13
> > >> > > > works just fine in GUI mode for these tests so JMeter 3.x
> > >should too.
> > >> > > >
> > >> > > > I'll give the nightly build a try.
> > >> > > >
> > >> > > > Tuukka
> > >> > > >
> > >> > > >
> > >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > >> > > > philippe.mouawad@gmail.com
> > >> > > > > wrote:
> > >> > > >
> > >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > >> > > > tuukka.mustonen@gmail.com
> > >> > > > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > @Philippe:
> > >> > > > > >
> > >> > > > > > 1. Yeah, I compared them with the same plan, same machine.
> > >Should
> > >> > be
> > >> > > > > > everything equal.
> > >> > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
> > >-Xmx512m)
> > >> and
> > >> > as
> > >> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak
> > >them much
> > >> > > > > because I
> > >> > > > > > don't believe the issue lies there.
> > >> > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > >> > > > > > 4. I am not familiar with that but I can try. If you have
> > >> > > instructions
> > >> > > > to
> > >> > > > > > point to, please do.
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > > 5. I am running with GUI.
> > >> > > > >
> > >> > > > >
> > >> > > > > This is a known anti-pattern .
> > >> > > > > switch to non gui mode if you want correct results.
> > >> > > > > http://jmeter.apache.org/usermanual/best-practices.html
> > >> > > > >
> > >> > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> > >> tuning_tips/
> > >> > > > >
> > >> > > > >
> > >> > > > > > About file sending:
> > >> > > > > >
> > >> > > > > > 6. Nope. I'm just doing simple GETs.
> > >> > > > >
> > >> > > > >  Could you give nightly build a try ?
> > >> > > > > http://jmeter.apache.org/nightly.html
> > >> > > > >
> > >> > > > >
> > >> > > > > > Tuukka
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> > >> > > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Also are you sending files in post body ?
> > >> > > > > > >
> > >> > > > > > > Thank you
> > >> > > > > > >
> > >> > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > >> > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hello,
> > >> > > > > > > > Few questions, please try to answer them all:
> > >> > > > > > > >
> > >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
> > >> machine...
> > >> > > > > > everything
> > >> > > > > > > > equal ?
> > >> > > > > > > > 2/ What are your Xmx settings
> > >> > > > > > > > 3/ does your plan download embedded resources ?
> > >> > > > > > > > 4/ when issue happens can you make 3 thread dumps at 5
> > >> seconds
> > >> > > > > interval
> > >> > > > > > > > and send file ? (use jstack)
> > >> > > > > > > > 5/ are you running gui or non gui ?
> > >> > > > > > > >
> > >> > > > > > > > Thanks
> > >> > > > > > > > Regards
> > >> > > > > > > >
> > >> > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > >> > > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > >> > > > > <javascript:;>');>>
> > >> > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > >> Hi,
> > >> > > > > > > >>
> > >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high
> > >CPU
> > >> > usage
> > >> > > > > that
> > >> > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up
> > >and
> > >> after
> > >> > > > having
> > >> > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it
> > >hits
> > >> 100%
> > >> > > on
> > >> > > > my
> > >> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I
> > >have to
> > >> > kill
> > >> > > > it.
> > >> > > > > > > >>
> > >> > > > > > > >> On 2.13 everything goes smoothly (with default
> > >settings).
> > >> CPU
> > >> > > > usage
> > >> > > > > > > stays
> > >> > > > > > > >> at ~30% even with 500 users.
> > >> > > > > > > >>
> > >> > > > > > > >> The experience is similar on both Windows 10 and Linux
> > >> (Ubuntu
> > >> > > > 14.04
> > >> > > > > > LTS
> > >> > > > > > > >> guest in Virtualbox).
> > >> > > > > > > >>
> > >> > > > > > > >> The problems I saw in JMeter 3:
> > >> > > > > > > >>
> > >> > > > > > > >> - java.net.SocketException: Permission denied: connect
> > >> > > > > > > >> - After a while it runs out of heap space
> > >> > > > > > > >>
> > >> > > > > > > >> For "Permission denied: connect" I tried to set
> > >> > > > > > > >> java.net.preferIPv4Stack=true
> > >> > > > > > > >> (in a few places to make sure it would really be in
> > >use) and
> > >> > > tried
> > >> > > > > to
> > >> > > > > > > run
> > >> > > > > > > >> JMeter as administrator (in Win). No effect.
> > >> > > > > > > >>
> > >> > > > > > > >> I also tried increasing heap size, but that didn't
> > >help
> > >> > either.
> > >> > > > And
> > >> > > > > I
> > >> > > > > > > >> think
> > >> > > > > > > >> memory size is not the root cause here - rather the
> > >result.
> > >> > > > > > > >>
> > >> > > > > > > >> I am normally connected to VPN but I also tried
> > >disabling
> > >> it.
> > >> > No
> > >> > > > > > effect.
> > >> > > > > > > >>
> > >> > > > > > > >> I tried disabling with and without IPv6 enabled in
> > >Windows
> > >> > > network
> > >> > > > > > > >> connection settings. No effect.
> > >> > > > > > > >>
> > >> > > > > > > >> I am running a few extra JMeter plugins. Tried cutting
> > >my
> > >> test
> > >> > > > > > scenario
> > >> > > > > > > to
> > >> > > > > > > >> minimum. No effect.
> > >> > > > > > > >>
> > >> > > > > > > >> I am running anti-virus software that I cannot
> > >disable, due
> > >> to
> > >> > > > this
> > >> > > > > > > being
> > >> > > > > > > >> forced by IT department. However, it's not a problem
> > >with
> > >> 2.13
> > >> > > so
> > >> > > > I
> > >> > > > > > > assume
> > >> > > > > > > >> it shouldn't be a problem here either.
> > >> > > > > > > >>
> > >> > > > > > > >> FINALLY, once I set HTTP request implementation from
> > >Apache
> > >> > > > > > HttpClient 4
> > >> > > > > > > >> to
> > >> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU
> > >stays
> > >> where
> > >> > > it
> > >> > > > > did
> > >> > > > > > > with
> > >> > > > > > > >> JMeter 2.13.
> > >> > > > > > > >>
> > >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update
> > >111 on
> > >> > > Linux).
> > >> > > > > > > >>
> > >> > > > > > > >> Why does (the recommended) HttpClient 4 cause problems
> > >here?
> > >> > Is
> > >> > > > > this a
> > >> > > > > > > >> know
> > >> > > > > > > >> bug/issue? JMeter 3.x is running newer version
> > >compared to
> > >> > 2.13,
> > >> > > > but
> > >> > > > > > > what
> > >> > > > > > > >> change in HttpClient 4 might cause the problems
> > >described?
> > >> > > > > > > >>
> > >> > > > > > > >> Any ideas?
> > >> > > > > > > >>
> > >> > > > > > > >> Regards,
> > >> > > > > > > >> Tuukka
> > >> > > > > > > >>
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > --
> > >> > > > > > > > Cordialement.
> > >> > > > > > > > Philippe Mouawad.
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > > > --
> > >> > > > > > > Cordialement.
> > >> > > > > > > Philippe Mouawad.
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > --
> > >> > > > > Cordialement.
> > >> > > > > Philippe Mouawad.
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Cordialement.
> > >> Philippe Mouawad.
> > >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> > For additional commands, e-mail: user-help@jmeter.apache.org
> >
> >
>
>
> --
> Cordialement.
> Philippe Mouawad.
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

Hello,
I've created this bugzilla for investigation and to share with you the plan
I have used to run my own tests:
- https://bz.apache.org/bugzilla/show_bug.cgi?id=60676

I use a Tomcat 8.5.11 with a localhost and a certificate generated and auto
signed.

So here is what I get on a Mac Book Pro Mid 2015 OSX El Capitan 2.5 Ghz
Intel Core i7
Java 8 u121
-Xmx512m -Xms512m
Test in GUI mode (ANTI-PATTERN)

2.13:
- CPU is at 100%
- No error
- No leak

3.2 nightly with default settings :
- CPU is at 100%
- No error
- No leak

3.2 nightly with this in user.properties :
httpclient4.time_to_live=0
httpclient4.validate_after_inactivity=0

- CPU is at 100%
- No error
- No leak

I attached the Heap Graphs to bugzilla and the plan.

So for now I don't  reproduce your issue but I am not in the same
configuration as I am not hitting the same website.

So
0/ Could you provide by running only 5 requests, the response header that
your website returns ?
1/ Could you try my test plan by replacing with your target host and see if
behaviour is the same ?
2/ Ensure nothing is running (No Spotify :-) ) except your JMeter and
monitor network as per Felix note
3/ Run 1 test after the other but ensure you wait a bit between each one


Thanks


On Wed, Feb 1, 2017 at 10:00 PM, Felix Schumacher <
felix.schumacher@internetallee.de> wrote:

>
>
> Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
> tuukka.mustonen@gmail.com>:
> >Ok I re-ran the test.
> >
> >Took Heap Dump just before JMeter froze.
> >
> >Took Thread Dumps right after JMeter froze.
> >
> >https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
>
> That folder seems to be empty now.
>
> Are you running the tests with a security manager enabled?
>
> Can you share the test or at least the used requests?
>
> Are you using a proxy and if so, how is it configured?
>
> Is jmeter running locally, or over something like rdp?
>
> Can you look at the used network connections like netstat -etapn would do
> on Linux?
>
> Regards,
>  Felix
>
> >
> >...
> >
> >For the fun of it, I also tried upgrading to the latest HttpClient
> >4.5.3
> >version. No effect.
> >
> >I also noticed that if I ping (google) during the perf testing, Windows
> >ping reports "General failure." when JMeter breaks. After killing it,
> >connectivity restores and there are replies from the server again. It
> >seems
> >like JMeter/HttpClient kills my whole network connection...
> >
> >Tuukka
> >
> >
> >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> >philippe.mouawad@gmail.com> wrote:
> >
> >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> ><tuukka.mustonen@gmail.com
> >> >
> >> wrote:
> >>
> >> > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
> >> slightly
> >> > different behavior in my Linux guest. I'll write more about that
> >later,
> >> but
> >> > for simplicity I'm testing only on Windows host now:
> >> >
> >> > Thread dumps are downloadable at:
> >> > https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> >> >
> >> > (Took them with 7-8s interval I think, hope it's ok).
> >> >
> >> > I took the thread dumps once JMeter started filling in errors,
> >which is
> >> > somewhat after my CPU usage hit 100% (and music playback started to
> >> distort
> >> > etc.).
> >> >
> >> > I also created the head dump, note that I took it after running the
> >test
> >> > and after restarting JMeter, as JMeter froze during the test and I
> >had to
> >> > kill it. DId you mean I should take it *during* the test?
> >>
> >>
> >> YES.
> >>
> >>
> >> > Anyway, before
> >> > sharing it I want to make sure it can't contain anything
> >confidential?
> >> > (parts of my test plan, passwords, environment variables,
> >anything...)
> >> >
> >> > Tuukka
> >> >
> >> >
> >> >
> >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> >> > support@ubikloadpack.com> wrote:
> >> >
> >> > > Hello,
> >> > > To investigate the problem:
> >> > > 1/ to create  thread dumps, install a JDK of same version as the
> >one
> >> > > running your JMeter.
> >> > >
> >> > > Run until you get the problem, then call using same user as the
> >one
> >> > running
> >> > > JMeter 3 times at 5s interval:
> >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> >> > >
> >> > > It will also help if you provide the structure of your Test Plan.
> >> > >
> >> > > 2/ It would also help if you can provide a Heap Dump. With JMeter
> >GUI,
> >> > use
> >> > > menu Help > Create a Heap Dump
> >> > > This will create a Heap Dump (rather big file) , it would help a
> >lot if
> >> > you
> >> > > can make it available.
> >> > >
> >> > > Regards
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> >> > tuukka.mustonen@gmail.com
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > @Philippe:
> >> > > >
> >> > > > Yeah, I am normally running in non-GUI mode, but to
> >> > > > study/inspect/understand this problem I am running in GUI mode.
> >> JMeter
> >> > > 2.13
> >> > > > works just fine in GUI mode for these tests so JMeter 3.x
> >should too.
> >> > > >
> >> > > > I'll give the nightly build a try.
> >> > > >
> >> > > > Tuukka
> >> > > >
> >> > > >
> >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> >> > > > philippe.mouawad@gmail.com
> >> > > > > wrote:
> >> > > >
> >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> >> > > > tuukka.mustonen@gmail.com
> >> > > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > @Philippe:
> >> > > > > >
> >> > > > > > 1. Yeah, I compared them with the same plan, same machine.
> >Should
> >> > be
> >> > > > > > everything equal.
> >> > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
> >-Xmx512m)
> >> and
> >> > as
> >> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak
> >them much
> >> > > > > because I
> >> > > > > > don't believe the issue lies there.
> >> > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> >> > > > > > 4. I am not familiar with that but I can try. If you have
> >> > > instructions
> >> > > > to
> >> > > > > > point to, please do.
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > > 5. I am running with GUI.
> >> > > > >
> >> > > > >
> >> > > > > This is a known anti-pattern .
> >> > > > > switch to non gui mode if you want correct results.
> >> > > > > http://jmeter.apache.org/usermanual/best-practices.html
> >> > > > >
> >> > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> >> tuning_tips/
> >> > > > >
> >> > > > >
> >> > > > > > About file sending:
> >> > > > > >
> >> > > > > > 6. Nope. I'm just doing simple GETs.
> >> > > > >
> >> > > > >  Could you give nightly build a try ?
> >> > > > > http://jmeter.apache.org/nightly.html
> >> > > > >
> >> > > > >
> >> > > > > > Tuukka
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> >> > > > > > > wrote:
> >> > > > > >
> >> > > > > > > Also are you sending files in post body ?
> >> > > > > > >
> >> > > > > > > Thank you
> >> > > > > > >
> >> > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> >> > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hello,
> >> > > > > > > > Few questions, please try to answer them all:
> >> > > > > > > >
> >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
> >> machine...
> >> > > > > > everything
> >> > > > > > > > equal ?
> >> > > > > > > > 2/ What are your Xmx settings
> >> > > > > > > > 3/ does your plan download embedded resources ?
> >> > > > > > > > 4/ when issue happens can you make 3 thread dumps at 5
> >> seconds
> >> > > > > interval
> >> > > > > > > > and send file ? (use jstack)
> >> > > > > > > > 5/ are you running gui or non gui ?
> >> > > > > > > >
> >> > > > > > > > Thanks
> >> > > > > > > > Regards
> >> > > > > > > >
> >> > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> >> > > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> >> > > > > <javascript:;>');>>
> >> > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > >> Hi,
> >> > > > > > > >>
> >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high
> >CPU
> >> > usage
> >> > > > > that
> >> > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up
> >and
> >> after
> >> > > > having
> >> > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it
> >hits
> >> 100%
> >> > > on
> >> > > > my
> >> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I
> >have to
> >> > kill
> >> > > > it.
> >> > > > > > > >>
> >> > > > > > > >> On 2.13 everything goes smoothly (with default
> >settings).
> >> CPU
> >> > > > usage
> >> > > > > > > stays
> >> > > > > > > >> at ~30% even with 500 users.
> >> > > > > > > >>
> >> > > > > > > >> The experience is similar on both Windows 10 and Linux
> >> (Ubuntu
> >> > > > 14.04
> >> > > > > > LTS
> >> > > > > > > >> guest in Virtualbox).
> >> > > > > > > >>
> >> > > > > > > >> The problems I saw in JMeter 3:
> >> > > > > > > >>
> >> > > > > > > >> - java.net.SocketException: Permission denied: connect
> >> > > > > > > >> - After a while it runs out of heap space
> >> > > > > > > >>
> >> > > > > > > >> For "Permission denied: connect" I tried to set
> >> > > > > > > >> java.net.preferIPv4Stack=true
> >> > > > > > > >> (in a few places to make sure it would really be in
> >use) and
> >> > > tried
> >> > > > > to
> >> > > > > > > run
> >> > > > > > > >> JMeter as administrator (in Win). No effect.
> >> > > > > > > >>
> >> > > > > > > >> I also tried increasing heap size, but that didn't
> >help
> >> > either.
> >> > > > And
> >> > > > > I
> >> > > > > > > >> think
> >> > > > > > > >> memory size is not the root cause here - rather the
> >result.
> >> > > > > > > >>
> >> > > > > > > >> I am normally connected to VPN but I also tried
> >disabling
> >> it.
> >> > No
> >> > > > > > effect.
> >> > > > > > > >>
> >> > > > > > > >> I tried disabling with and without IPv6 enabled in
> >Windows
> >> > > network
> >> > > > > > > >> connection settings. No effect.
> >> > > > > > > >>
> >> > > > > > > >> I am running a few extra JMeter plugins. Tried cutting
> >my
> >> test
> >> > > > > > scenario
> >> > > > > > > to
> >> > > > > > > >> minimum. No effect.
> >> > > > > > > >>
> >> > > > > > > >> I am running anti-virus software that I cannot
> >disable, due
> >> to
> >> > > > this
> >> > > > > > > being
> >> > > > > > > >> forced by IT department. However, it's not a problem
> >with
> >> 2.13
> >> > > so
> >> > > > I
> >> > > > > > > assume
> >> > > > > > > >> it shouldn't be a problem here either.
> >> > > > > > > >>
> >> > > > > > > >> FINALLY, once I set HTTP request implementation from
> >Apache
> >> > > > > > HttpClient 4
> >> > > > > > > >> to
> >> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU
> >stays
> >> where
> >> > > it
> >> > > > > did
> >> > > > > > > with
> >> > > > > > > >> JMeter 2.13.
> >> > > > > > > >>
> >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update
> >111 on
> >> > > Linux).
> >> > > > > > > >>
> >> > > > > > > >> Why does (the recommended) HttpClient 4 cause problems
> >here?
> >> > Is
> >> > > > > this a
> >> > > > > > > >> know
> >> > > > > > > >> bug/issue? JMeter 3.x is running newer version
> >compared to
> >> > 2.13,
> >> > > > but
> >> > > > > > > what
> >> > > > > > > >> change in HttpClient 4 might cause the problems
> >described?
> >> > > > > > > >>
> >> > > > > > > >> Any ideas?
> >> > > > > > > >>
> >> > > > > > > >> Regards,
> >> > > > > > > >> Tuukka
> >> > > > > > > >>
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Cordialement.
> >> > > > > > > > Philippe Mouawad.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > Cordialement.
> >> > > > > > > Philippe Mouawad.
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Cordialement.
> >> > > > > Philippe Mouawad.
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Cordialement.
> >> Philippe Mouawad.
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>


-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

@Felix: Yeah, sorry about that. I had some private exchange with Philippe
who asked my to create new proper dumps. I guess it doesn't hurt to share
them with everyone. You can find them here:

JMeter 2.13:

https://drive.google.com/drive/folders/0B1uBdVuLED5NYTktXzI1TzlIOUk?
usp=sharing

JMeter 3.1:

https://drive.google.com/drive/folders/0B1uBdVuLED5NejZaanZ2UHNlblk?
usp=sharing

Answers to questions:

- I have not configured JVM so I guess I don't have security manager enabled
- For test plan, you can check
https://bz.apache.org/bugzilla/attachment.cgi?id=34703 that Philippe
simplified from my test plan.
- Not running proxy per se, but I am running over company's VPN connection.
However, I've also tried without it, and there was no difference.
- Running JMeter locally on my machine.
- I am testing solely on Windows now, as I think Virtualboxed linux guest
adds complexity to the system. I guess I could try sniffing what Windows
says about the connections, however.

Tuukka


On Wed, Feb 1, 2017 at 11:00 PM, Felix Schumacher <
felix.schumacher@internetallee.de> wrote:

>
>
> Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <
> tuukka.mustonen@gmail.com>:
> >Ok I re-ran the test.
> >
> >Took Heap Dump just before JMeter froze.
> >
> >Took Thread Dumps right after JMeter froze.
> >
> >https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA
>
> That folder seems to be empty now.
>
> Are you running the tests with a security manager enabled?
>
> Can you share the test or at least the used requests?
>
> Are you using a proxy and if so, how is it configured?
>
> Is jmeter running locally, or over something like rdp?
>
> Can you look at the used network connections like netstat -etapn would do
> on Linux?
>
> Regards,
>  Felix
>
> >
> >...
> >
> >For the fun of it, I also tried upgrading to the latest HttpClient
> >4.5.3
> >version. No effect.
> >
> >I also noticed that if I ping (google) during the perf testing, Windows
> >ping reports "General failure." when JMeter breaks. After killing it,
> >connectivity restores and there are replies from the server again. It
> >seems
> >like JMeter/HttpClient kills my whole network connection...
> >
> >Tuukka
> >
> >
> >On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
> >philippe.mouawad@gmail.com> wrote:
> >
> >> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
> ><tuukka.mustonen@gmail.com
> >> >
> >> wrote:
> >>
> >> > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
> >> slightly
> >> > different behavior in my Linux guest. I'll write more about that
> >later,
> >> but
> >> > for simplicity I'm testing only on Windows host now:
> >> >
> >> > Thread dumps are downloadable at:
> >> > https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> >> >
> >> > (Took them with 7-8s interval I think, hope it's ok).
> >> >
> >> > I took the thread dumps once JMeter started filling in errors,
> >which is
> >> > somewhat after my CPU usage hit 100% (and music playback started to
> >> distort
> >> > etc.).
> >> >
> >> > I also created the head dump, note that I took it after running the
> >test
> >> > and after restarting JMeter, as JMeter froze during the test and I
> >had to
> >> > kill it. DId you mean I should take it *during* the test?
> >>
> >>
> >> YES.
> >>
> >>
> >> > Anyway, before
> >> > sharing it I want to make sure it can't contain anything
> >confidential?
> >> > (parts of my test plan, passwords, environment variables,
> >anything...)
> >> >
> >> > Tuukka
> >> >
> >> >
> >> >
> >> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> >> > support@ubikloadpack.com> wrote:
> >> >
> >> > > Hello,
> >> > > To investigate the problem:
> >> > > 1/ to create  thread dumps, install a JDK of same version as the
> >one
> >> > > running your JMeter.
> >> > >
> >> > > Run until you get the problem, then call using same user as the
> >one
> >> > running
> >> > > JMeter 3 times at 5s interval:
> >> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> >> > >
> >> > > It will also help if you provide the structure of your Test Plan.
> >> > >
> >> > > 2/ It would also help if you can provide a Heap Dump. With JMeter
> >GUI,
> >> > use
> >> > > menu Help > Create a Heap Dump
> >> > > This will create a Heap Dump (rather big file) , it would help a
> >lot if
> >> > you
> >> > > can make it available.
> >> > >
> >> > > Regards
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> >> > tuukka.mustonen@gmail.com
> >> > > >
> >> > > wrote:
> >> > >
> >> > > > @Philippe:
> >> > > >
> >> > > > Yeah, I am normally running in non-GUI mode, but to
> >> > > > study/inspect/understand this problem I am running in GUI mode.
> >> JMeter
> >> > > 2.13
> >> > > > works just fine in GUI mode for these tests so JMeter 3.x
> >should too.
> >> > > >
> >> > > > I'll give the nightly build a try.
> >> > > >
> >> > > > Tuukka
> >> > > >
> >> > > >
> >> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> >> > > > philippe.mouawad@gmail.com
> >> > > > > wrote:
> >> > > >
> >> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> >> > > > tuukka.mustonen@gmail.com
> >> > > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > @Philippe:
> >> > > > > >
> >> > > > > > 1. Yeah, I compared them with the same plan, same machine.
> >Should
> >> > be
> >> > > > > > everything equal.
> >> > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
> >-Xmx512m)
> >> and
> >> > as
> >> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak
> >them much
> >> > > > > because I
> >> > > > > > don't believe the issue lies there.
> >> > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> >> > > > > > 4. I am not familiar with that but I can try. If you have
> >> > > instructions
> >> > > > to
> >> > > > > > point to, please do.
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > > 5. I am running with GUI.
> >> > > > >
> >> > > > >
> >> > > > > This is a known anti-pattern .
> >> > > > > switch to non gui mode if you want correct results.
> >> > > > > http://jmeter.apache.org/usermanual/best-practices.html
> >> > > > >
> >> > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> >> tuning_tips/
> >> > > > >
> >> > > > >
> >> > > > > > About file sending:
> >> > > > > >
> >> > > > > > 6. Nope. I'm just doing simple GETs.
> >> > > > >
> >> > > > >  Could you give nightly build a try ?
> >> > > > > http://jmeter.apache.org/nightly.html
> >> > > > >
> >> > > > >
> >> > > > > > Tuukka
> >> > > > > >
> >> > > > > >
> >> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> >> > > > > > philippe.mouawad@gmail.com <javascript:;>
> >> > > > > > > wrote:
> >> > > > > >
> >> > > > > > > Also are you sending files in post body ?
> >> > > > > > >
> >> > > > > > > Thank you
> >> > > > > > >
> >> > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> >> > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hello,
> >> > > > > > > > Few questions, please try to answer them all:
> >> > > > > > > >
> >> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
> >> machine...
> >> > > > > > everything
> >> > > > > > > > equal ?
> >> > > > > > > > 2/ What are your Xmx settings
> >> > > > > > > > 3/ does your plan download embedded resources ?
> >> > > > > > > > 4/ when issue happens can you make 3 thread dumps at 5
> >> seconds
> >> > > > > interval
> >> > > > > > > > and send file ? (use jstack)
> >> > > > > > > > 5/ are you running gui or non gui ?
> >> > > > > > > >
> >> > > > > > > > Thanks
> >> > > > > > > > Regards
> >> > > > > > > >
> >> > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> >> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> >> > > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> >> > > > > <javascript:;>');>>
> >> > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > >> Hi,
> >> > > > > > > >>
> >> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high
> >CPU
> >> > usage
> >> > > > > that
> >> > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up
> >and
> >> after
> >> > > > having
> >> > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it
> >hits
> >> 100%
> >> > > on
> >> > > > my
> >> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I
> >have to
> >> > kill
> >> > > > it.
> >> > > > > > > >>
> >> > > > > > > >> On 2.13 everything goes smoothly (with default
> >settings).
> >> CPU
> >> > > > usage
> >> > > > > > > stays
> >> > > > > > > >> at ~30% even with 500 users.
> >> > > > > > > >>
> >> > > > > > > >> The experience is similar on both Windows 10 and Linux
> >> (Ubuntu
> >> > > > 14.04
> >> > > > > > LTS
> >> > > > > > > >> guest in Virtualbox).
> >> > > > > > > >>
> >> > > > > > > >> The problems I saw in JMeter 3:
> >> > > > > > > >>
> >> > > > > > > >> - java.net.SocketException: Permission denied: connect
> >> > > > > > > >> - After a while it runs out of heap space
> >> > > > > > > >>
> >> > > > > > > >> For "Permission denied: connect" I tried to set
> >> > > > > > > >> java.net.preferIPv4Stack=true
> >> > > > > > > >> (in a few places to make sure it would really be in
> >use) and
> >> > > tried
> >> > > > > to
> >> > > > > > > run
> >> > > > > > > >> JMeter as administrator (in Win). No effect.
> >> > > > > > > >>
> >> > > > > > > >> I also tried increasing heap size, but that didn't
> >help
> >> > either.
> >> > > > And
> >> > > > > I
> >> > > > > > > >> think
> >> > > > > > > >> memory size is not the root cause here - rather the
> >result.
> >> > > > > > > >>
> >> > > > > > > >> I am normally connected to VPN but I also tried
> >disabling
> >> it.
> >> > No
> >> > > > > > effect.
> >> > > > > > > >>
> >> > > > > > > >> I tried disabling with and without IPv6 enabled in
> >Windows
> >> > > network
> >> > > > > > > >> connection settings. No effect.
> >> > > > > > > >>
> >> > > > > > > >> I am running a few extra JMeter plugins. Tried cutting
> >my
> >> test
> >> > > > > > scenario
> >> > > > > > > to
> >> > > > > > > >> minimum. No effect.
> >> > > > > > > >>
> >> > > > > > > >> I am running anti-virus software that I cannot
> >disable, due
> >> to
> >> > > > this
> >> > > > > > > being
> >> > > > > > > >> forced by IT department. However, it's not a problem
> >with
> >> 2.13
> >> > > so
> >> > > > I
> >> > > > > > > assume
> >> > > > > > > >> it shouldn't be a problem here either.
> >> > > > > > > >>
> >> > > > > > > >> FINALLY, once I set HTTP request implementation from
> >Apache
> >> > > > > > HttpClient 4
> >> > > > > > > >> to
> >> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU
> >stays
> >> where
> >> > > it
> >> > > > > did
> >> > > > > > > with
> >> > > > > > > >> JMeter 2.13.
> >> > > > > > > >>
> >> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update
> >111 on
> >> > > Linux).
> >> > > > > > > >>
> >> > > > > > > >> Why does (the recommended) HttpClient 4 cause problems
> >here?
> >> > Is
> >> > > > > this a
> >> > > > > > > >> know
> >> > > > > > > >> bug/issue? JMeter 3.x is running newer version
> >compared to
> >> > 2.13,
> >> > > > but
> >> > > > > > > what
> >> > > > > > > >> change in HttpClient 4 might cause the problems
> >described?
> >> > > > > > > >>
> >> > > > > > > >> Any ideas?
> >> > > > > > > >>
> >> > > > > > > >> Regards,
> >> > > > > > > >> Tuukka
> >> > > > > > > >>
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Cordialement.
> >> > > > > > > > Philippe Mouawad.
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > > > --
> >> > > > > > > Cordialement.
> >> > > > > > > Philippe Mouawad.
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > > >
> >> > > > > --
> >> > > > > Cordialement.
> >> > > > > Philippe Mouawad.
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >>
> >>
> >> --
> >> Cordialement.
> >> Philippe Mouawad.
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
>
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Felix Schumacher <fe...@internetallee.de>.


Am 1. Februar 2017 12:57:01 MEZ schrieb Tuukka Mustonen <tu...@gmail.com>:
>Ok I re-ran the test.
>
>Took Heap Dump just before JMeter froze.
>
>Took Thread Dumps right after JMeter froze.
>
>https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA

That folder seems to be empty now.

Are you running the tests with a security manager enabled?

Can you share the test or at least the used requests?

Are you using a proxy and if so, how is it configured?

Is jmeter running locally, or over something like rdp?

Can you look at the used network connections like netstat -etapn would do on Linux?

Regards,
 Felix

>
>...
>
>For the fun of it, I also tried upgrading to the latest HttpClient
>4.5.3
>version. No effect.
>
>I also noticed that if I ping (google) during the perf testing, Windows
>ping reports "General failure." when JMeter breaks. After killing it,
>connectivity restores and there are replies from the server again. It
>seems
>like JMeter/HttpClient kills my whole network connection...
>
>Tuukka
>
>
>On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
>philippe.mouawad@gmail.com> wrote:
>
>> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen
><tuukka.mustonen@gmail.com
>> >
>> wrote:
>>
>> > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
>> slightly
>> > different behavior in my Linux guest. I'll write more about that
>later,
>> but
>> > for simplicity I'm testing only on Windows host now:
>> >
>> > Thread dumps are downloadable at:
>> > https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
>> >
>> > (Took them with 7-8s interval I think, hope it's ok).
>> >
>> > I took the thread dumps once JMeter started filling in errors,
>which is
>> > somewhat after my CPU usage hit 100% (and music playback started to
>> distort
>> > etc.).
>> >
>> > I also created the head dump, note that I took it after running the
>test
>> > and after restarting JMeter, as JMeter froze during the test and I
>had to
>> > kill it. DId you mean I should take it *during* the test?
>>
>>
>> YES.
>>
>>
>> > Anyway, before
>> > sharing it I want to make sure it can't contain anything
>confidential?
>> > (parts of my test plan, passwords, environment variables,
>anything...)
>> >
>> > Tuukka
>> >
>> >
>> >
>> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
>> > support@ubikloadpack.com> wrote:
>> >
>> > > Hello,
>> > > To investigate the problem:
>> > > 1/ to create  thread dumps, install a JDK of same version as the
>one
>> > > running your JMeter.
>> > >
>> > > Run until you get the problem, then call using same user as the
>one
>> > running
>> > > JMeter 3 times at 5s interval:
>> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
>> > >
>> > > It will also help if you provide the structure of your Test Plan.
>> > >
>> > > 2/ It would also help if you can provide a Heap Dump. With JMeter
>GUI,
>> > use
>> > > menu Help > Create a Heap Dump
>> > > This will create a Heap Dump (rather big file) , it would help a
>lot if
>> > you
>> > > can make it available.
>> > >
>> > > Regards
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
>> > tuukka.mustonen@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > @Philippe:
>> > > >
>> > > > Yeah, I am normally running in non-GUI mode, but to
>> > > > study/inspect/understand this problem I am running in GUI mode.
>> JMeter
>> > > 2.13
>> > > > works just fine in GUI mode for these tests so JMeter 3.x
>should too.
>> > > >
>> > > > I'll give the nightly build a try.
>> > > >
>> > > > Tuukka
>> > > >
>> > > >
>> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
>> > > > philippe.mouawad@gmail.com
>> > > > > wrote:
>> > > >
>> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> > > > tuukka.mustonen@gmail.com
>> > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > @Philippe:
>> > > > > >
>> > > > > > 1. Yeah, I compared them with the same plan, same machine.
>Should
>> > be
>> > > > > > everything equal.
>> > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m
>-Xmx512m)
>> and
>> > as
>> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak
>them much
>> > > > > because I
>> > > > > > don't believe the issue lies there.
>> > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
>> > > > > > 4. I am not familiar with that but I can try. If you have
>> > > instructions
>> > > > to
>> > > > > > point to, please do.
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > > 5. I am running with GUI.
>> > > > >
>> > > > >
>> > > > > This is a known anti-pattern .
>> > > > > switch to non gui mode if you want correct results.
>> > > > > http://jmeter.apache.org/usermanual/best-practices.html
>> > > > >
>> > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
>> tuning_tips/
>> > > > >
>> > > > >
>> > > > > > About file sending:
>> > > > > >
>> > > > > > 6. Nope. I'm just doing simple GETs.
>> > > > >
>> > > > >  Could you give nightly build a try ?
>> > > > > http://jmeter.apache.org/nightly.html
>> > > > >
>> > > > >
>> > > > > > Tuukka
>> > > > > >
>> > > > > >
>> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
>> > > > > > philippe.mouawad@gmail.com <javascript:;>
>> > > > > > > wrote:
>> > > > > >
>> > > > > > > Also are you sending files in post body ?
>> > > > > > >
>> > > > > > > Thank you
>> > > > > > >
>> > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
>> > > > > > > philippe.mouawad@gmail.com <javascript:;>>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hello,
>> > > > > > > > Few questions, please try to answer them all:
>> > > > > > > >
>> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
>> machine...
>> > > > > > everything
>> > > > > > > > equal ?
>> > > > > > > > 2/ What are your Xmx settings
>> > > > > > > > 3/ does your plan download embedded resources ?
>> > > > > > > > 4/ when issue happens can you make 3 thread dumps at 5
>> seconds
>> > > > > interval
>> > > > > > > > and send file ? (use jstack)
>> > > > > > > > 5/ are you running gui or non gui ?
>> > > > > > > >
>> > > > > > > > Thanks
>> > > > > > > > Regards
>> > > > > > > >
>> > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
>> > > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
>> > > > > <javascript:;>');>>
>> > > > > > wrote:
>> > > > > > > >
>> > > > > > > >> Hi,
>> > > > > > > >>
>> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high
>CPU
>> > usage
>> > > > > that
>> > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up
>and
>> after
>> > > > having
>> > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it
>hits
>> 100%
>> > > on
>> > > > my
>> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I
>have to
>> > kill
>> > > > it.
>> > > > > > > >>
>> > > > > > > >> On 2.13 everything goes smoothly (with default
>settings).
>> CPU
>> > > > usage
>> > > > > > > stays
>> > > > > > > >> at ~30% even with 500 users.
>> > > > > > > >>
>> > > > > > > >> The experience is similar on both Windows 10 and Linux
>> (Ubuntu
>> > > > 14.04
>> > > > > > LTS
>> > > > > > > >> guest in Virtualbox).
>> > > > > > > >>
>> > > > > > > >> The problems I saw in JMeter 3:
>> > > > > > > >>
>> > > > > > > >> - java.net.SocketException: Permission denied: connect
>> > > > > > > >> - After a while it runs out of heap space
>> > > > > > > >>
>> > > > > > > >> For "Permission denied: connect" I tried to set
>> > > > > > > >> java.net.preferIPv4Stack=true
>> > > > > > > >> (in a few places to make sure it would really be in
>use) and
>> > > tried
>> > > > > to
>> > > > > > > run
>> > > > > > > >> JMeter as administrator (in Win). No effect.
>> > > > > > > >>
>> > > > > > > >> I also tried increasing heap size, but that didn't
>help
>> > either.
>> > > > And
>> > > > > I
>> > > > > > > >> think
>> > > > > > > >> memory size is not the root cause here - rather the
>result.
>> > > > > > > >>
>> > > > > > > >> I am normally connected to VPN but I also tried
>disabling
>> it.
>> > No
>> > > > > > effect.
>> > > > > > > >>
>> > > > > > > >> I tried disabling with and without IPv6 enabled in
>Windows
>> > > network
>> > > > > > > >> connection settings. No effect.
>> > > > > > > >>
>> > > > > > > >> I am running a few extra JMeter plugins. Tried cutting
>my
>> test
>> > > > > > scenario
>> > > > > > > to
>> > > > > > > >> minimum. No effect.
>> > > > > > > >>
>> > > > > > > >> I am running anti-virus software that I cannot
>disable, due
>> to
>> > > > this
>> > > > > > > being
>> > > > > > > >> forced by IT department. However, it's not a problem
>with
>> 2.13
>> > > so
>> > > > I
>> > > > > > > assume
>> > > > > > > >> it shouldn't be a problem here either.
>> > > > > > > >>
>> > > > > > > >> FINALLY, once I set HTTP request implementation from
>Apache
>> > > > > > HttpClient 4
>> > > > > > > >> to
>> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU
>stays
>> where
>> > > it
>> > > > > did
>> > > > > > > with
>> > > > > > > >> JMeter 2.13.
>> > > > > > > >>
>> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update
>111 on
>> > > Linux).
>> > > > > > > >>
>> > > > > > > >> Why does (the recommended) HttpClient 4 cause problems
>here?
>> > Is
>> > > > > this a
>> > > > > > > >> know
>> > > > > > > >> bug/issue? JMeter 3.x is running newer version
>compared to
>> > 2.13,
>> > > > but
>> > > > > > > what
>> > > > > > > >> change in HttpClient 4 might cause the problems
>described?
>> > > > > > > >>
>> > > > > > > >> Any ideas?
>> > > > > > > >>
>> > > > > > > >> Regards,
>> > > > > > > >> Tuukka
>> > > > > > > >>
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Cordialement.
>> > > > > > > > Philippe Mouawad.
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > > > --
>> > > > > > > Cordialement.
>> > > > > > > Philippe Mouawad.
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Cordialement.
>> > > > > Philippe Mouawad.
>> > > > >
>> > > >
>> > >
>> >
>>
>>
>>
>> --
>> Cordialement.
>> Philippe Mouawad.
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

Ok I re-ran the test.

Took Heap Dump just before JMeter froze.

Took Thread Dumps right after JMeter froze.

https://drive.google.com/drive/folders/0B1uBdVuLED5NNFNqallGam13MTA

...

For the fun of it, I also tried upgrading to the latest HttpClient 4.5.3
version. No effect.

I also noticed that if I ping (google) during the perf testing, Windows
ping reports "General failure." when JMeter breaks. After killing it,
connectivity restores and there are replies from the server again. It seems
like JMeter/HttpClient kills my whole network connection...

Tuukka


On Wed, Feb 1, 2017 at 11:44 AM, Philippe Mouawad <
philippe.mouawad@gmail.com> wrote:

> On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen <tuukka.mustonen@gmail.com
> >
> wrote:
>
> > Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
> slightly
> > different behavior in my Linux guest. I'll write more about that later,
> but
> > for simplicity I'm testing only on Windows host now:
> >
> > Thread dumps are downloadable at:
> > https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
> >
> > (Took them with 7-8s interval I think, hope it's ok).
> >
> > I took the thread dumps once JMeter started filling in errors, which is
> > somewhat after my CPU usage hit 100% (and music playback started to
> distort
> > etc.).
> >
> > I also created the head dump, note that I took it after running the test
> > and after restarting JMeter, as JMeter froze during the test and I had to
> > kill it. DId you mean I should take it *during* the test?
>
>
> YES.
>
>
> > Anyway, before
> > sharing it I want to make sure it can't contain anything confidential?
> > (parts of my test plan, passwords, environment variables, anything...)
> >
> > Tuukka
> >
> >
> >
> > On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> > support@ubikloadpack.com> wrote:
> >
> > > Hello,
> > > To investigate the problem:
> > > 1/ to create  thread dumps, install a JDK of same version as the one
> > > running your JMeter.
> > >
> > > Run until you get the problem, then call using same user as the one
> > running
> > > JMeter 3 times at 5s interval:
> > >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> > >
> > > It will also help if you provide the structure of your Test Plan.
> > >
> > > 2/ It would also help if you can provide a Heap Dump. With JMeter GUI,
> > use
> > > menu Help > Create a Heap Dump
> > > This will create a Heap Dump (rather big file) , it would help a lot if
> > you
> > > can make it available.
> > >
> > > Regards
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> > tuukka.mustonen@gmail.com
> > > >
> > > wrote:
> > >
> > > > @Philippe:
> > > >
> > > > Yeah, I am normally running in non-GUI mode, but to
> > > > study/inspect/understand this problem I am running in GUI mode.
> JMeter
> > > 2.13
> > > > works just fine in GUI mode for these tests so JMeter 3.x should too.
> > > >
> > > > I'll give the nightly build a try.
> > > >
> > > > Tuukka
> > > >
> > > >
> > > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > > > philippe.mouawad@gmail.com
> > > > > wrote:
> > > >
> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > tuukka.mustonen@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > @Philippe:
> > > > > >
> > > > > > 1. Yeah, I compared them with the same plan, same machine. Should
> > be
> > > > > > everything equal.
> > > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m -Xmx512m)
> and
> > as
> > > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them much
> > > > > because I
> > > > > > don't believe the issue lies there.
> > > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > > > > > 4. I am not familiar with that but I can try. If you have
> > > instructions
> > > > to
> > > > > > point to, please do.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > 5. I am running with GUI.
> > > > >
> > > > >
> > > > > This is a known anti-pattern .
> > > > > switch to non gui mode if you want correct results.
> > > > > http://jmeter.apache.org/usermanual/best-practices.html
> > > > >
> > > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_
> tuning_tips/
> > > > >
> > > > >
> > > > > > About file sending:
> > > > > >
> > > > > > 6. Nope. I'm just doing simple GETs.
> > > > >
> > > > >  Could you give nightly build a try ?
> > > > > http://jmeter.apache.org/nightly.html
> > > > >
> > > > >
> > > > > > Tuukka
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > > > wrote:
> > > > > >
> > > > > > > Also are you sending files in post body ?
> > > > > > >
> > > > > > > Thank you
> > > > > > >
> > > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > > > > > philippe.mouawad@gmail.com <javascript:;>>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > > Few questions, please try to answer them all:
> > > > > > > >
> > > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape
> machine...
> > > > > > everything
> > > > > > > > equal ?
> > > > > > > > 2/ What are your Xmx settings
> > > > > > > > 3/ does your plan download embedded resources ?
> > > > > > > > 4/ when issue happens can you make 3 thread dumps at 5
> seconds
> > > > > interval
> > > > > > > > and send file ? (use jstack)
> > > > > > > > 5/ are you running gui or non gui ?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Regards
> > > > > > > >
> > > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > > > > <javascript:;>');>>
> > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi,
> > > > > > > >>
> > > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high CPU
> > usage
> > > > > that
> > > > > > > >> renders JMeter unusable. CPU usage ramps quickly up and
> after
> > > > having
> > > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it hits
> 100%
> > > on
> > > > my
> > > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I have to
> > kill
> > > > it.
> > > > > > > >>
> > > > > > > >> On 2.13 everything goes smoothly (with default settings).
> CPU
> > > > usage
> > > > > > > stays
> > > > > > > >> at ~30% even with 500 users.
> > > > > > > >>
> > > > > > > >> The experience is similar on both Windows 10 and Linux
> (Ubuntu
> > > > 14.04
> > > > > > LTS
> > > > > > > >> guest in Virtualbox).
> > > > > > > >>
> > > > > > > >> The problems I saw in JMeter 3:
> > > > > > > >>
> > > > > > > >> - java.net.SocketException: Permission denied: connect
> > > > > > > >> - After a while it runs out of heap space
> > > > > > > >>
> > > > > > > >> For "Permission denied: connect" I tried to set
> > > > > > > >> java.net.preferIPv4Stack=true
> > > > > > > >> (in a few places to make sure it would really be in use) and
> > > tried
> > > > > to
> > > > > > > run
> > > > > > > >> JMeter as administrator (in Win). No effect.
> > > > > > > >>
> > > > > > > >> I also tried increasing heap size, but that didn't help
> > either.
> > > > And
> > > > > I
> > > > > > > >> think
> > > > > > > >> memory size is not the root cause here - rather the result.
> > > > > > > >>
> > > > > > > >> I am normally connected to VPN but I also tried disabling
> it.
> > No
> > > > > > effect.
> > > > > > > >>
> > > > > > > >> I tried disabling with and without IPv6 enabled in Windows
> > > network
> > > > > > > >> connection settings. No effect.
> > > > > > > >>
> > > > > > > >> I am running a few extra JMeter plugins. Tried cutting my
> test
> > > > > > scenario
> > > > > > > to
> > > > > > > >> minimum. No effect.
> > > > > > > >>
> > > > > > > >> I am running anti-virus software that I cannot disable, due
> to
> > > > this
> > > > > > > being
> > > > > > > >> forced by IT department. However, it's not a problem with
> 2.13
> > > so
> > > > I
> > > > > > > assume
> > > > > > > >> it shouldn't be a problem here either.
> > > > > > > >>
> > > > > > > >> FINALLY, once I set HTTP request implementation from Apache
> > > > > > HttpClient 4
> > > > > > > >> to
> > > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays
> where
> > > it
> > > > > did
> > > > > > > with
> > > > > > > >> JMeter 2.13.
> > > > > > > >>
> > > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update 111 on
> > > Linux).
> > > > > > > >>
> > > > > > > >> Why does (the recommended) HttpClient 4 cause problems here?
> > Is
> > > > > this a
> > > > > > > >> know
> > > > > > > >> bug/issue? JMeter 3.x is running newer version compared to
> > 2.13,
> > > > but
> > > > > > > what
> > > > > > > >> change in HttpClient 4 might cause the problems described?
> > > > > > > >>
> > > > > > > >> Any ideas?
> > > > > > > >>
> > > > > > > >> Regards,
> > > > > > > >> Tuukka
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Cordialement.
> > > > > > > > Philippe Mouawad.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Cordialement.
> > > > > > > Philippe Mouawad.
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cordialement.
> > > > > Philippe Mouawad.
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

On Wed, Feb 1, 2017 at 9:55 AM, Tuukka Mustonen <tu...@gmail.com>
wrote:

> Hi, I'm running solely on Win 10 now. In re-attempts I think I saw slightly
> different behavior in my Linux guest. I'll write more about that later, but
> for simplicity I'm testing only on Windows host now:
>
> Thread dumps are downloadable at:
> https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA
>
> (Took them with 7-8s interval I think, hope it's ok).
>
> I took the thread dumps once JMeter started filling in errors, which is
> somewhat after my CPU usage hit 100% (and music playback started to distort
> etc.).
>
> I also created the head dump, note that I took it after running the test
> and after restarting JMeter, as JMeter froze during the test and I had to
> kill it. DId you mean I should take it *during* the test?


YES.


> Anyway, before
> sharing it I want to make sure it can't contain anything confidential?
> (parts of my test plan, passwords, environment variables, anything...)
>
> Tuukka
>
>
>
> On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> support@ubikloadpack.com> wrote:
>
> > Hello,
> > To investigate the problem:
> > 1/ to create  thread dumps, install a JDK of same version as the one
> > running your JMeter.
> >
> > Run until you get the problem, then call using same user as the one
> running
> > JMeter 3 times at 5s interval:
> >  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
> >
> > It will also help if you provide the structure of your Test Plan.
> >
> > 2/ It would also help if you can provide a Heap Dump. With JMeter GUI,
> use
> > menu Help > Create a Heap Dump
> > This will create a Heap Dump (rather big file) , it would help a lot if
> you
> > can make it available.
> >
> > Regards
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
> tuukka.mustonen@gmail.com
> > >
> > wrote:
> >
> > > @Philippe:
> > >
> > > Yeah, I am normally running in non-GUI mode, but to
> > > study/inspect/understand this problem I am running in GUI mode. JMeter
> > 2.13
> > > works just fine in GUI mode for these tests so JMeter 3.x should too.
> > >
> > > I'll give the nightly build a try.
> > >
> > > Tuukka
> > >
> > >
> > > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > > philippe.mouawad@gmail.com
> > > > wrote:
> > >
> > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > tuukka.mustonen@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > @Philippe:
> > > > >
> > > > > 1. Yeah, I compared them with the same plan, same machine. Should
> be
> > > > > everything equal.
> > > > > 2. I tried both the defaults in jmeter.bat (-Xms512m -Xmx512m) and
> as
> > > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them much
> > > > because I
> > > > > don't believe the issue lies there.
> > > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > > > > 4. I am not familiar with that but I can try. If you have
> > instructions
> > > to
> > > > > point to, please do.
> > > >
> > > >
> > > >
> > > >
> > > > > 5. I am running with GUI.
> > > >
> > > >
> > > > This is a known anti-pattern .
> > > > switch to non gui mode if you want correct results.
> > > > http://jmeter.apache.org/usermanual/best-practices.html
> > > >
> > > > http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
> > > >
> > > >
> > > > > About file sending:
> > > > >
> > > > > 6. Nope. I'm just doing simple GETs.
> > > >
> > > >  Could you give nightly build a try ?
> > > > http://jmeter.apache.org/nightly.html
> > > >
> > > >
> > > > > Tuukka
> > > > >
> > > > >
> > > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > > wrote:
> > > > >
> > > > > > Also are you sending files in post body ?
> > > > > >
> > > > > > Thank you
> > > > > >
> > > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > > > > philippe.mouawad@gmail.com <javascript:;>>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > > Few questions, please try to answer them all:
> > > > > > >
> > > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape machine...
> > > > > everything
> > > > > > > equal ?
> > > > > > > 2/ What are your Xmx settings
> > > > > > > 3/ does your plan download embedded resources ?
> > > > > > > 4/ when issue happens can you make 3 thread dumps at 5 seconds
> > > > interval
> > > > > > > and send file ? (use jstack)
> > > > > > > 5/ are you running gui or non gui ?
> > > > > > >
> > > > > > > Thanks
> > > > > > > Regards
> > > > > > >
> > > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > > > <javascript:;>');>>
> > > > > wrote:
> > > > > > >
> > > > > > >> Hi,
> > > > > > >>
> > > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high CPU
> usage
> > > > that
> > > > > > >> renders JMeter unusable. CPU usage ramps quickly up and after
> > > having
> > > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it hits 100%
> > on
> > > my
> > > > > > >> i7-4600. Shortly after, whole JMeter freezes and I have to
> kill
> > > it.
> > > > > > >>
> > > > > > >> On 2.13 everything goes smoothly (with default settings). CPU
> > > usage
> > > > > > stays
> > > > > > >> at ~30% even with 500 users.
> > > > > > >>
> > > > > > >> The experience is similar on both Windows 10 and Linux (Ubuntu
> > > 14.04
> > > > > LTS
> > > > > > >> guest in Virtualbox).
> > > > > > >>
> > > > > > >> The problems I saw in JMeter 3:
> > > > > > >>
> > > > > > >> - java.net.SocketException: Permission denied: connect
> > > > > > >> - After a while it runs out of heap space
> > > > > > >>
> > > > > > >> For "Permission denied: connect" I tried to set
> > > > > > >> java.net.preferIPv4Stack=true
> > > > > > >> (in a few places to make sure it would really be in use) and
> > tried
> > > > to
> > > > > > run
> > > > > > >> JMeter as administrator (in Win). No effect.
> > > > > > >>
> > > > > > >> I also tried increasing heap size, but that didn't help
> either.
> > > And
> > > > I
> > > > > > >> think
> > > > > > >> memory size is not the root cause here - rather the result.
> > > > > > >>
> > > > > > >> I am normally connected to VPN but I also tried disabling it.
> No
> > > > > effect.
> > > > > > >>
> > > > > > >> I tried disabling with and without IPv6 enabled in Windows
> > network
> > > > > > >> connection settings. No effect.
> > > > > > >>
> > > > > > >> I am running a few extra JMeter plugins. Tried cutting my test
> > > > > scenario
> > > > > > to
> > > > > > >> minimum. No effect.
> > > > > > >>
> > > > > > >> I am running anti-virus software that I cannot disable, due to
> > > this
> > > > > > being
> > > > > > >> forced by IT department. However, it's not a problem with 2.13
> > so
> > > I
> > > > > > assume
> > > > > > >> it shouldn't be a problem here either.
> > > > > > >>
> > > > > > >> FINALLY, once I set HTTP request implementation from Apache
> > > > > HttpClient 4
> > > > > > >> to
> > > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays where
> > it
> > > > did
> > > > > > with
> > > > > > >> JMeter 2.13.
> > > > > > >>
> > > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update 111 on
> > Linux).
> > > > > > >>
> > > > > > >> Why does (the recommended) HttpClient 4 cause problems here?
> Is
> > > > this a
> > > > > > >> know
> > > > > > >> bug/issue? JMeter 3.x is running newer version compared to
> 2.13,
> > > but
> > > > > > what
> > > > > > >> change in HttpClient 4 might cause the problems described?
> > > > > > >>
> > > > > > >> Any ideas?
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Tuukka
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Cordialement.
> > > > > > > Philippe Mouawad.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Cordialement.
> > > > > > Philippe Mouawad.
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Cordialement.
> > > > Philippe Mouawad.
> > > >
> > >
> >
>



-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

Oh, and I did try nightly build. It did not have effect.

The dumps were created in JMeter 3.1 that I assume should be more stable
than the nightly.

Tuukka


On Wed, Feb 1, 2017 at 10:55 AM, Tuukka Mustonen <tu...@gmail.com>
wrote:

> Hi, I'm running solely on Win 10 now. In re-attempts I think I saw
> slightly different behavior in my Linux guest. I'll write more about that
> later, but for simplicity I'm testing only on Windows host now:
>
> Thread dumps are downloadable at: https://drive.google.com/drive/folders/
> 0B1uBdVuLED5NS1RFbEFuSS1iejA
>
> (Took them with 7-8s interval I think, hope it's ok).
>
> I took the thread dumps once JMeter started filling in errors, which is
> somewhat after my CPU usage hit 100% (and music playback started to distort
> etc.).
>
> I also created the head dump, note that I took it after running the test
> and after restarting JMeter, as JMeter froze during the test and I had to
> kill it. DId you mean I should take it *during* the test? Anyway, before
> sharing it I want to make sure it can't contain anything confidential?
> (parts of my test plan, passwords, environment variables, anything...)
>
> Tuukka
>
>
>
> On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
> support@ubikloadpack.com> wrote:
>
>> Hello,
>> To investigate the problem:
>> 1/ to create  thread dumps, install a JDK of same version as the one
>> running your JMeter.
>>
>> Run until you get the problem, then call using same user as the one
>> running
>> JMeter 3 times at 5s interval:
>>  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
>>
>> It will also help if you provide the structure of your Test Plan.
>>
>> 2/ It would also help if you can provide a Heap Dump. With JMeter GUI, use
>> menu Help > Create a Heap Dump
>> This will create a Heap Dump (rather big file) , it would help a lot if
>> you
>> can make it available.
>>
>> Regards
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <
>> tuukka.mustonen@gmail.com>
>> wrote:
>>
>> > @Philippe:
>> >
>> > Yeah, I am normally running in non-GUI mode, but to
>> > study/inspect/understand this problem I am running in GUI mode. JMeter
>> 2.13
>> > works just fine in GUI mode for these tests so JMeter 3.x should too.
>> >
>> > I'll give the nightly build a try.
>> >
>> > Tuukka
>> >
>> >
>> > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
>> > philippe.mouawad@gmail.com
>> > > wrote:
>> >
>> > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> > tuukka.mustonen@gmail.com
>> > > >
>> > > wrote:
>> > >
>> > > > @Philippe:
>> > > >
>> > > > 1. Yeah, I compared them with the same plan, same machine. Should be
>> > > > everything equal.
>> > > > 2. I tried both the defaults in jmeter.bat (-Xms512m -Xmx512m) and
>> as
>> > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them much
>> > > because I
>> > > > don't believe the issue lies there.
>> > > > 3. No, I do just simple HTTP GETs (against RESTy API).
>> > > > 4. I am not familiar with that but I can try. If you have
>> instructions
>> > to
>> > > > point to, please do.
>> > >
>> > >
>> > >
>> > >
>> > > > 5. I am running with GUI.
>> > >
>> > >
>> > > This is a known anti-pattern .
>> > > switch to non gui mode if you want correct results.
>> > > http://jmeter.apache.org/usermanual/best-practices.html
>> > >
>> > > http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
>> > >
>> > >
>> > > > About file sending:
>> > > >
>> > > > 6. Nope. I'm just doing simple GETs.
>> > >
>> > >  Could you give nightly build a try ?
>> > > http://jmeter.apache.org/nightly.html
>> > >
>> > >
>> > > > Tuukka
>> > > >
>> > > >
>> > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
>> > > > philippe.mouawad@gmail.com <javascript:;>
>> > > > > wrote:
>> > > >
>> > > > > Also are you sending files in post body ?
>> > > > >
>> > > > > Thank you
>> > > > >
>> > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
>> > > > > philippe.mouawad@gmail.com <javascript:;>>
>> > > > > wrote:
>> > > > >
>> > > > > > Hello,
>> > > > > > Few questions, please try to answer them all:
>> > > > > >
>> > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape machine...
>> > > > everything
>> > > > > > equal ?
>> > > > > > 2/ What are your Xmx settings
>> > > > > > 3/ does your plan download embedded resources ?
>> > > > > > 4/ when issue happens can you make 3 thread dumps at 5 seconds
>> > > interval
>> > > > > > and send file ? (use jstack)
>> > > > > > 5/ are you running gui or non gui ?
>> > > > > >
>> > > > > > Thanks
>> > > > > > Regards
>> > > > > >
>> > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
>> > > > > tuukka.mustonen@gmail.com <javascript:;>
>> > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
>> > > <javascript:;>');>>
>> > > > wrote:
>> > > > > >
>> > > > > >> Hi,
>> > > > > >>
>> > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high CPU
>> usage
>> > > that
>> > > > > >> renders JMeter unusable. CPU usage ramps quickly up and after
>> > having
>> > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it hits 100%
>> on
>> > my
>> > > > > >> i7-4600. Shortly after, whole JMeter freezes and I have to kill
>> > it.
>> > > > > >>
>> > > > > >> On 2.13 everything goes smoothly (with default settings). CPU
>> > usage
>> > > > > stays
>> > > > > >> at ~30% even with 500 users.
>> > > > > >>
>> > > > > >> The experience is similar on both Windows 10 and Linux (Ubuntu
>> > 14.04
>> > > > LTS
>> > > > > >> guest in Virtualbox).
>> > > > > >>
>> > > > > >> The problems I saw in JMeter 3:
>> > > > > >>
>> > > > > >> - java.net.SocketException: Permission denied: connect
>> > > > > >> - After a while it runs out of heap space
>> > > > > >>
>> > > > > >> For "Permission denied: connect" I tried to set
>> > > > > >> java.net.preferIPv4Stack=true
>> > > > > >> (in a few places to make sure it would really be in use) and
>> tried
>> > > to
>> > > > > run
>> > > > > >> JMeter as administrator (in Win). No effect.
>> > > > > >>
>> > > > > >> I also tried increasing heap size, but that didn't help either.
>> > And
>> > > I
>> > > > > >> think
>> > > > > >> memory size is not the root cause here - rather the result.
>> > > > > >>
>> > > > > >> I am normally connected to VPN but I also tried disabling it.
>> No
>> > > > effect.
>> > > > > >>
>> > > > > >> I tried disabling with and without IPv6 enabled in Windows
>> network
>> > > > > >> connection settings. No effect.
>> > > > > >>
>> > > > > >> I am running a few extra JMeter plugins. Tried cutting my test
>> > > > scenario
>> > > > > to
>> > > > > >> minimum. No effect.
>> > > > > >>
>> > > > > >> I am running anti-virus software that I cannot disable, due to
>> > this
>> > > > > being
>> > > > > >> forced by IT department. However, it's not a problem with 2.13
>> so
>> > I
>> > > > > assume
>> > > > > >> it shouldn't be a problem here either.
>> > > > > >>
>> > > > > >> FINALLY, once I set HTTP request implementation from Apache
>> > > > HttpClient 4
>> > > > > >> to
>> > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays where
>> it
>> > > did
>> > > > > with
>> > > > > >> JMeter 2.13.
>> > > > > >>
>> > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update 111 on
>> Linux).
>> > > > > >>
>> > > > > >> Why does (the recommended) HttpClient 4 cause problems here? Is
>> > > this a
>> > > > > >> know
>> > > > > >> bug/issue? JMeter 3.x is running newer version compared to
>> 2.13,
>> > but
>> > > > > what
>> > > > > >> change in HttpClient 4 might cause the problems described?
>> > > > > >>
>> > > > > >> Any ideas?
>> > > > > >>
>> > > > > >> Regards,
>> > > > > >> Tuukka
>> > > > > >>
>> > > > > >
>> > > > > >
>> > > > > > --
>> > > > > > Cordialement.
>> > > > > > Philippe Mouawad.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > > > > --
>> > > > > Cordialement.
>> > > > > Philippe Mouawad.
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Cordialement.
>> > > Philippe Mouawad.
>> > >
>> >
>>
>
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

Hi, I'm running solely on Win 10 now. In re-attempts I think I saw slightly
different behavior in my Linux guest. I'll write more about that later, but
for simplicity I'm testing only on Windows host now:

Thread dumps are downloadable at:
https://drive.google.com/drive/folders/0B1uBdVuLED5NS1RFbEFuSS1iejA

(Took them with 7-8s interval I think, hope it's ok).

I took the thread dumps once JMeter started filling in errors, which is
somewhat after my CPU usage hit 100% (and music playback started to distort
etc.).

I also created the head dump, note that I took it after running the test
and after restarting JMeter, as JMeter froze during the test and I had to
kill it. DId you mean I should take it *during* the test? Anyway, before
sharing it I want to make sure it can't contain anything confidential?
(parts of my test plan, passwords, environment variables, anything...)

Tuukka



On Wed, Feb 1, 2017 at 10:10 AM, UBIK LOAD PACK Support <
support@ubikloadpack.com> wrote:

> Hello,
> To investigate the problem:
> 1/ to create  thread dumps, install a JDK of same version as the one
> running your JMeter.
>
> Run until you get the problem, then call using same user as the one running
> JMeter 3 times at 5s interval:
>  <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt
>
> It will also help if you provide the structure of your Test Plan.
>
> 2/ It would also help if you can provide a Heap Dump. With JMeter GUI, use
> menu Help > Create a Heap Dump
> This will create a Heap Dump (rather big file) , it would help a lot if you
> can make it available.
>
> Regards
>
>
>
>
>
>
>
> On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <tuukka.mustonen@gmail.com
> >
> wrote:
>
> > @Philippe:
> >
> > Yeah, I am normally running in non-GUI mode, but to
> > study/inspect/understand this problem I am running in GUI mode. JMeter
> 2.13
> > works just fine in GUI mode for these tests so JMeter 3.x should too.
> >
> > I'll give the nightly build a try.
> >
> > Tuukka
> >
> >
> > On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> > philippe.mouawad@gmail.com
> > > wrote:
> >
> > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > tuukka.mustonen@gmail.com
> > > >
> > > wrote:
> > >
> > > > @Philippe:
> > > >
> > > > 1. Yeah, I compared them with the same plan, same machine. Should be
> > > > everything equal.
> > > > 2. I tried both the defaults in jmeter.bat (-Xms512m -Xmx512m) and as
> > > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them much
> > > because I
> > > > don't believe the issue lies there.
> > > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > > > 4. I am not familiar with that but I can try. If you have
> instructions
> > to
> > > > point to, please do.
> > >
> > >
> > >
> > >
> > > > 5. I am running with GUI.
> > >
> > >
> > > This is a known anti-pattern .
> > > switch to non gui mode if you want correct results.
> > > http://jmeter.apache.org/usermanual/best-practices.html
> > >
> > > http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
> > >
> > >
> > > > About file sending:
> > > >
> > > > 6. Nope. I'm just doing simple GETs.
> > >
> > >  Could you give nightly build a try ?
> > > http://jmeter.apache.org/nightly.html
> > >
> > >
> > > > Tuukka
> > > >
> > > >
> > > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > > philippe.mouawad@gmail.com <javascript:;>
> > > > > wrote:
> > > >
> > > > > Also are you sending files in post body ?
> > > > >
> > > > > Thank you
> > > > >
> > > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > > > philippe.mouawad@gmail.com <javascript:;>>
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > > Few questions, please try to answer them all:
> > > > > >
> > > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape machine...
> > > > everything
> > > > > > equal ?
> > > > > > 2/ What are your Xmx settings
> > > > > > 3/ does your plan download embedded resources ?
> > > > > > 4/ when issue happens can you make 3 thread dumps at 5 seconds
> > > interval
> > > > > > and send file ? (use jstack)
> > > > > > 5/ are you running gui or non gui ?
> > > > > >
> > > > > > Thanks
> > > > > > Regards
> > > > > >
> > > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > > <javascript:;>');>>
> > > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high CPU usage
> > > that
> > > > > >> renders JMeter unusable. CPU usage ramps quickly up and after
> > having
> > > > > >> ~100-200 concurrent users or so (in ~30 seconds), it hits 100%
> on
> > my
> > > > > >> i7-4600. Shortly after, whole JMeter freezes and I have to kill
> > it.
> > > > > >>
> > > > > >> On 2.13 everything goes smoothly (with default settings). CPU
> > usage
> > > > > stays
> > > > > >> at ~30% even with 500 users.
> > > > > >>
> > > > > >> The experience is similar on both Windows 10 and Linux (Ubuntu
> > 14.04
> > > > LTS
> > > > > >> guest in Virtualbox).
> > > > > >>
> > > > > >> The problems I saw in JMeter 3:
> > > > > >>
> > > > > >> - java.net.SocketException: Permission denied: connect
> > > > > >> - After a while it runs out of heap space
> > > > > >>
> > > > > >> For "Permission denied: connect" I tried to set
> > > > > >> java.net.preferIPv4Stack=true
> > > > > >> (in a few places to make sure it would really be in use) and
> tried
> > > to
> > > > > run
> > > > > >> JMeter as administrator (in Win). No effect.
> > > > > >>
> > > > > >> I also tried increasing heap size, but that didn't help either.
> > And
> > > I
> > > > > >> think
> > > > > >> memory size is not the root cause here - rather the result.
> > > > > >>
> > > > > >> I am normally connected to VPN but I also tried disabling it. No
> > > > effect.
> > > > > >>
> > > > > >> I tried disabling with and without IPv6 enabled in Windows
> network
> > > > > >> connection settings. No effect.
> > > > > >>
> > > > > >> I am running a few extra JMeter plugins. Tried cutting my test
> > > > scenario
> > > > > to
> > > > > >> minimum. No effect.
> > > > > >>
> > > > > >> I am running anti-virus software that I cannot disable, due to
> > this
> > > > > being
> > > > > >> forced by IT department. However, it's not a problem with 2.13
> so
> > I
> > > > > assume
> > > > > >> it shouldn't be a problem here either.
> > > > > >>
> > > > > >> FINALLY, once I set HTTP request implementation from Apache
> > > > HttpClient 4
> > > > > >> to
> > > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays where
> it
> > > did
> > > > > with
> > > > > >> JMeter 2.13.
> > > > > >>
> > > > > >> I'm running Oracle JVM 8 (update 121 on Win, update 111 on
> Linux).
> > > > > >>
> > > > > >> Why does (the recommended) HttpClient 4 cause problems here? Is
> > > this a
> > > > > >> know
> > > > > >> bug/issue? JMeter 3.x is running newer version compared to 2.13,
> > but
> > > > > what
> > > > > >> change in HttpClient 4 might cause the problems described?
> > > > > >>
> > > > > >> Any ideas?
> > > > > >>
> > > > > >> Regards,
> > > > > >> Tuukka
> > > > > >>
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Cordialement.
> > > > > > Philippe Mouawad.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Cordialement.
> > > > > Philippe Mouawad.
> > > > >
> > > >
> > >
> > >
> > > --
> > > Cordialement.
> > > Philippe Mouawad.
> > >
> >
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by UBIK LOAD PACK Support <su...@ubikloadpack.com>.

Hello,
To investigate the problem:
1/ to create  thread dumps, install a JDK of same version as the one
running your JMeter.

Run until you get the problem, then call using same user as the one running
JMeter 3 times at 5s interval:
 <jdk_home>/bin/jstack -l <pid of JMeter> >> output_file.txt

It will also help if you provide the structure of your Test Plan.

2/ It would also help if you can provide a Heap Dump. With JMeter GUI, use
menu Help > Create a Heap Dump
This will create a Heap Dump (rather big file) , it would help a lot if you
can make it available.

Regards







On Wed, Feb 1, 2017 at 9:03 AM, Tuukka Mustonen <tu...@gmail.com>
wrote:

> @Philippe:
>
> Yeah, I am normally running in non-GUI mode, but to
> study/inspect/understand this problem I am running in GUI mode. JMeter 2.13
> works just fine in GUI mode for these tests so JMeter 3.x should too.
>
> I'll give the nightly build a try.
>
> Tuukka
>
>
> On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <
> philippe.mouawad@gmail.com
> > wrote:
>
> > On Wednesday, February 1, 2017, Tuukka Mustonen <
> tuukka.mustonen@gmail.com
> > >
> > wrote:
> >
> > > @Philippe:
> > >
> > > 1. Yeah, I compared them with the same plan, same machine. Should be
> > > everything equal.
> > > 2. I tried both the defaults in jmeter.bat (-Xms512m -Xmx512m) and as
> > > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them much
> > because I
> > > don't believe the issue lies there.
> > > 3. No, I do just simple HTTP GETs (against RESTy API).
> > > 4. I am not familiar with that but I can try. If you have instructions
> to
> > > point to, please do.
> >
> >
> >
> >
> > > 5. I am running with GUI.
> >
> >
> > This is a known anti-pattern .
> > switch to non gui mode if you want correct results.
> > http://jmeter.apache.org/usermanual/best-practices.html
> >
> > http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
> >
> >
> > > About file sending:
> > >
> > > 6. Nope. I'm just doing simple GETs.
> >
> >  Could you give nightly build a try ?
> > http://jmeter.apache.org/nightly.html
> >
> >
> > > Tuukka
> > >
> > >
> > > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > > philippe.mouawad@gmail.com <javascript:;>
> > > > wrote:
> > >
> > > > Also are you sending files in post body ?
> > > >
> > > > Thank you
> > > >
> > > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > > philippe.mouawad@gmail.com <javascript:;>>
> > > > wrote:
> > > >
> > > > > Hello,
> > > > > Few questions, please try to answer them all:
> > > > >
> > > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape machine...
> > > everything
> > > > > equal ?
> > > > > 2/ What are your Xmx settings
> > > > > 3/ does your plan download embedded resources ?
> > > > > 4/ when issue happens can you make 3 thread dumps at 5 seconds
> > interval
> > > > > and send file ? (use jstack)
> > > > > 5/ are you running gui or non gui ?
> > > > >
> > > > > Thanks
> > > > > Regards
> > > > >
> > > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > > tuukka.mustonen@gmail.com <javascript:;>
> > > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> > <javascript:;>');>>
> > > wrote:
> > > > >
> > > > >> Hi,
> > > > >>
> > > > >> After upgrading to JMeter 3.0 or 3.1 I experience high CPU usage
> > that
> > > > >> renders JMeter unusable. CPU usage ramps quickly up and after
> having
> > > > >> ~100-200 concurrent users or so (in ~30 seconds), it hits 100% on
> my
> > > > >> i7-4600. Shortly after, whole JMeter freezes and I have to kill
> it.
> > > > >>
> > > > >> On 2.13 everything goes smoothly (with default settings). CPU
> usage
> > > > stays
> > > > >> at ~30% even with 500 users.
> > > > >>
> > > > >> The experience is similar on both Windows 10 and Linux (Ubuntu
> 14.04
> > > LTS
> > > > >> guest in Virtualbox).
> > > > >>
> > > > >> The problems I saw in JMeter 3:
> > > > >>
> > > > >> - java.net.SocketException: Permission denied: connect
> > > > >> - After a while it runs out of heap space
> > > > >>
> > > > >> For "Permission denied: connect" I tried to set
> > > > >> java.net.preferIPv4Stack=true
> > > > >> (in a few places to make sure it would really be in use) and tried
> > to
> > > > run
> > > > >> JMeter as administrator (in Win). No effect.
> > > > >>
> > > > >> I also tried increasing heap size, but that didn't help either.
> And
> > I
> > > > >> think
> > > > >> memory size is not the root cause here - rather the result.
> > > > >>
> > > > >> I am normally connected to VPN but I also tried disabling it. No
> > > effect.
> > > > >>
> > > > >> I tried disabling with and without IPv6 enabled in Windows network
> > > > >> connection settings. No effect.
> > > > >>
> > > > >> I am running a few extra JMeter plugins. Tried cutting my test
> > > scenario
> > > > to
> > > > >> minimum. No effect.
> > > > >>
> > > > >> I am running anti-virus software that I cannot disable, due to
> this
> > > > being
> > > > >> forced by IT department. However, it's not a problem with 2.13 so
> I
> > > > assume
> > > > >> it shouldn't be a problem here either.
> > > > >>
> > > > >> FINALLY, once I set HTTP request implementation from Apache
> > > HttpClient 4
> > > > >> to
> > > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays where it
> > did
> > > > with
> > > > >> JMeter 2.13.
> > > > >>
> > > > >> I'm running Oracle JVM 8 (update 121 on Win, update 111 on Linux).
> > > > >>
> > > > >> Why does (the recommended) HttpClient 4 cause problems here? Is
> > this a
> > > > >> know
> > > > >> bug/issue? JMeter 3.x is running newer version compared to 2.13,
> but
> > > > what
> > > > >> change in HttpClient 4 might cause the problems described?
> > > > >>
> > > > >> Any ideas?
> > > > >>
> > > > >> Regards,
> > > > >> Tuukka
> > > > >>
> > > > >
> > > > >
> > > > > --
> > > > > Cordialement.
> > > > > Philippe Mouawad.
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > > --
> > > > Cordialement.
> > > > Philippe Mouawad.
> > > >
> > >
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

@Philippe:

Yeah, I am normally running in non-GUI mode, but to
study/inspect/understand this problem I am running in GUI mode. JMeter 2.13
works just fine in GUI mode for these tests so JMeter 3.x should too.

I'll give the nightly build a try.

Tuukka


On Wed, Feb 1, 2017 at 9:59 AM, Philippe Mouawad <philippe.mouawad@gmail.com
> wrote:

> On Wednesday, February 1, 2017, Tuukka Mustonen <tuukka.mustonen@gmail.com
> >
> wrote:
>
> > @Philippe:
> >
> > 1. Yeah, I compared them with the same plan, same machine. Should be
> > everything equal.
> > 2. I tried both the defaults in jmeter.bat (-Xms512m -Xmx512m) and as
> > slightly increased (-Xms512m -Xmx1024m). I didn't tweak them much
> because I
> > don't believe the issue lies there.
> > 3. No, I do just simple HTTP GETs (against RESTy API).
> > 4. I am not familiar with that but I can try. If you have instructions to
> > point to, please do.
>
>
>
>
> > 5. I am running with GUI.
>
>
> This is a known anti-pattern .
> switch to non gui mode if you want correct results.
> http://jmeter.apache.org/usermanual/best-practices.html
>
> http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
>
>
> > About file sending:
> >
> > 6. Nope. I'm just doing simple GETs.
>
>  Could you give nightly build a try ?
> http://jmeter.apache.org/nightly.html
>
>
> > Tuukka
> >
> >
> > On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> > philippe.mouawad@gmail.com <javascript:;>
> > > wrote:
> >
> > > Also are you sending files in post body ?
> > >
> > > Thank you
> > >
> > > On Wednesday, February 1, 2017, Philippe Mouawad <
> > > philippe.mouawad@gmail.com <javascript:;>>
> > > wrote:
> > >
> > > > Hello,
> > > > Few questions, please try to answer them all:
> > > >
> > > > 1/ Did you compare 2.13 and 3.1 with same plan/sape machine...
> > everything
> > > > equal ?
> > > > 2/ What are your Xmx settings
> > > > 3/ does your plan download embedded resources ?
> > > > 4/ when issue happens can you make 3 thread dumps at 5 seconds
> interval
> > > > and send file ? (use jstack)
> > > > 5/ are you running gui or non gui ?
> > > >
> > > > Thanks
> > > > Regards
> > > >
> > > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > > tuukka.mustonen@gmail.com <javascript:;>
> > > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com
> <javascript:;>');>>
> > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> After upgrading to JMeter 3.0 or 3.1 I experience high CPU usage
> that
> > > >> renders JMeter unusable. CPU usage ramps quickly up and after having
> > > >> ~100-200 concurrent users or so (in ~30 seconds), it hits 100% on my
> > > >> i7-4600. Shortly after, whole JMeter freezes and I have to kill it.
> > > >>
> > > >> On 2.13 everything goes smoothly (with default settings). CPU usage
> > > stays
> > > >> at ~30% even with 500 users.
> > > >>
> > > >> The experience is similar on both Windows 10 and Linux (Ubuntu 14.04
> > LTS
> > > >> guest in Virtualbox).
> > > >>
> > > >> The problems I saw in JMeter 3:
> > > >>
> > > >> - java.net.SocketException: Permission denied: connect
> > > >> - After a while it runs out of heap space
> > > >>
> > > >> For "Permission denied: connect" I tried to set
> > > >> java.net.preferIPv4Stack=true
> > > >> (in a few places to make sure it would really be in use) and tried
> to
> > > run
> > > >> JMeter as administrator (in Win). No effect.
> > > >>
> > > >> I also tried increasing heap size, but that didn't help either. And
> I
> > > >> think
> > > >> memory size is not the root cause here - rather the result.
> > > >>
> > > >> I am normally connected to VPN but I also tried disabling it. No
> > effect.
> > > >>
> > > >> I tried disabling with and without IPv6 enabled in Windows network
> > > >> connection settings. No effect.
> > > >>
> > > >> I am running a few extra JMeter plugins. Tried cutting my test
> > scenario
> > > to
> > > >> minimum. No effect.
> > > >>
> > > >> I am running anti-virus software that I cannot disable, due to this
> > > being
> > > >> forced by IT department. However, it's not a problem with 2.13 so I
> > > assume
> > > >> it shouldn't be a problem here either.
> > > >>
> > > >> FINALLY, once I set HTTP request implementation from Apache
> > HttpClient 4
> > > >> to
> > > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays where it
> did
> > > with
> > > >> JMeter 2.13.
> > > >>
> > > >> I'm running Oracle JVM 8 (update 121 on Win, update 111 on Linux).
> > > >>
> > > >> Why does (the recommended) HttpClient 4 cause problems here? Is
> this a
> > > >> know
> > > >> bug/issue? JMeter 3.x is running newer version compared to 2.13, but
> > > what
> > > >> change in HttpClient 4 might cause the problems described?
> > > >>
> > > >> Any ideas?
> > > >>
> > > >> Regards,
> > > >> Tuukka
> > > >>
> > > >
> > > >
> > > > --
> > > > Cordialement.
> > > > Philippe Mouawad.
> > > >
> > > >
> > > >
> > > >
> > >
> > > --
> > > Cordialement.
> > > Philippe Mouawad.
> > >
> >
>
>
> --
> Cordialement.
> Philippe Mouawad.
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

On Wednesday, February 1, 2017, Tuukka Mustonen <tu...@gmail.com>
wrote:

> @Philippe:
>
> 1. Yeah, I compared them with the same plan, same machine. Should be
> everything equal.
> 2. I tried both the defaults in jmeter.bat (-Xms512m -Xmx512m) and as
> slightly increased (-Xms512m -Xmx1024m). I didn't tweak them much because I
> don't believe the issue lies there.
> 3. No, I do just simple HTTP GETs (against RESTy API).
> 4. I am not familiar with that but I can try. If you have instructions to
> point to, please do.




> 5. I am running with GUI.


This is a known anti-pattern .
switch to non gui mode if you want correct results.
http://jmeter.apache.org/usermanual/best-practices.html

http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/


> About file sending:
>
> 6. Nope. I'm just doing simple GETs.

 Could you give nightly build a try ?
http://jmeter.apache.org/nightly.html


> Tuukka
>
>
> On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <
> philippe.mouawad@gmail.com <javascript:;>
> > wrote:
>
> > Also are you sending files in post body ?
> >
> > Thank you
> >
> > On Wednesday, February 1, 2017, Philippe Mouawad <
> > philippe.mouawad@gmail.com <javascript:;>>
> > wrote:
> >
> > > Hello,
> > > Few questions, please try to answer them all:
> > >
> > > 1/ Did you compare 2.13 and 3.1 with same plan/sape machine...
> everything
> > > equal ?
> > > 2/ What are your Xmx settings
> > > 3/ does your plan download embedded resources ?
> > > 4/ when issue happens can you make 3 thread dumps at 5 seconds interval
> > > and send file ? (use jstack)
> > > 5/ are you running gui or non gui ?
> > >
> > > Thanks
> > > Regards
> > >
> > > On Wednesday, February 1, 2017, Tuukka Mustonen <
> > tuukka.mustonen@gmail.com <javascript:;>
> > > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com <javascript:;>');>>
> wrote:
> > >
> > >> Hi,
> > >>
> > >> After upgrading to JMeter 3.0 or 3.1 I experience high CPU usage that
> > >> renders JMeter unusable. CPU usage ramps quickly up and after having
> > >> ~100-200 concurrent users or so (in ~30 seconds), it hits 100% on my
> > >> i7-4600. Shortly after, whole JMeter freezes and I have to kill it.
> > >>
> > >> On 2.13 everything goes smoothly (with default settings). CPU usage
> > stays
> > >> at ~30% even with 500 users.
> > >>
> > >> The experience is similar on both Windows 10 and Linux (Ubuntu 14.04
> LTS
> > >> guest in Virtualbox).
> > >>
> > >> The problems I saw in JMeter 3:
> > >>
> > >> - java.net.SocketException: Permission denied: connect
> > >> - After a while it runs out of heap space
> > >>
> > >> For "Permission denied: connect" I tried to set
> > >> java.net.preferIPv4Stack=true
> > >> (in a few places to make sure it would really be in use) and tried to
> > run
> > >> JMeter as administrator (in Win). No effect.
> > >>
> > >> I also tried increasing heap size, but that didn't help either. And I
> > >> think
> > >> memory size is not the root cause here - rather the result.
> > >>
> > >> I am normally connected to VPN but I also tried disabling it. No
> effect.
> > >>
> > >> I tried disabling with and without IPv6 enabled in Windows network
> > >> connection settings. No effect.
> > >>
> > >> I am running a few extra JMeter plugins. Tried cutting my test
> scenario
> > to
> > >> minimum. No effect.
> > >>
> > >> I am running anti-virus software that I cannot disable, due to this
> > being
> > >> forced by IT department. However, it's not a problem with 2.13 so I
> > assume
> > >> it shouldn't be a problem here either.
> > >>
> > >> FINALLY, once I set HTTP request implementation from Apache
> HttpClient 4
> > >> to
> > >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays where it did
> > with
> > >> JMeter 2.13.
> > >>
> > >> I'm running Oracle JVM 8 (update 121 on Win, update 111 on Linux).
> > >>
> > >> Why does (the recommended) HttpClient 4 cause problems here? Is this a
> > >> know
> > >> bug/issue? JMeter 3.x is running newer version compared to 2.13, but
> > what
> > >> change in HttpClient 4 might cause the problems described?
> > >>
> > >> Any ideas?
> > >>
> > >> Regards,
> > >> Tuukka
> > >>
> > >
> > >
> > > --
> > > Cordialement.
> > > Philippe Mouawad.
> > >
> > >
> > >
> > >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
>


-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Tuukka Mustonen <tu...@gmail.com>.

@Philippe:

1. Yeah, I compared them with the same plan, same machine. Should be
everything equal.
2. I tried both the defaults in jmeter.bat (-Xms512m -Xmx512m) and as
slightly increased (-Xms512m -Xmx1024m). I didn't tweak them much because I
don't believe the issue lies there.
3. No, I do just simple HTTP GETs (against RESTy API).
4. I am not familiar with that but I can try. If you have instructions to
point to, please do.
5. I am running with GUI.

About file sending:

6. Nope. I'm just doing simple GETs.

Tuukka


On Wed, Feb 1, 2017 at 9:43 AM, Philippe Mouawad <philippe.mouawad@gmail.com
> wrote:

> Also are you sending files in post body ?
>
> Thank you
>
> On Wednesday, February 1, 2017, Philippe Mouawad <
> philippe.mouawad@gmail.com>
> wrote:
>
> > Hello,
> > Few questions, please try to answer them all:
> >
> > 1/ Did you compare 2.13 and 3.1 with same plan/sape machine... everything
> > equal ?
> > 2/ What are your Xmx settings
> > 3/ does your plan download embedded resources ?
> > 4/ when issue happens can you make 3 thread dumps at 5 seconds interval
> > and send file ? (use jstack)
> > 5/ are you running gui or non gui ?
> >
> > Thanks
> > Regards
> >
> > On Wednesday, February 1, 2017, Tuukka Mustonen <
> tuukka.mustonen@gmail.com
> > <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com');>> wrote:
> >
> >> Hi,
> >>
> >> After upgrading to JMeter 3.0 or 3.1 I experience high CPU usage that
> >> renders JMeter unusable. CPU usage ramps quickly up and after having
> >> ~100-200 concurrent users or so (in ~30 seconds), it hits 100% on my
> >> i7-4600. Shortly after, whole JMeter freezes and I have to kill it.
> >>
> >> On 2.13 everything goes smoothly (with default settings). CPU usage
> stays
> >> at ~30% even with 500 users.
> >>
> >> The experience is similar on both Windows 10 and Linux (Ubuntu 14.04 LTS
> >> guest in Virtualbox).
> >>
> >> The problems I saw in JMeter 3:
> >>
> >> - java.net.SocketException: Permission denied: connect
> >> - After a while it runs out of heap space
> >>
> >> For "Permission denied: connect" I tried to set
> >> java.net.preferIPv4Stack=true
> >> (in a few places to make sure it would really be in use) and tried to
> run
> >> JMeter as administrator (in Win). No effect.
> >>
> >> I also tried increasing heap size, but that didn't help either. And I
> >> think
> >> memory size is not the root cause here - rather the result.
> >>
> >> I am normally connected to VPN but I also tried disabling it. No effect.
> >>
> >> I tried disabling with and without IPv6 enabled in Windows network
> >> connection settings. No effect.
> >>
> >> I am running a few extra JMeter plugins. Tried cutting my test scenario
> to
> >> minimum. No effect.
> >>
> >> I am running anti-virus software that I cannot disable, due to this
> being
> >> forced by IT department. However, it's not a problem with 2.13 so I
> assume
> >> it shouldn't be a problem here either.
> >>
> >> FINALLY, once I set HTTP request implementation from Apache HttpClient 4
> >> to
> >> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays where it did
> with
> >> JMeter 2.13.
> >>
> >> I'm running Oracle JVM 8 (update 121 on Win, update 111 on Linux).
> >>
> >> Why does (the recommended) HttpClient 4 cause problems here? Is this a
> >> know
> >> bug/issue? JMeter 3.x is running newer version compared to 2.13, but
> what
> >> change in HttpClient 4 might cause the problems described?
> >>
> >> Any ideas?
> >>
> >> Regards,
> >> Tuukka
> >>
> >
> >
> > --
> > Cordialement.
> > Philippe Mouawad.
> >
> >
> >
> >
>
> --
> Cordialement.
> Philippe Mouawad.
>

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

Also are you sending files in post body ?

Thank you

On Wednesday, February 1, 2017, Philippe Mouawad <ph...@gmail.com>
wrote:

> Hello,
> Few questions, please try to answer them all:
>
> 1/ Did you compare 2.13 and 3.1 with same plan/sape machine... everything
> equal ?
> 2/ What are your Xmx settings
> 3/ does your plan download embedded resources ?
> 4/ when issue happens can you make 3 thread dumps at 5 seconds interval
> and send file ? (use jstack)
> 5/ are you running gui or non gui ?
>
> Thanks
> Regards
>
> On Wednesday, February 1, 2017, Tuukka Mustonen <tuukka.mustonen@gmail.com
> <javascript:_e(%7B%7D,'cvml','tuukka.mustonen@gmail.com');>> wrote:
>
>> Hi,
>>
>> After upgrading to JMeter 3.0 or 3.1 I experience high CPU usage that
>> renders JMeter unusable. CPU usage ramps quickly up and after having
>> ~100-200 concurrent users or so (in ~30 seconds), it hits 100% on my
>> i7-4600. Shortly after, whole JMeter freezes and I have to kill it.
>>
>> On 2.13 everything goes smoothly (with default settings). CPU usage stays
>> at ~30% even with 500 users.
>>
>> The experience is similar on both Windows 10 and Linux (Ubuntu 14.04 LTS
>> guest in Virtualbox).
>>
>> The problems I saw in JMeter 3:
>>
>> - java.net.SocketException: Permission denied: connect
>> - After a while it runs out of heap space
>>
>> For "Permission denied: connect" I tried to set
>> java.net.preferIPv4Stack=true
>> (in a few places to make sure it would really be in use) and tried to run
>> JMeter as administrator (in Win). No effect.
>>
>> I also tried increasing heap size, but that didn't help either. And I
>> think
>> memory size is not the root cause here - rather the result.
>>
>> I am normally connected to VPN but I also tried disabling it. No effect.
>>
>> I tried disabling with and without IPv6 enabled in Windows network
>> connection settings. No effect.
>>
>> I am running a few extra JMeter plugins. Tried cutting my test scenario to
>> minimum. No effect.
>>
>> I am running anti-virus software that I cannot disable, due to this being
>> forced by IT department. However, it's not a problem with 2.13 so I assume
>> it shouldn't be a problem here either.
>>
>> FINALLY, once I set HTTP request implementation from Apache HttpClient 4
>> to
>> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays where it did with
>> JMeter 2.13.
>>
>> I'm running Oracle JVM 8 (update 121 on Win, update 111 on Linux).
>>
>> Why does (the recommended) HttpClient 4 cause problems here? Is this a
>> know
>> bug/issue? JMeter 3.x is running newer version compared to 2.13, but what
>> change in HttpClient 4 might cause the problems described?
>>
>> Any ideas?
>>
>> Regards,
>> Tuukka
>>
>
>
> --
> Cordialement.
> Philippe Mouawad.
>
>
>
>

-- 
Cordialement.
Philippe Mouawad.

Re: High CPU utilization in JMeter 3.x with HttpClient 4 leads to freeze

Posted by Philippe Mouawad <ph...@gmail.com>.

Hello,
Few questions, please try to answer them all:

1/ Did you compare 2.13 and 3.1 with same plan/sape machine... everything
equal ?
2/ What are your Xmx settings
3/ does your plan download embedded resources ?
4/ when issue happens can you make 3 thread dumps at 5 seconds interval and
send file ? (use jstack)
5/ are you running gui or non gui ?

Thanks
Regards

On Wednesday, February 1, 2017, Tuukka Mustonen <tu...@gmail.com>
wrote:

> Hi,
>
> After upgrading to JMeter 3.0 or 3.1 I experience high CPU usage that
> renders JMeter unusable. CPU usage ramps quickly up and after having
> ~100-200 concurrent users or so (in ~30 seconds), it hits 100% on my
> i7-4600. Shortly after, whole JMeter freezes and I have to kill it.
>
> On 2.13 everything goes smoothly (with default settings). CPU usage stays
> at ~30% even with 500 users.
>
> The experience is similar on both Windows 10 and Linux (Ubuntu 14.04 LTS
> guest in Virtualbox).
>
> The problems I saw in JMeter 3:
>
> - java.net.SocketException: Permission denied: connect
> - After a while it runs out of heap space
>
> For "Permission denied: connect" I tried to set
> java.net.preferIPv4Stack=true
> (in a few places to make sure it would really be in use) and tried to run
> JMeter as administrator (in Win). No effect.
>
> I also tried increasing heap size, but that didn't help either. And I think
> memory size is not the root cause here - rather the result.
>
> I am normally connected to VPN but I also tried disabling it. No effect.
>
> I tried disabling with and without IPv6 enabled in Windows network
> connection settings. No effect.
>
> I am running a few extra JMeter plugins. Tried cutting my test scenario to
> minimum. No effect.
>
> I am running anti-virus software that I cannot disable, due to this being
> forced by IT department. However, it's not a problem with 2.13 so I assume
> it shouldn't be a problem here either.
>
> FINALLY, once I set HTTP request implementation from Apache HttpClient 4 to
> HttpClient 3.1 or JAVA, the problem vanishes - CPU stays where it did with
> JMeter 2.13.
>
> I'm running Oracle JVM 8 (update 121 on Win, update 111 on Linux).
>
> Why does (the recommended) HttpClient 4 cause problems here? Is this a know
> bug/issue? JMeter 3.x is running newer version compared to 2.13, but what
> change in HttpClient 4 might cause the problems described?
>
> Any ideas?
>
> Regards,
> Tuukka
>


-- 
Cordialement.
Philippe Mouawad.