You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Antony Bowesman <An...@williamhill.com.au> on 2018/03/19 02:37:35 UTC
Feedback and question re Java memory/GC settings for Jmeter load
generator CPU issue
Hi,
I just thought I'd send in some info about a problem I've been looking at recently - with a question of best GC settings
I have a number of JMeter load generators (LG) and I have been seeing CPU spikes on the boxes during a test. I am monitoring CPU and memory from within a Java sampler, so have the following charts
1. First chart shows the request/sec rate (RHS axis) in blue and the CPU max % in yellow (sampled every 5s). The blue vertical lines indicate a drop in request rate (as recorded by the request finishing and therefore being logged) an a corresponding spike to 'catch up'. I note that the spikes always correspond to a spike in CPU.
2. The second shows the spikes appearing to correlate with the increase in committed memory
3. The third is after the JVM setting change. Note the behaviour still occurs in CPU/request rate with a CPU spike in the green circle, but not until the later stages. (NB: CPU scale is CPU% * 200 to fit on the graph)
This behaviour is the same across all the LGs and happens regardless of the way the target hosts are reached across the network, so I believe it's a JVM/host issue.
The original memory settings were
-Xms1G -Xmx12G -XX:NewSize=1024m -XX:MaxNewSize=4096m
But I changed -Xms12G so that all memory is allocated initially and that makes a huge change to the behaviour.
However, I still see the CPU spike. Has anyone got some optimum GC settings they have used that can avoid this?
Thanks
Antony
[cid:image005.jpg@01D3BF87.6FA2B260]
[cid:image006.jpg@01D3BF87.6FA2B260]
[cid:image009.jpg@01D3BF87.6FA2B260]
RE: Feedback and question re Java memory/GC settings for Jmeter load
generator CPU issue
Posted by Antony Bowesman <An...@williamhill.com.au>.
Felix, I ran with G1. Results are very interesting.
Shame I can't post images, but
Using Parallel GC
- Baseline CPU was smooth and low, reaching a normal rate of about 11% at the peak end of the test (~82,000 requests/min with 6,080 VU)
- memory reached the max 12GB after 1 hour of the test, then did GC, causing spikes of CPU to about 60% and a break in requests of about 2.9 seconds
- memory then dropped to just under 8GB
Using G1 GC
- Baseline CPU was very spiky, fluctuating between 15% and 30% at the same point in the test compared to ParallelGC
- memory never reached max, it hit 6.6GB at 20 minutes, before dropping to 4GB and from then on until the test end, slowly saw-toothed up to 7.5GB, never getting close to the max
- no break in request pattern
GC log analyser says this (hope tables are not too stuffed up)
Young Generation, allocated=4 gb, peak=1.01 gb
Old Generation, allocated=8 gb, peak=6.1 gb
Meta Space, allocated=1.02 gb, peak=22.44 mb
Young + Old + Meta space, allocated=13.02 gb, peak=7.13 gb
Avg Pause GC Time = 90 ms
Max Pause GC Time = 190 ms
Duration (secs) No. of GCs Percentage
0 - 0.1 837 78.151%
0.1 - 0.2 234 100.0%
A substantial CPU cost to achieve that but I have plenty of capacity on those boxes. I did not run CMS
Antony
> -----Original Message-----
> From: Felix Schumacher [mailto:felix.schumacher@internetallee.de]
> Sent: Tuesday, 20 March 2018 7:41 PM
> To: JMeter Users List <us...@jmeter.apache.org>
> Subject: RE: Feedback and question re Java memory/GC settings for Jmeter
> load generator CPU issue
>
>
>
> Am 19. März 2018 22:53:19 MEZ schrieb Antony Bowesman
> <An...@williamhill.com.au>:
> >Mmm, I saw the images had gone too :(
> >
> >I have set up to do a gc log next time I run the test and will dig into
> >it. I've been using the default Java8 GC, which is Parallel, so I am
> >going to use CMS to see if that makes a difference. I gather it is
> >supposed to favour shorter pauses, so I'll see what happens and post
> >back results.
>
> As you have 12gb of heap, you could try to use g1, too.
>
> On the other hand side, this seems to be quite a lot of heap. What are you
> doing in your test plan?
>
> And as a nice plus, you could tell us about the used versions for jvm, jmeter
> and os.
>
> Regards,
> Felix
>
>
>
> >
> >Cheers
> >Antony
> >
> >
> >> -----Original Message-----
> >> From: Kirk Pepperdine [mailto:kirk.pepperdine@gmail.com]
> >> Sent: Monday, 19 March 2018 4:39 PM
> >> To: JMeter Users List <us...@jmeter.apache.org>
> >> Subject: Re: Feedback and question re Java memory/GC settings for
> >Jmeter
> >> load generator CPU issue
> >>
> >> Hi,
> >>
> >> The images seem to have been filter out of my email at least.
> >>
> >> Can you collect and post a GC log. Most likely young gen is too small
> >but a gc
> >> log would confirm this.
> >>
> >> Kind regards,
> >> Kirk
> >>
> >> > On Mar 19, 2018, at 3:37 AM, Antony Bowesman
> >> <An...@williamhill.com.au> wrote:
> >> >
> >> > Hi,
> >> >
> >> > I just thought I’d send in some info about a problem I’ve been
> >looking at
> >> recently – with a question of best GC settings
> >> >
> >> > I have a number of JMeter load generators (LG) and I have been
> >seeing
> >> CPU spikes on the boxes during a test. I am monitoring CPU and memory
> >> from within a Java sampler, so have the following charts
> >> >
> >> > 1. First chart shows the request/sec rate (RHS axis) in blue
> >and the CPU
> >> max % in yellow (sampled every 5s). The blue vertical lines indicate
> >a drop in
> >> request rate (as recorded by the request finishing and therefore
> >being
> >> logged) an a corresponding spike to ‘catch up’. I note that the
> >spikes always
> >> correspond to a spike in CPU.
> >> > 2. The second shows the spikes appearing to correlate with
> >the increase
> >> in committed memory
> >> > 3. The third is after the JVM setting change. Note the
> >behaviour still
> >> occurs in CPU/request rate with a CPU spike in the green circle, but
> >not until
> >> the later stages. (NB: CPU scale is CPU% * 200 to fit on the graph)
> >> >
> >> > This behaviour is the same across all the LGs and happens
> >regardless of the
> >> way the target hosts are reached across the network, so I believe
> >it’s a
> >> JVM/host issue.
> >> >
> >> > The original memory settings were
> >> >
> >> > -Xms1G -Xmx12G -XX:NewSize=1024m -XX:MaxNewSize=4096m
> >> >
> >> > But I changed –Xms12G so that all memory is allocated initially and
> >that
> >> makes a huge change to the behaviour.
> >> >
> >> > However, I still see the CPU spike. Has anyone got some optimum GC
> >> settings they have used that can avoid this?
> >> >
> >> > Thanks
> >> > Antony
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >For additional commands, e-mail: user-help@jmeter.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
RE: Feedback and question re Java memory/GC settings for Jmeter load
generator CPU issue
Posted by Antony Bowesman <An...@williamhill.com.au>.
Felix, thanks for the g1 tip, I will try that.
Using JMeter 3.3 on Amazon c4.2xlarge, running Linux 4.9.20-11.31.amzn1.x86_64
JVM
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)
Each host runs 6080 VU and we run up to 15 hosts. All the samplers are Java samplers, about 12, with custom code that manages thousands of variables that tweak how the threads interact with the server and the 200+ endpoints hit. It's testing pure end-to-end, so is simulating the type of traffic seen on busy days.
I've implemented a framework for the samplers that write out metric information for the running threads which goes direct to a Splunk instance, so we get real time reporting, so we just start our instances and load with an ansible playbook and don't have to use the distributed mechanism of JMeter.
It's actually a port of some code that used to run on a tool called eggPlant, so it's not your typical JMeter test plan. I've done quite a bit of work to reduce the memory footprint, as eggplant only supported a 32bit JVM :( hence the port to JMeter.
We run about 75K requests/min from each host
Antony
> -----Original Message-----
> From: Felix Schumacher [mailto:felix.schumacher@internetallee.de]
> Sent: Tuesday, 20 March 2018 7:41 PM
> To: JMeter Users List <us...@jmeter.apache.org>
> Subject: RE: Feedback and question re Java memory/GC settings for Jmeter
> load generator CPU issue
>
>
>
> Am 19. März 2018 22:53:19 MEZ schrieb Antony Bowesman
> <An...@williamhill.com.au>:
> >Mmm, I saw the images had gone too :(
> >
> >I have set up to do a gc log next time I run the test and will dig into
> >it. I've been using the default Java8 GC, which is Parallel, so I am
> >going to use CMS to see if that makes a difference. I gather it is
> >supposed to favour shorter pauses, so I'll see what happens and post
> >back results.
>
> As you have 12gb of heap, you could try to use g1, too.
>
> On the other hand side, this seems to be quite a lot of heap. What are you
> doing in your test plan?
>
> And as a nice plus, you could tell us about the used versions for jvm, jmeter
> and os.
>
> Regards,
> Felix
>
>
>
> >
> >Cheers
> >Antony
> >
> >
> >> -----Original Message-----
> >> From: Kirk Pepperdine [mailto:kirk.pepperdine@gmail.com]
> >> Sent: Monday, 19 March 2018 4:39 PM
> >> To: JMeter Users List <us...@jmeter.apache.org>
> >> Subject: Re: Feedback and question re Java memory/GC settings for
> >Jmeter
> >> load generator CPU issue
> >>
> >> Hi,
> >>
> >> The images seem to have been filter out of my email at least.
> >>
> >> Can you collect and post a GC log. Most likely young gen is too small
> >but a gc
> >> log would confirm this.
> >>
> >> Kind regards,
> >> Kirk
> >>
> >> > On Mar 19, 2018, at 3:37 AM, Antony Bowesman
> >> <An...@williamhill.com.au> wrote:
> >> >
> >> > Hi,
> >> >
> >> > I just thought I’d send in some info about a problem I’ve been
> >looking at
> >> recently – with a question of best GC settings
> >> >
> >> > I have a number of JMeter load generators (LG) and I have been
> >seeing
> >> CPU spikes on the boxes during a test. I am monitoring CPU and memory
> >> from within a Java sampler, so have the following charts
> >> >
> >> > 1. First chart shows the request/sec rate (RHS axis) in blue
> >and the CPU
> >> max % in yellow (sampled every 5s). The blue vertical lines indicate
> >a drop in
> >> request rate (as recorded by the request finishing and therefore
> >being
> >> logged) an a corresponding spike to ‘catch up’. I note that the
> >spikes always
> >> correspond to a spike in CPU.
> >> > 2. The second shows the spikes appearing to correlate with
> >the increase
> >> in committed memory
> >> > 3. The third is after the JVM setting change. Note the
> >behaviour still
> >> occurs in CPU/request rate with a CPU spike in the green circle, but
> >not until
> >> the later stages. (NB: CPU scale is CPU% * 200 to fit on the graph)
> >> >
> >> > This behaviour is the same across all the LGs and happens
> >regardless of the
> >> way the target hosts are reached across the network, so I believe
> >it’s a
> >> JVM/host issue.
> >> >
> >> > The original memory settings were
> >> >
> >> > -Xms1G -Xmx12G -XX:NewSize=1024m -XX:MaxNewSize=4096m
> >> >
> >> > But I changed –Xms12G so that all memory is allocated initially and
> >that
> >> makes a huge change to the behaviour.
> >> >
> >> > However, I still see the CPU spike. Has anyone got some optimum GC
> >> settings they have used that can avoid this?
> >> >
> >> > Thanks
> >> > Antony
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >
> >
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> >For additional commands, e-mail: user-help@jmeter.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
> For additional commands, e-mail: user-help@jmeter.apache.org
RE: Feedback and question re Java memory/GC settings for Jmeter load generator CPU issue
Posted by Felix Schumacher <fe...@internetallee.de>.
Am 19. März 2018 22:53:19 MEZ schrieb Antony Bowesman <An...@williamhill.com.au>:
>Mmm, I saw the images had gone too :(
>
>I have set up to do a gc log next time I run the test and will dig into
>it. I've been using the default Java8 GC, which is Parallel, so I am
>going to use CMS to see if that makes a difference. I gather it is
>supposed to favour shorter pauses, so I'll see what happens and post
>back results.
As you have 12gb of heap, you could try to use g1, too.
On the other hand side, this seems to be quite a lot of heap. What are you doing in your test plan?
And as a nice plus, you could tell us about the used versions for jvm, jmeter and os.
Regards,
Felix
>
>Cheers
>Antony
>
>
>> -----Original Message-----
>> From: Kirk Pepperdine [mailto:kirk.pepperdine@gmail.com]
>> Sent: Monday, 19 March 2018 4:39 PM
>> To: JMeter Users List <us...@jmeter.apache.org>
>> Subject: Re: Feedback and question re Java memory/GC settings for
>Jmeter
>> load generator CPU issue
>>
>> Hi,
>>
>> The images seem to have been filter out of my email at least.
>>
>> Can you collect and post a GC log. Most likely young gen is too small
>but a gc
>> log would confirm this.
>>
>> Kind regards,
>> Kirk
>>
>> > On Mar 19, 2018, at 3:37 AM, Antony Bowesman
>> <An...@williamhill.com.au> wrote:
>> >
>> > Hi,
>> >
>> > I just thought I’d send in some info about a problem I’ve been
>looking at
>> recently – with a question of best GC settings
>> >
>> > I have a number of JMeter load generators (LG) and I have been
>seeing
>> CPU spikes on the boxes during a test. I am monitoring CPU and memory
>> from within a Java sampler, so have the following charts
>> >
>> > 1. First chart shows the request/sec rate (RHS axis) in blue
>and the CPU
>> max % in yellow (sampled every 5s). The blue vertical lines indicate
>a drop in
>> request rate (as recorded by the request finishing and therefore
>being
>> logged) an a corresponding spike to ‘catch up’. I note that the
>spikes always
>> correspond to a spike in CPU.
>> > 2. The second shows the spikes appearing to correlate with
>the increase
>> in committed memory
>> > 3. The third is after the JVM setting change. Note the
>behaviour still
>> occurs in CPU/request rate with a CPU spike in the green circle, but
>not until
>> the later stages. (NB: CPU scale is CPU% * 200 to fit on the graph)
>> >
>> > This behaviour is the same across all the LGs and happens
>regardless of the
>> way the target hosts are reached across the network, so I believe
>it’s a
>> JVM/host issue.
>> >
>> > The original memory settings were
>> >
>> > -Xms1G -Xmx12G -XX:NewSize=1024m -XX:MaxNewSize=4096m
>> >
>> > But I changed –Xms12G so that all memory is allocated initially and
>that
>> makes a huge change to the behaviour.
>> >
>> > However, I still see the CPU spike. Has anyone got some optimum GC
>> settings they have used that can avoid this?
>> >
>> > Thanks
>> > Antony
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
>For additional commands, e-mail: user-help@jmeter.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
RE: Feedback and question re Java memory/GC settings for Jmeter load
generator CPU issue
Posted by Antony Bowesman <An...@williamhill.com.au>.
Mmm, I saw the images had gone too :(
I have set up to do a gc log next time I run the test and will dig into it. I've been using the default Java8 GC, which is Parallel, so I am going to use CMS to see if that makes a difference. I gather it is supposed to favour shorter pauses, so I'll see what happens and post back results.
Cheers
Antony
> -----Original Message-----
> From: Kirk Pepperdine [mailto:kirk.pepperdine@gmail.com]
> Sent: Monday, 19 March 2018 4:39 PM
> To: JMeter Users List <us...@jmeter.apache.org>
> Subject: Re: Feedback and question re Java memory/GC settings for Jmeter
> load generator CPU issue
>
> Hi,
>
> The images seem to have been filter out of my email at least.
>
> Can you collect and post a GC log. Most likely young gen is too small but a gc
> log would confirm this.
>
> Kind regards,
> Kirk
>
> > On Mar 19, 2018, at 3:37 AM, Antony Bowesman
> <An...@williamhill.com.au> wrote:
> >
> > Hi,
> >
> > I just thought I’d send in some info about a problem I’ve been looking at
> recently – with a question of best GC settings
> >
> > I have a number of JMeter load generators (LG) and I have been seeing
> CPU spikes on the boxes during a test. I am monitoring CPU and memory
> from within a Java sampler, so have the following charts
> >
> > 1. First chart shows the request/sec rate (RHS axis) in blue and the CPU
> max % in yellow (sampled every 5s). The blue vertical lines indicate a drop in
> request rate (as recorded by the request finishing and therefore being
> logged) an a corresponding spike to ‘catch up’. I note that the spikes always
> correspond to a spike in CPU.
> > 2. The second shows the spikes appearing to correlate with the increase
> in committed memory
> > 3. The third is after the JVM setting change. Note the behaviour still
> occurs in CPU/request rate with a CPU spike in the green circle, but not until
> the later stages. (NB: CPU scale is CPU% * 200 to fit on the graph)
> >
> > This behaviour is the same across all the LGs and happens regardless of the
> way the target hosts are reached across the network, so I believe it’s a
> JVM/host issue.
> >
> > The original memory settings were
> >
> > -Xms1G -Xmx12G -XX:NewSize=1024m -XX:MaxNewSize=4096m
> >
> > But I changed –Xms12G so that all memory is allocated initially and that
> makes a huge change to the behaviour.
> >
> > However, I still see the CPU spike. Has anyone got some optimum GC
> settings they have used that can avoid this?
> >
> > Thanks
> > Antony
> >
> >
> >
> >
> >
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: Feedback and question re Java memory/GC settings for Jmeter load
generator CPU issue
Posted by Kirk Pepperdine <ki...@gmail.com>.
Hi,
The images seem to have been filter out of my email at least.
Can you collect and post a GC log. Most likely young gen is too small but a gc log would confirm this.
Kind regards,
Kirk
> On Mar 19, 2018, at 3:37 AM, Antony Bowesman <An...@williamhill.com.au> wrote:
>
> Hi,
>
> I just thought I’d send in some info about a problem I’ve been looking at recently – with a question of best GC settings
>
> I have a number of JMeter load generators (LG) and I have been seeing CPU spikes on the boxes during a test. I am monitoring CPU and memory from within a Java sampler, so have the following charts
>
> 1. First chart shows the request/sec rate (RHS axis) in blue and the CPU max % in yellow (sampled every 5s). The blue vertical lines indicate a drop in request rate (as recorded by the request finishing and therefore being logged) an a corresponding spike to ‘catch up’. I note that the spikes always correspond to a spike in CPU.
> 2. The second shows the spikes appearing to correlate with the increase in committed memory
> 3. The third is after the JVM setting change. Note the behaviour still occurs in CPU/request rate with a CPU spike in the green circle, but not until the later stages. (NB: CPU scale is CPU% * 200 to fit on the graph)
>
> This behaviour is the same across all the LGs and happens regardless of the way the target hosts are reached across the network, so I believe it’s a JVM/host issue.
>
> The original memory settings were
>
> -Xms1G -Xmx12G -XX:NewSize=1024m -XX:MaxNewSize=4096m
>
> But I changed –Xms12G so that all memory is allocated initially and that makes a huge change to the behaviour.
>
> However, I still see the CPU spike. Has anyone got some optimum GC settings they have used that can avoid this?
>
> Thanks
> Antony
>
>
>
>
>
>
>