You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "George I. Develekos" <gd...@omilia.com> on 2016/11/15 15:46:36 UTC

Please help with Tomcat Garbage Collection

Hello guys,

We are having problems on a production system with very long "full GC" 
times, as long as1200sec real time (!!!).

We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat 7.0.64.

Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC   
-XX:+CMSIncrementalMode

No other custom memory-related settings are in place.

Looking at the GC log, the last few Full-GC entries are:

1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K), 
2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm : 
169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times: user=1.96 
sys=0.13, real=2.11 secs]

2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark: 
2.558/1212.733 secs] [*Times: user=113.05 sys=28.01, real=**1212.49 
**secs] **
*
3539.969: [Full GC 3539.969: [CMS3540.056: [CMS-concurrent-sweep: 
1.571/23.223 secs] [Times: user=6.12 sys=1.36, real=*23.21 secs*]

4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K), 
2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm : 
169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times: user=2.18 
sys=0.14, real=2.34 secs]


What can we do?

Thanks,

George





---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Please help with Tomcat Garbage Collection

Posted by "George I. Develekos" <gd...@omilia.com>.
One thing I didn't respond to:

The VM host is unlikely to have something to do here. We have a 
coincidence (in time) of three events, consistently.

1. Full GC
2. High CPU
3. App freezes

So it seems really unlikely the host is responsible.

The odd thing is that, the same app, on the same OS, in other 
installations, where of course we also have Full-GC periodically, never 
shows any freeze.

Regards,

George I. Develekos | SeniorSW Engineer |t: _+30.210.6930664_| e: 
gdevelekos@omilia.com <ma...@omilia.com>

------------------------------------------------------------------------

Technology that Listens, Understands and Cares   - www.omilia.com 
<http://www.omilia.com/>

On 11/16/2016 6:11 PM, John.E.Gregg@wellsfargo.com wrote:
>
> Sorry for top posting.  The format got weird.
>
> Those numbers aren\u2019t bad.  The most important number for me is the 
> throughput on the summary tab.  Yours is 99.38%.  That means the JVM 
> was doing real work 99.38% of the time and garbage collecting the 
> other .62%.  You could improve that if you worked hard enough, but 
> going from 99.4 to 99.5 or 99.6 probably isn\u2019t worth the effort.  If 
> the number was 90% or something, then you\u2019d have more room for 
> improvement.
>
> Look at the GC performance numbers on the summary tab.  Obviously 
> minor GCs are much faster in this regard than major GCs. You can 
> reduce your total GC time by increasing the size of your young 
> generation.  You will get more or slower young collections but fewer 
> and faster old collections.  Overall the total time will be less than 
> it is now and the longest pauses will be shorter.
>
> As others have said, though, something doesn\u2019t add up.  CMS is only 
> stop-the-world during certain phases.  (Not the ones with \u201cconcurrent\u201d 
> in the name.)  If you feel these GC events coincide with pauses in 
> your app, you can try a thread dump or three (kill -3 <pid>) during 
> the pause.  Use a tool like Samurai to parse the output.  This might 
> only be practical for longer pauses, though.
>
> Also, is it possible the VM itself is having a problem?  Maybe you 
> should talk to your virtualization team to see how stressed the 
> hardware is.  VMWare has an informative java best practices doc: 
> http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/enterprise-java-applications-on-vmware-best-practices-guide.pdf
>
> John
>
> *From:*George I. Develekos [mailto:gdevelekos@omilia.com]
> *Sent:* Wednesday, November 16, 2016 5:21 AM
> *To:* users@tomcat.apache.org
> *Subject:* Re: Please help with Tomcat Garbage Collection
>
> I'm attaching three screenshots of the GCViewer app as it processed 
> the complete *gc.log* file (about 19 hours).
>
> Please have a look and advise on what I can do to limit Full-GC times. 
> As of now I have a recommendation to increase the Young Gen..
>
> The setup in summary:
>
> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat 
> 7.0.64.
>
> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC   
> -XX:+CMSIncrementalMode
>
>
>
>
>
>
> On 11/15/2016 11:45 PM, John.E.Gregg@wellsfargo.com 
> <ma...@wellsfargo.com> wrote:
>
>         -----Original Message-----
>
>         From: George I. Develekos [mailto:gdevelekos@omilia.com]
>
>         Sent: Tuesday, November 15, 2016 3:00 PM
>
>         To:users@tomcat.apache.org <ma...@tomcat.apache.org>
>
>         Subject: Re: Please help with Tomcat Garbage Collection
>
>         The system does very little swapping, both when it's GC'ing and when it's not.
>
>         Less than 100MB worth of swap is taken.
>
>         Giving Tomcat its own HW is not an option at this time, especially as there's no
>
>         guarantee it'll solve the problem. Besides it would be a VM anyway, not physical
>
>         dedicated HW.  The current server is also a VM.
>
>         On 15-Nov-16 10:55 PM, Zden\u011bk Henek wrote:
>
>             I would start with moving this tomcat to its own hw.
>
>             Did you check swap? This long pauses could be because part of your
>
>             heap is swapped to hdd
>
>             Regards,
>
>             Zdenek Henek
>
>             On Tue, Nov 15, 2016, 21:37 George I. Develekos
>
>             <gd...@omilia.com> <ma...@omilia.com>
>
>             wrote:
>
>                 On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
>
>                     George,
>
>                     On 11/15/16 10:46 AM, George I. Develekos wrote:
>
>                         Hello guys,
>
>                         We are having problems on a production system with very long "full
>
>                         GC" times, as long as1200sec real time (!!!).
>
>                         We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
>
>                         7.0.64.
>
>                         Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
>
>                         -XX:+CMSIncrementalMode
>
>                         No other custom memory-related settings are in place.
>
>                         Looking at the GC log, the last few Full-GC entries are:
>
>                         1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
>
>                         2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
>
>                         169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
>
>                         user=1.96 sys=0.13, real=2.11 secs]
>
>                         2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
>
>                         2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
>
>                         real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
>
>                         [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
>
>                         user=6.12 sys=1.36, real=*23.21 secs*]
>
>                         4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
>
>                         2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
>
>                         169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
>
>                         user=2.18 sys=0.14, real=2.34 secs]
>
>                         What can we do?
>
>                     1367.020 Full GC duration=2.11s
>
>                     2579.317 Full GC duration=1212.49s
>
>                     So your full GC immediately started another full GC that took 20
>
>                     minutes ?
>
>                     Are you only showing certain FULL GC activity from your log, or is
>
>                     that everything?
>
>                     CMS should have a mark and then a sweep each time, but your times
>
>                     don't seem to add up.
>
>                     also note that the whole point of CMS is that there isn't any
>
>                     stop-the-world during the mark portion of the process.
>
>                     Are you actually experiencing a problem, or are you just suffering
>
>                     from instrumentor's remorse?
>
>                     - -chris
>
>                 Chris,
>
>                 What I listed is the result of the command:
>
>                 grep "Full GC" gc.log
>
>                 So (obviously) I have skipped other GC activity, i.e. whatever GC
>
>                 activity didn't include the "Full GC" string.
>
>                 Yes we are having app trouble due to the GC delays so this is a real
>
>                 problem. Our application has real-time constraints so the GC delays
>
>                 cannot be tolerated. I selected those GC options _in order to avoid
>
>                 _long GC times.
>
>                 Additionally, these periods coincide with high CPU for that JVM
>
>                 process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
>
>                 Once GC is done, our app rushes to catch up with tasks that had to
>
>                 wait for GC to finish.
>
>                 Answering another question from a member who has kindly responded,
>
>                 yes the server is running other stuff. Basically it runs three
>
>                 tomcats, the main one being this one. It also runs a DB2 database
>
>                 that has close-to-zero activity.
>
>                 George
>
>     It might be helpful if you could post a larger chunk of your GC log, at least long enough to cover the start and end of the CMS phases and maybe even more.  Additionally, try using a tool like GCViewer to analyze the log.
>
>     How many CPUs do you have?  60% CPU usage isn't usually a big deal.
>
>     Like Chris already said, this is not a stop-the-world phase, so your application should continue working in parallel with the garbage collector.
>
>     Looks like your young generation is only 150MB (2080128k - 1926784k.)  That's very small for a 2-5GB heap.  Are you explicitly setting it somewhere or is the JVM choosing that for you?  It's so small that your old generation might be filling up faster that it should, leading to more frequent full collections.  You could try setting the young generation to something like 25-50% of the total heap.  You'd get a lot of small pauses as the young gen is collected but fewer long ones.
>
>     John
>
>     ---------------------------------------------------------------------
>
>     To unsubscribe, e-mail:users-unsubscribe@tomcat.apache.org
>     <ma...@tomcat.apache.org>
>
>     For additional commands, e-mail:users-help@tomcat.apache.org <ma...@tomcat.apache.org>
>
>
>
> ------------------------------------------------------------------------
>
> Avast logo 
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> 	
>
> This email has been checked for viruses by Avast antivirus software.
> www.avast.com 
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> 
>
>



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Please help with Tomcat Garbage Collection

Posted by "George I. Develekos" <gd...@omilia.com>.
The data I posted was collected as Tomcat was restarted and for the next 
19 hours.

You see too much GC thrashing, are you saying the GC process finds too 
much G to clean up? Should I urge the developers to do some GC of their own?

I give 2G to start with because it may not actually /need /5g, and 
experience has shown that it doesn't. I know giving it all at the start 
is better but I don't have memory to spare. I'll probably up from 2/5 yo 
3/5.

To boost YoungGen, should I do it with an explicit setting or with the 
radio thingy? Set it to 3, perhaps?

On 16-Nov-16 6:59 PM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> George,
>
> On 11/16/16 11:18 AM, George I. Develekos wrote:
>> On another installation with higher  load, the JVM has "selected"
>> to give  YoungGen 250MB or so (as opposed to 150M here), and I
>> have confirmed that Full-GC is much less frequent so I'll go with
>> that next time I have downtime.
> +1
>
> - From the screenshots you posted, it looks like everything is healthy
> EXCEPT that your young generation said 100% used *and* the "average
> freed full GC" says nearly 700M per collection.
>
> I've never used GCViewer, but if I'm reading that correctly, it's
> saying that you are averaging more than 50% turnover in your tenured
> generation during each full gc. (You have ~1300M in your tenured
> generation and 700M is being collected each time.) That's a HUGE
> amount of GC thrashing.
>
> Were these samples taken of a stably-running production system, or was
> this immediately after a restart, or in a lab or something?
>
>> When exactly can I expect to see my app freeze? Isn't it during
>> Full-GC?
> John has already answered this. Feel free to ask follow-up questions.
>
>> Does the increase in CPU during those Full-GC times make sense?
> Absolutely. Most of the time, applications are waiting for other
> things to happen: read/write from the network, disk, whatever. GC is
> 100% CPU-bound activity, so you'll see a spike in the CPU usage
> whenever the GC is running. Since some of the phases of CMS are
> concurrent (that's what the "C" stands for), they won't pause your
> application. Unfortunately, some phases are NOT concurrent, and the
> application needs to be paused (this is appropriately called
> "stop-the-world" because that's exactly what happens). Fortunately,
> those phases are designed to be very short in time.
>
> I have two recommendations for you, which really boils down to a
> single one:
>
> 1. Set Xms and Xmx to be the same value
>
> If you want to give 5G to your application, then give it all 5G. Don't
> wait for the application to lazily grow to fill the space. It just
> slows everything down. If you want a 2G heap, then use a 2G heap.
> Splitting the heap into initial and max sizes is silly for a server
> process.
>
> 2. Increase the young generation size by *at least* a factor of 2,
> possibly 4. If you do #1 above, the JVM will likely re-size the young
> generation for you anyway, since it will be starting with a bigger
> heap. So you might not have to do this one at all.
>
> Finally, the best course of action would be to upgrade to a more
> recent JVM. You said initially that you were stuck, but you might not
> be. You can use Oracle's JVM or OpenJVM running fairly easily, though
> you might have to maintain and/or update it manually if there is no
> package from RedHat for it.
>
> I've never used package-managed JVMs because they tend to require
> dozens or hundreds of other packages to be installed. Installation is
> typically untarring a tarball, upgrading is just as easy, and
> downgrading is a snap (just don't delete the old directory when
> upgrading). The only downside is that the package manager won't go
> fetch the archive and run the installer for you.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJYLJCEAAoJEBzwKT+lPKRYyrkP/RuvH0SKRcEGfDGWybq8LQl1
> MXy6fPk1HtHrkRAgEtQhxmEh/Za64E993R7hXnKNiGjwz2FVp2Fz4rq81Y/cxuSw
> PEO01mfwY7y+j97zO2PQvYubnEhHIiPZZ5IYnjVMqZtmdX6hETCiWAx6vFYcCnL/
> 9ZGmHXDuCcI1uxDNJ4rq/mJcFyMjRLMR1uufrGwv37nbwXAKCQPIiYb6lALyof2X
> 7HwZUNAYi+MAJnQJSub4/QM/Bm4IUJn4h47RKia2ar2PaEIOSZihqLP7JRjTlTA7
> U6KvDUfiBKNh7ECOvAXxFhZxjK3NEpjBU3/6beRIVcL5K5klOGfXTofDACEACH47
> 5bfmLrQChR0jPbsBlQcLwRQQxVdsj6RJi0u96Anyn5ElLfZiKUexQDx6X0oXeV6i
> n5o4En0Rme7GzVtcEh3FmY2y4vFqCAGgwZ3lK2caXH6AqpR0WsyRuLISK4UtjcxP
> NJIXpURtsfNMNqHTH/US6VftfC2lIQNJB0mRzMdnl7KrA6U6mXmn21ZAiPT64lkw
> O8JQ49fF3giszo+x37EVkaA+YFn86ap4UzxwH+Tz86xd8qckihFTvGLr48cCjlJn
> zhyR8x5ir+Q1CW9Q3GhSZJTtC5j2LY94vF7I5pZdln34aM/37u/iXlesBaEzwqGW
> Sx00lN+4sNzr9jPZez5C
> =Q0qd
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Please help with Tomcat Garbage Collection

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

George,

On 11/16/16 11:18 AM, George I. Develekos wrote:
> On another installation with higher  load, the JVM has "selected"
> to give  YoungGen 250MB or so (as opposed to 150M here), and I
> have confirmed that Full-GC is much less frequent so I'll go with
> that next time I have downtime.

+1

- From the screenshots you posted, it looks like everything is healthy
EXCEPT that your young generation said 100% used *and* the "average
freed full GC" says nearly 700M per collection.

I've never used GCViewer, but if I'm reading that correctly, it's
saying that you are averaging more than 50% turnover in your tenured
generation during each full gc. (You have ~1300M in your tenured
generation and 700M is being collected each time.) That's a HUGE
amount of GC thrashing.

Were these samples taken of a stably-running production system, or was
this immediately after a restart, or in a lab or something?

> When exactly can I expect to see my app freeze? Isn't it during
> Full-GC?

John has already answered this. Feel free to ask follow-up questions.

> Does the increase in CPU during those Full-GC times make sense?

Absolutely. Most of the time, applications are waiting for other
things to happen: read/write from the network, disk, whatever. GC is
100% CPU-bound activity, so you'll see a spike in the CPU usage
whenever the GC is running. Since some of the phases of CMS are
concurrent (that's what the "C" stands for), they won't pause your
application. Unfortunately, some phases are NOT concurrent, and the
application needs to be paused (this is appropriately called
"stop-the-world" because that's exactly what happens). Fortunately,
those phases are designed to be very short in time.

I have two recommendations for you, which really boils down to a
single one:

1. Set Xms and Xmx to be the same value

If you want to give 5G to your application, then give it all 5G. Don't
wait for the application to lazily grow to fill the space. It just
slows everything down. If you want a 2G heap, then use a 2G heap.
Splitting the heap into initial and max sizes is silly for a server
process.

2. Increase the young generation size by *at least* a factor of 2,
possibly 4. If you do #1 above, the JVM will likely re-size the young
generation for you anyway, since it will be starting with a bigger
heap. So you might not have to do this one at all.

Finally, the best course of action would be to upgrade to a more
recent JVM. You said initially that you were stuck, but you might not
be. You can use Oracle's JVM or OpenJVM running fairly easily, though
you might have to maintain and/or update it manually if there is no
package from RedHat for it.

I've never used package-managed JVMs because they tend to require
dozens or hundreds of other packages to be installed. Installation is
typically untarring a tarball, upgrading is just as easy, and
downgrading is a snap (just don't delete the old directory when
upgrading). The only downside is that the package manager won't go
fetch the archive and run the installer for you.

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYLJCEAAoJEBzwKT+lPKRYyrkP/RuvH0SKRcEGfDGWybq8LQl1
MXy6fPk1HtHrkRAgEtQhxmEh/Za64E993R7hXnKNiGjwz2FVp2Fz4rq81Y/cxuSw
PEO01mfwY7y+j97zO2PQvYubnEhHIiPZZ5IYnjVMqZtmdX6hETCiWAx6vFYcCnL/
9ZGmHXDuCcI1uxDNJ4rq/mJcFyMjRLMR1uufrGwv37nbwXAKCQPIiYb6lALyof2X
7HwZUNAYi+MAJnQJSub4/QM/Bm4IUJn4h47RKia2ar2PaEIOSZihqLP7JRjTlTA7
U6KvDUfiBKNh7ECOvAXxFhZxjK3NEpjBU3/6beRIVcL5K5klOGfXTofDACEACH47
5bfmLrQChR0jPbsBlQcLwRQQxVdsj6RJi0u96Anyn5ElLfZiKUexQDx6X0oXeV6i
n5o4En0Rme7GzVtcEh3FmY2y4vFqCAGgwZ3lK2caXH6AqpR0WsyRuLISK4UtjcxP
NJIXpURtsfNMNqHTH/US6VftfC2lIQNJB0mRzMdnl7KrA6U6mXmn21ZAiPT64lkw
O8JQ49fF3giszo+x37EVkaA+YFn86ap4UzxwH+Tz86xd8qckihFTvGLr48cCjlJn
zhyR8x5ir+Q1CW9Q3GhSZJTtC5j2LY94vF7I5pZdln34aM/37u/iXlesBaEzwqGW
Sx00lN+4sNzr9jPZez5C
=Q0qd
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


RE: Please help with Tomcat Garbage Collection

Posted by Jo...@wellsfargo.com.


> -----Original Message-----
> From: George I. Develekos [mailto:gdevelekos@omilia.com]
> Sent: Wednesday, November 16, 2016 10:18 AM
> To: users@tomcat.apache.org
> Subject: Re: Please help with Tomcat Garbage Collection
> 
> I appreciate the detailed response.
> 
> On another installation with higher  load, the JVM has "selected" to give
> YoungGen 250MB or so (as opposed to 150M here), and I have confirmed that
> Full-GC is much less frequent so I'll go with that next time I have downtime.
> 
> When exactly can I expect to see my app freeze? Isn't it during Full-GC?
> 
> Does the increase in CPU during those Full-GC times make sense?
> 
> 
> 
> Regards,
> 
> George I. Develekos | SeniorSW Engineer |t: _+30.210.6930664_| e:
> gdevelekos@omilia.com <ma...@omilia.com>
> 
> ------------------------------------------------------------------------
> 
> Technology that Listens, Understands and Cares   - www.omilia.com
> <http://www.omilia.com/>
> 
> On 11/16/2016 6:11 PM, John.E.Gregg@wellsfargo.com wrote:
> >
> > Sorry for top posting.  The format got weird.
> >
> > Those numbers aren’t bad.  The most important number for me is the
> > throughput on the summary tab.  Yours is 99.38%.  That means the JVM
> > was doing real work 99.38% of the time and garbage collecting the
> > other .62%.  You could improve that if you worked hard enough, but
> > going from 99.4 to 99.5 or 99.6 probably isn’t worth the effort.  If
> > the number was 90% or something, then you’d have more room for
> > improvement.
> >
> > Look at the GC performance numbers on the summary tab.  Obviously
> > minor GCs are much faster in this regard than major GCs. You can
> > reduce your total GC time by increasing the size of your young
> > generation.  You will get more or slower young collections but fewer
> > and faster old collections.  Overall the total time will be less than
> > it is now and the longest pauses will be shorter.
> >
> > As others have said, though, something doesn’t add up.  CMS is only
> > stop-the-world during certain phases.  (Not the ones with “concurrent”
> > in the name.)  If you feel these GC events coincide with pauses in
> > your app, you can try a thread dump or three (kill -3 <pid>) during
> > the pause.  Use a tool like Samurai to parse the output.  This might
> > only be practical for longer pauses, though.
> >
> > Also, is it possible the VM itself is having a problem?  Maybe you
> > should talk to your virtualization team to see how stressed the
> > hardware is.  VMWare has an informative java best practices doc:
> >
> http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techp
> > aper/enterprise-java-applications-on-vmware-best-practices-guide.pdf
> >
> > John
> >
> > *From:*George I. Develekos [mailto:gdevelekos@omilia.com]
> > *Sent:* Wednesday, November 16, 2016 5:21 AM
> > *To:* users@tomcat.apache.org
> > *Subject:* Re: Please help with Tomcat Garbage Collection
> >
> > I'm attaching three screenshots of the GCViewer app as it processed
> > the complete *gc.log* file (about 19 hours).
> >
> > Please have a look and advise on what I can do to limit Full-GC times.
> > As of now I have a recommendation to increase the Young Gen..
> >
> > The setup in summary:
> >
> > We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
> > 7.0.64.
> >
> > Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
> > -XX:+CMSIncrementalMode
> >
> >
> >
> >
> >
> >
> > On 11/15/2016 11:45 PM, John.E.Gregg@wellsfargo.com
> > <ma...@wellsfargo.com> wrote:
> >
> >         -----Original Message-----
> >
> >         From: George I. Develekos [mailto:gdevelekos@omilia.com]
> >
> >         Sent: Tuesday, November 15, 2016 3:00 PM
> >
> >         To:users@tomcat.apache.org <ma...@tomcat.apache.org>
> >
> >         Subject: Re: Please help with Tomcat Garbage Collection
> >
> >         The system does very little swapping, both when it's GC'ing and when it's
> not.
> >
> >         Less than 100MB worth of swap is taken.
> >
> >         Giving Tomcat its own HW is not an option at this time,
> > especially as there's no
> >
> >         guarantee it'll solve the problem. Besides it would be a VM
> > anyway, not physical
> >
> >         dedicated HW.  The current server is also a VM.
> >
> >         On 15-Nov-16 10:55 PM, Zdeněk Henek wrote:
> >
> >             I would start with moving this tomcat to its own hw.
> >
> >             Did you check swap? This long pauses could be because part
> > of your
> >
> >             heap is swapped to hdd
> >
> >             Regards,
> >
> >             Zdenek Henek
> >
> >             On Tue, Nov 15, 2016, 21:37 George I. Develekos
> >
> >             <gd...@omilia.com> <ma...@omilia.com>
> >
> >             wrote:
> >
> >                 On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
> >
> >                     George,
> >
> >                     On 11/15/16 10:46 AM, George I. Develekos wrote:
> >
> >                         Hello guys,
> >
> >                         We are having problems on a production system
> > with very long "full
> >
> >                         GC" times, as long as1200sec real time (!!!).
> >
> >                         We are using Java 6 (stuck with CentOS 5.8 at
> > this time) and Tomcat
> >
> >                         7.0.64.
> >
> >                         Xmx is 5G, Xms is 2G, and GC options are
> > -XX:+UseConcMarkSweepGC
> >
> >                         -XX:+CMSIncrementalMode
> >
> >                         No other custom memory-related settings are in place.
> >
> >                         Looking at the GC log, the last few Full-GC entries are:
> >
> >                         1367.020: [Full GC 1367.020: [CMS:
> > 1178831K->527456K(1926784K),
> >
> >                         2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
> >
> >                         169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs]
> [Times:
> >
> >                         user=1.96 sys=0.13, real=2.11 secs]
> >
> >                         2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-
> mark:
> >
> >                         2.558/1212.733 secs] [*Times: user=113.05
> > sys=28.01,
> >
> >                         real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
> >
> >                         [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs]
> [Times:
> >
> >                         user=6.12 sys=1.36, real=*23.21 secs*]
> >
> >                         4070.456: [Full GC 4070.457: [CMS:
> > 1252569K->591200K(1926784K),
> >
> >                         2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
> >
> >                         169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs]
> [Times:
> >
> >                         user=2.18 sys=0.14, real=2.34 secs]
> >
> >                         What can we do?
> >
> >                     1367.020 Full GC duration=2.11s
> >
> >                     2579.317 Full GC duration=1212.49s
> >
> >                     So your full GC immediately started another full
> > GC that took 20
> >
> >                     minutes ?
> >
> >                     Are you only showing certain FULL GC activity from
> > your log, or is
> >
> >                     that everything?
> >
> >                     CMS should have a mark and then a sweep each time,
> > but your times
> >
> >                     don't seem to add up.
> >
> >                     also note that the whole point of CMS is that
> > there isn't any
> >
> >                     stop-the-world during the mark portion of the process.
> >
> >                     Are you actually experiencing a problem, or are
> > you just suffering
> >
> >                     from instrumentor's remorse?
> >
> >                     - -chris
> >
> >                 Chris,
> >
> >                 What I listed is the result of the command:
> >
> >                 grep "Full GC" gc.log
> >
> >                 So (obviously) I have skipped other GC activity, i.e.
> > whatever GC
> >
> >                 activity didn't include the "Full GC" string.
> >
> >                 Yes we are having app trouble due to the GC delays so
> > this is a real
> >
> >                 problem. Our application has real-time constraints so
> > the GC delays
> >
> >                 cannot be tolerated. I selected those GC options _in
> > order to avoid
> >
> >                 _long GC times.
> >
> >                 Additionally, these periods coincide with high CPU for
> > that JVM
> >
> >                 process.  From 5-20% CPU where it is normally, it jumps to 60% ore
> more.
> >
> >                 Once GC is done, our app rushes to catch up with tasks
> > that had to
> >
> >                 wait for GC to finish.
> >
> >                 Answering another question from a member who has
> > kindly responded,
> >
> >                 yes the server is running other stuff. Basically it
> > runs three
> >
> >                 tomcats, the main one being this one. It also runs a
> > DB2 database
> >
> >                 that has close-to-zero activity.
> >
> >                 George
> >
> >     It might be helpful if you could post a larger chunk of your GC log, at least
> long enough to cover the start and end of the CMS phases and maybe even
> more.  Additionally, try using a tool like GCViewer to analyze the log.
> >
> >     How many CPUs do you have?  60% CPU usage isn't usually a big deal.
> >
> >     Like Chris already said, this is not a stop-the-world phase, so your
> application should continue working in parallel with the garbage collector.
> >
> >     Looks like your young generation is only 150MB (2080128k - 1926784k.)
> That's very small for a 2-5GB heap.  Are you explicitly setting it somewhere or is
> the JVM choosing that for you?  It's so small that your old generation might be
> filling up faster that it should, leading to more frequent full collections.  You
> could try setting the young generation to something like 25-50% of the total
> heap.  You'd get a lot of small pauses as the young gen is collected but fewer
> long ones.
> >
> >     John
> >
> >
> > ---------------------------------------------------------------------
> >
> >     To unsubscribe, e-mail:users-unsubscribe@tomcat.apache.org
> >     <ma...@tomcat.apache.org>
> >
> >     For additional commands, e-mail:users-help@tomcat.apache.org
> > <ma...@tomcat.apache.org>
> >
> >
> >
> > ----------------------------------------------------------------------
> > --
> >
> > Avast logo
> > <https://www.avast.com/sig-
> email?utm_medium=email&utm_source=link&utm_
> > campaign=sig-email&utm_content=emailclient>
> >
> >
> >
> > This email has been checked for viruses by Avast antivirus software.
> > www.avast.com
> > <https://www.avast.com/sig-
> email?utm_medium=email&utm_source=link&utm_
> > campaign=sig-email&utm_content=emailclient>
> >
> >
> 
> 
> 
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus

CMS consists of phases.  It's not just one monolithic activity.  The stop-the-world pauses are the initial-mark and remark phases.  These are identified in the log as CMS-initial-mark and CMS-remark.  (All young GCs are stop-the-world, too, by the way.)  Otherwise your app should continue to run concurrently with GC.  That's what's nice about CMS: technically the collections take longer than with the parallel collector, but the app continues to work normally during most of that time.

There probably is an increase in CPU during that time, but the collections are so short you might not notice.  You might notice an increase in response time, but it could be hard to separate the impact of GC from the rest of the noise in the system (response time variability in backends, etc.)

John


Re: Please help with Tomcat Garbage Collection

Posted by "George I. Develekos" <gd...@omilia.com>.
I appreciate the detailed response.

On another installation with higher  load, the JVM has "selected" to 
give  YoungGen 250MB or so (as opposed to 150M here), and I have 
confirmed that Full-GC is much less frequent so I'll go with that next 
time I have downtime.

When exactly can I expect to see my app freeze? Isn't it during Full-GC?

Does the increase in CPU during those Full-GC times make sense?



Regards,

George I. Develekos | SeniorSW Engineer |t: _+30.210.6930664_| e: 
gdevelekos@omilia.com <ma...@omilia.com>

------------------------------------------------------------------------

Technology that Listens, Understands and Cares   - www.omilia.com 
<http://www.omilia.com/>

On 11/16/2016 6:11 PM, John.E.Gregg@wellsfargo.com wrote:
>
> Sorry for top posting.  The format got weird.
>
> Those numbers aren\u2019t bad.  The most important number for me is the 
> throughput on the summary tab.  Yours is 99.38%.  That means the JVM 
> was doing real work 99.38% of the time and garbage collecting the 
> other .62%.  You could improve that if you worked hard enough, but 
> going from 99.4 to 99.5 or 99.6 probably isn\u2019t worth the effort.  If 
> the number was 90% or something, then you\u2019d have more room for 
> improvement.
>
> Look at the GC performance numbers on the summary tab.  Obviously 
> minor GCs are much faster in this regard than major GCs. You can 
> reduce your total GC time by increasing the size of your young 
> generation.  You will get more or slower young collections but fewer 
> and faster old collections.  Overall the total time will be less than 
> it is now and the longest pauses will be shorter.
>
> As others have said, though, something doesn\u2019t add up.  CMS is only 
> stop-the-world during certain phases.  (Not the ones with \u201cconcurrent\u201d 
> in the name.)  If you feel these GC events coincide with pauses in 
> your app, you can try a thread dump or three (kill -3 <pid>) during 
> the pause.  Use a tool like Samurai to parse the output.  This might 
> only be practical for longer pauses, though.
>
> Also, is it possible the VM itself is having a problem?  Maybe you 
> should talk to your virtualization team to see how stressed the 
> hardware is.  VMWare has an informative java best practices doc: 
> http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/enterprise-java-applications-on-vmware-best-practices-guide.pdf
>
> John
>
> *From:*George I. Develekos [mailto:gdevelekos@omilia.com]
> *Sent:* Wednesday, November 16, 2016 5:21 AM
> *To:* users@tomcat.apache.org
> *Subject:* Re: Please help with Tomcat Garbage Collection
>
> I'm attaching three screenshots of the GCViewer app as it processed 
> the complete *gc.log* file (about 19 hours).
>
> Please have a look and advise on what I can do to limit Full-GC times. 
> As of now I have a recommendation to increase the Young Gen..
>
> The setup in summary:
>
> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat 
> 7.0.64.
>
> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC   
> -XX:+CMSIncrementalMode
>
>
>
>
>
>
> On 11/15/2016 11:45 PM, John.E.Gregg@wellsfargo.com 
> <ma...@wellsfargo.com> wrote:
>
>         -----Original Message-----
>
>         From: George I. Develekos [mailto:gdevelekos@omilia.com]
>
>         Sent: Tuesday, November 15, 2016 3:00 PM
>
>         To:users@tomcat.apache.org <ma...@tomcat.apache.org>
>
>         Subject: Re: Please help with Tomcat Garbage Collection
>
>         The system does very little swapping, both when it's GC'ing and when it's not.
>
>         Less than 100MB worth of swap is taken.
>
>         Giving Tomcat its own HW is not an option at this time, especially as there's no
>
>         guarantee it'll solve the problem. Besides it would be a VM anyway, not physical
>
>         dedicated HW.  The current server is also a VM.
>
>         On 15-Nov-16 10:55 PM, Zden\u011bk Henek wrote:
>
>             I would start with moving this tomcat to its own hw.
>
>             Did you check swap? This long pauses could be because part of your
>
>             heap is swapped to hdd
>
>             Regards,
>
>             Zdenek Henek
>
>             On Tue, Nov 15, 2016, 21:37 George I. Develekos
>
>             <gd...@omilia.com> <ma...@omilia.com>
>
>             wrote:
>
>                 On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
>
>                     George,
>
>                     On 11/15/16 10:46 AM, George I. Develekos wrote:
>
>                         Hello guys,
>
>                         We are having problems on a production system with very long "full
>
>                         GC" times, as long as1200sec real time (!!!).
>
>                         We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
>
>                         7.0.64.
>
>                         Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
>
>                         -XX:+CMSIncrementalMode
>
>                         No other custom memory-related settings are in place.
>
>                         Looking at the GC log, the last few Full-GC entries are:
>
>                         1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
>
>                         2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
>
>                         169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
>
>                         user=1.96 sys=0.13, real=2.11 secs]
>
>                         2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
>
>                         2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
>
>                         real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
>
>                         [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
>
>                         user=6.12 sys=1.36, real=*23.21 secs*]
>
>                         4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
>
>                         2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
>
>                         169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
>
>                         user=2.18 sys=0.14, real=2.34 secs]
>
>                         What can we do?
>
>                     1367.020 Full GC duration=2.11s
>
>                     2579.317 Full GC duration=1212.49s
>
>                     So your full GC immediately started another full GC that took 20
>
>                     minutes ?
>
>                     Are you only showing certain FULL GC activity from your log, or is
>
>                     that everything?
>
>                     CMS should have a mark and then a sweep each time, but your times
>
>                     don't seem to add up.
>
>                     also note that the whole point of CMS is that there isn't any
>
>                     stop-the-world during the mark portion of the process.
>
>                     Are you actually experiencing a problem, or are you just suffering
>
>                     from instrumentor's remorse?
>
>                     - -chris
>
>                 Chris,
>
>                 What I listed is the result of the command:
>
>                 grep "Full GC" gc.log
>
>                 So (obviously) I have skipped other GC activity, i.e. whatever GC
>
>                 activity didn't include the "Full GC" string.
>
>                 Yes we are having app trouble due to the GC delays so this is a real
>
>                 problem. Our application has real-time constraints so the GC delays
>
>                 cannot be tolerated. I selected those GC options _in order to avoid
>
>                 _long GC times.
>
>                 Additionally, these periods coincide with high CPU for that JVM
>
>                 process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
>
>                 Once GC is done, our app rushes to catch up with tasks that had to
>
>                 wait for GC to finish.
>
>                 Answering another question from a member who has kindly responded,
>
>                 yes the server is running other stuff. Basically it runs three
>
>                 tomcats, the main one being this one. It also runs a DB2 database
>
>                 that has close-to-zero activity.
>
>                 George
>
>     It might be helpful if you could post a larger chunk of your GC log, at least long enough to cover the start and end of the CMS phases and maybe even more.  Additionally, try using a tool like GCViewer to analyze the log.
>
>     How many CPUs do you have?  60% CPU usage isn't usually a big deal.
>
>     Like Chris already said, this is not a stop-the-world phase, so your application should continue working in parallel with the garbage collector.
>
>     Looks like your young generation is only 150MB (2080128k - 1926784k.)  That's very small for a 2-5GB heap.  Are you explicitly setting it somewhere or is the JVM choosing that for you?  It's so small that your old generation might be filling up faster that it should, leading to more frequent full collections.  You could try setting the young generation to something like 25-50% of the total heap.  You'd get a lot of small pauses as the young gen is collected but fewer long ones.
>
>     John
>
>     ---------------------------------------------------------------------
>
>     To unsubscribe, e-mail:users-unsubscribe@tomcat.apache.org
>     <ma...@tomcat.apache.org>
>
>     For additional commands, e-mail:users-help@tomcat.apache.org <ma...@tomcat.apache.org>
>
>
>
> ------------------------------------------------------------------------
>
> Avast logo 
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
> 	
>
> This email has been checked for viruses by Avast antivirus software.
> www.avast.com 
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> 
>
>



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

RE: Please help with Tomcat Garbage Collection

Posted by Jo...@wellsfargo.com.
Sorry for top posting.  The format got weird.

Those numbers aren’t bad.  The most important number for me is the throughput on the summary tab.  Yours is 99.38%.  That means the JVM was doing real work 99.38% of the time and garbage collecting the other .62%.  You could improve that if you worked hard enough, but going from 99.4 to 99.5 or 99.6 probably isn’t worth the effort.  If the number was 90% or something, then you’d have more room for improvement.

Look at the GC performance numbers on the summary tab.  Obviously minor GCs are much faster in this regard than major GCs.  You can reduce your total GC time by increasing the size of your young generation.  You will get more or slower young collections but fewer and faster old collections.  Overall the total time will be less than it is now and the longest pauses will be shorter.

As others have said, though, something doesn’t add up.  CMS is only stop-the-world during certain phases.  (Not the ones with “concurrent” in the name.)  If you feel these GC events coincide with pauses in your app, you can try a thread dump or three (kill -3 <pid>) during the pause.  Use a tool like Samurai to parse the output.  This might only be practical for longer pauses, though.

Also, is it possible the VM itself is having a problem?  Maybe you should talk to your virtualization team to see how stressed the hardware is.  VMWare has an informative java best practices doc: http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/enterprise-java-applications-on-vmware-best-practices-guide.pdf

John




From: George I. Develekos [mailto:gdevelekos@omilia.com]
Sent: Wednesday, November 16, 2016 5:21 AM
To: users@tomcat.apache.org
Subject: Re: Please help with Tomcat Garbage Collection

I'm attaching three screenshots of the GCViewer app as it processed the complete gc.log file (about 19 hours).

Please have a look and advise on what I can do to limit Full-GC times. As of now I have a recommendation to increase the Young Gen..

The setup in summary:

We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat 7.0.64.

Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC   -XX:+CMSIncrementalMode


[cid:image001.png@01D23FEE.84B81AB0]

[cid:image002.png@01D23FEE.84B81AB0]


[cid:image003.png@01D23FEE.84B81AB0]
On 11/15/2016 11:45 PM, John.E.Gregg@wellsfargo.com<ma...@wellsfargo.com> wrote:







-----Original Message-----

From: George I. Develekos [mailto:gdevelekos@omilia.com]

Sent: Tuesday, November 15, 2016 3:00 PM

To: users@tomcat.apache.org<ma...@tomcat.apache.org>

Subject: Re: Please help with Tomcat Garbage Collection



The system does very little swapping, both when it's GC'ing and when it's not.

Less than 100MB worth of swap is taken.



Giving Tomcat its own HW is not an option at this time, especially as there's no

guarantee it'll solve the problem. Besides it would be a VM anyway, not physical

dedicated HW.  The current server is also a VM.





On 15-Nov-16 10:55 PM, Zdeněk Henek wrote:

I would start with moving this tomcat to its own hw.



Did you check swap? This long pauses could be because part of your

heap is swapped to hdd



Regards,

Zdenek Henek



On Tue, Nov 15, 2016, 21:37 George I. Develekos

<gd...@omilia.com>

wrote:



On 15-Nov-16 10:22 PM, Christopher Schultz wrote:

George,



On 11/15/16 10:46 AM, George I. Develekos wrote:

Hello guys,



We are having problems on a production system with very long "full

GC" times, as long as1200sec real time (!!!).



We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat

7.0.64.



Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC

-XX:+CMSIncrementalMode



No other custom memory-related settings are in place.



Looking at the GC log, the last few Full-GC entries are:



1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),

2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :

169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:

user=1.96 sys=0.13, real=2.11 secs]



2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:

2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,

real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:

[CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:

user=6.12 sys=1.36, real=*23.21 secs*]



4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),

2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :

169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:

user=2.18 sys=0.14, real=2.34 secs]





What can we do?

1367.020 Full GC duration=2.11s

2579.317 Full GC duration=1212.49s



So your full GC immediately started another full GC that took 20

minutes ?



Are you only showing certain FULL GC activity from your log, or is

that everything?



CMS should have a mark and then a sweep each time, but your times

don't seem to add up.



also note that the whole point of CMS is that there isn't any

stop-the-world during the mark portion of the process.



Are you actually experiencing a problem, or are you just suffering

from instrumentor's remorse?



- -chris



Chris,



What I listed is the result of the command:



grep "Full GC" gc.log



So (obviously) I have skipped other GC activity, i.e. whatever GC

activity didn't include the "Full GC" string.



Yes we are having app trouble due to the GC delays so this is a real

problem. Our application has real-time constraints so the GC delays

cannot be tolerated. I selected those GC options _in order to avoid

_long GC times.



Additionally, these periods coincide with high CPU for that JVM

process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.

Once GC is done, our app rushes to catch up with tasks that had to

wait for GC to finish.



Answering another question from a member who has kindly responded,

yes the server is running other stuff. Basically it runs three

tomcats, the main one being this one. It also runs a DB2 database

that has close-to-zero activity.



George







It might be helpful if you could post a larger chunk of your GC log, at least long enough to cover the start and end of the CMS phases and maybe even more.  Additionally, try using a tool like GCViewer to analyze the log.



How many CPUs do you have?  60% CPU usage isn't usually a big deal.



Like Chris already said, this is not a stop-the-world phase, so your application should continue working in parallel with the garbage collector.



Looks like your young generation is only 150MB (2080128k - 1926784k.)  That's very small for a 2-5GB heap.  Are you explicitly setting it somewhere or is the JVM choosing that for you?  It's so small that your old generation might be filling up faster that it should, leading to more frequent full collections.  You could try setting the young generation to something like 25-50% of the total heap.  You'd get a lot of small pauses as the young gen is collected but fewer long ones.



John



---------------------------------------------------------------------

To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org<ma...@tomcat.apache.org>

For additional commands, e-mail: users-help@tomcat.apache.org<ma...@tomcat.apache.org>




________________________________
[Avast logo]<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>


This email has been checked for viruses by Avast antivirus software.
www.avast.com<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>



Re: Please help with Tomcat Garbage Collection

Posted by "George I. Develekos" <gd...@omilia.com>.
I'm attaching three screenshots of the GCViewer app as it processed the 
complete *gc.log* file (about 19 hours).

Please have a look and advise on what I can do to limit Full-GC times. 
As of now I have a recommendation to increase the Young Gen..

The setup in summary:

We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat 7.0.64.

Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC   
-XX:+CMSIncrementalMode










On 11/15/2016 11:45 PM, John.E.Gregg@wellsfargo.com wrote:
>
>
>> -----Original Message-----
>> From: George I. Develekos [mailto:gdevelekos@omilia.com]
>> Sent: Tuesday, November 15, 2016 3:00 PM
>> To: users@tomcat.apache.org
>> Subject: Re: Please help with Tomcat Garbage Collection
>>
>> The system does very little swapping, both when it's GC'ing and when it's not.
>> Less than 100MB worth of swap is taken.
>>
>> Giving Tomcat its own HW is not an option at this time, especially as there's no
>> guarantee it'll solve the problem. Besides it would be a VM anyway, not physical
>> dedicated HW.  The current server is also a VM.
>>
>>
>> On 15-Nov-16 10:55 PM, Zden\u011bk Henek wrote:
>>> I would start with moving this tomcat to its own hw.
>>>
>>> Did you check swap? This long pauses could be because part of your
>>> heap is swapped to hdd
>>>
>>> Regards,
>>> Zdenek Henek
>>>
>>> On Tue, Nov 15, 2016, 21:37 George I. Develekos
>>> <gd...@omilia.com>
>>> wrote:
>>>
>>>> On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
>>>>> George,
>>>>>
>>>>> On 11/15/16 10:46 AM, George I. Develekos wrote:
>>>>>> Hello guys,
>>>>>>
>>>>>> We are having problems on a production system with very long "full
>>>>>> GC" times, as long as1200sec real time (!!!).
>>>>>>
>>>>>> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
>>>>>> 7.0.64.
>>>>>>
>>>>>> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
>>>>>> -XX:+CMSIncrementalMode
>>>>>>
>>>>>> No other custom memory-related settings are in place.
>>>>>>
>>>>>> Looking at the GC log, the last few Full-GC entries are:
>>>>>>
>>>>>> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
>>>>>> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
>>>>>> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
>>>>>> user=1.96 sys=0.13, real=2.11 secs]
>>>>>>
>>>>>> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
>>>>>> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
>>>>>> real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
>>>>>> [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
>>>>>> user=6.12 sys=1.36, real=*23.21 secs*]
>>>>>>
>>>>>> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
>>>>>> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
>>>>>> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
>>>>>> user=2.18 sys=0.14, real=2.34 secs]
>>>>>>
>>>>>>
>>>>>> What can we do?
>>>>> 1367.020 Full GC duration=2.11s
>>>>> 2579.317 Full GC duration=1212.49s
>>>>>
>>>>> So your full GC immediately started another full GC that took 20
>>>>> minutes ?
>>>>>
>>>>> Are you only showing certain FULL GC activity from your log, or is
>>>>> that everything?
>>>>>
>>>>> CMS should have a mark and then a sweep each time, but your times
>>>>> don't seem to add up.
>>>>>
>>>>> also note that the whole point of CMS is that there isn't any
>>>>> stop-the-world during the mark portion of the process.
>>>>>
>>>>> Are you actually experiencing a problem, or are you just suffering
>>>>> from instrumentor's remorse?
>>>>>
>>>>> - -chris
>>>>>
>>>> Chris,
>>>>
>>>> What I listed is the result of the command:
>>>>
>>>> grep "Full GC" gc.log
>>>>
>>>> So (obviously) I have skipped other GC activity, i.e. whatever GC
>>>> activity didn't include the "Full GC" string.
>>>>
>>>> Yes we are having app trouble due to the GC delays so this is a real
>>>> problem. Our application has real-time constraints so the GC delays
>>>> cannot be tolerated. I selected those GC options _in order to avoid
>>>> _long GC times.
>>>>
>>>> Additionally, these periods coincide with high CPU for that JVM
>>>> process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
>>>> Once GC is done, our app rushes to catch up with tasks that had to
>>>> wait for GC to finish.
>>>>
>>>> Answering another question from a member who has kindly responded,
>>>> yes the server is running other stuff. Basically it runs three
>>>> tomcats, the main one being this one. It also runs a DB2 database
>>>> that has close-to-zero activity.
>>>>
>>>> George
>>>>
>>>>
> It might be helpful if you could post a larger chunk of your GC log, at least long enough to cover the start and end of the CMS phases and maybe even more.  Additionally, try using a tool like GCViewer to analyze the log.
>
> How many CPUs do you have?  60% CPU usage isn't usually a big deal.
>
> Like Chris already said, this is not a stop-the-world phase, so your application should continue working in parallel with the garbage collector.
>
> Looks like your young generation is only 150MB (2080128k - 1926784k.)  That's very small for a 2-5GB heap.  Are you explicitly setting it somewhere or is the JVM choosing that for you?  It's so small that your old generation might be filling up faster that it should, leading to more frequent full collections.  You could try setting the young generation to something like 25-50% of the total heap.  You'd get a lot of small pauses as the young gen is collected but fewer long ones.
>
> John
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Please help with Tomcat Garbage Collection

Posted by "George I. Develekos" <gd...@omilia.com>.

On 15-Nov-16 11:45 PM, John.E.Gregg@wellsfargo.com wrote:
>
>
>> -----Original Message-----
>> From: George I. Develekos [mailto:gdevelekos@omilia.com]
>> Sent: Tuesday, November 15, 2016 3:00 PM
>> To: users@tomcat.apache.org
>> Subject: Re: Please help with Tomcat Garbage Collection
>>
>> The system does very little swapping, both when it's GC'ing and when it's not.
>> Less than 100MB worth of swap is taken.
>>
>> Giving Tomcat its own HW is not an option at this time, especially as there's no
>> guarantee it'll solve the problem. Besides it would be a VM anyway, not physical
>> dedicated HW.  The current server is also a VM.
>>
>>
>> On 15-Nov-16 10:55 PM, Zden\u011bk Henek wrote:
>>> I would start with moving this tomcat to its own hw.
>>>
>>> Did you check swap? This long pauses could be because part of your
>>> heap is swapped to hdd
>>>
>>> Regards,
>>> Zdenek Henek
>>>
>>> On Tue, Nov 15, 2016, 21:37 George I. Develekos
>>> <gd...@omilia.com>
>>> wrote:
>>>
>>>> On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
>>>>> George,
>>>>>
>>>>> On 11/15/16 10:46 AM, George I. Develekos wrote:
>>>>>> Hello guys,
>>>>>>
>>>>>> We are having problems on a production system with very long "full
>>>>>> GC" times, as long as1200sec real time (!!!).
>>>>>>
>>>>>> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
>>>>>> 7.0.64.
>>>>>>
>>>>>> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
>>>>>> -XX:+CMSIncrementalMode
>>>>>>
>>>>>> No other custom memory-related settings are in place.
>>>>>>
>>>>>> Looking at the GC log, the last few Full-GC entries are:
>>>>>>
>>>>>> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
>>>>>> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
>>>>>> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
>>>>>> user=1.96 sys=0.13, real=2.11 secs]
>>>>>>
>>>>>> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
>>>>>> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
>>>>>> real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
>>>>>> [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
>>>>>> user=6.12 sys=1.36, real=*23.21 secs*]
>>>>>>
>>>>>> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
>>>>>> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
>>>>>> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
>>>>>> user=2.18 sys=0.14, real=2.34 secs]
>>>>>>
>>>>>>
>>>>>> What can we do?
>>>>> 1367.020 Full GC duration=2.11s
>>>>> 2579.317 Full GC duration=1212.49s
>>>>>
>>>>> So your full GC immediately started another full GC that took 20
>>>>> minutes ?
>>>>>
>>>>> Are you only showing certain FULL GC activity from your log, or is
>>>>> that everything?
>>>>>
>>>>> CMS should have a mark and then a sweep each time, but your times
>>>>> don't seem to add up.
>>>>>
>>>>> also note that the whole point of CMS is that there isn't any
>>>>> stop-the-world during the mark portion of the process.
>>>>>
>>>>> Are you actually experiencing a problem, or are you just suffering
>>>>> from instrumentor's remorse?
>>>>>
>>>>> - -chris
>>>>>
>>>> Chris,
>>>>
>>>> What I listed is the result of the command:
>>>>
>>>> grep "Full GC" gc.log
>>>>
>>>> So (obviously) I have skipped other GC activity, i.e. whatever GC
>>>> activity didn't include the "Full GC" string.
>>>>
>>>> Yes we are having app trouble due to the GC delays so this is a real
>>>> problem. Our application has real-time constraints so the GC delays
>>>> cannot be tolerated. I selected those GC options _in order to avoid
>>>> _long GC times.
>>>>
>>>> Additionally, these periods coincide with high CPU for that JVM
>>>> process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
>>>> Once GC is done, our app rushes to catch up with tasks that had to
>>>> wait for GC to finish.
>>>>
>>>> Answering another question from a member who has kindly responded,
>>>> yes the server is running other stuff. Basically it runs three
>>>> tomcats, the main one being this one. It also runs a DB2 database
>>>> that has close-to-zero activity.
>>>>
>>>> George
>>>>
>>>>
> It might be helpful if you could post a larger chunk of your GC log, at least long enough to cover the start and end of the CMS phases and maybe even more.  Additionally, try using a tool like GCViewer to analyze the log.
>
> How many CPUs do you have?  60% CPU usage isn't usually a big deal.
>
> Like Chris already said, this is not a stop-the-world phase, so your application should continue working in parallel with the garbage collector.
>
> Looks like your young generation is only 150MB (2080128k - 1926784k.)  That's very small for a 2-5GB heap.  Are you explicitly setting it somewhere or is the JVM choosing that for you?  It's so small that your old generation might be filling up faster that it should, leading to more frequent full collections.  You could try setting the young generation to something like 25-50% of the total heap.  You'd get a lot of small pauses as the young gen is collected but fewer long ones.
>
> John
>
John,

I have a couple of cores. One would be enough for what the server does.

I'll post a larger log chunk tomorrow.  I'll look into GCViewer, thanks.

I don't set the young gen, It's selected by the JVM. I don't even know 
what that is, to be honest. I do too many things for my company to know 
such esoteric things about each one of them. Server admin, network 
admin, DB admin, all admin....

Nevertheless, if setting the Young gen makes GC less frequent it won't 
really help, because still I'll have periods of "crisis", just not as 
frequently. Ideally I want to keep Full-GC under 3sec, and from what 
I've read it's a realistic goal.

Looking at application logs, it looks like app tasks DO stop running 
during Full-GC.



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


RE: Please help with Tomcat Garbage Collection

Posted by Jo...@wellsfargo.com.


> -----Original Message-----
> From: George I. Develekos [mailto:gdevelekos@omilia.com]
> Sent: Tuesday, November 15, 2016 3:00 PM
> To: users@tomcat.apache.org
> Subject: Re: Please help with Tomcat Garbage Collection
> 
> The system does very little swapping, both when it's GC'ing and when it's not.
> Less than 100MB worth of swap is taken.
> 
> Giving Tomcat its own HW is not an option at this time, especially as there's no
> guarantee it'll solve the problem. Besides it would be a VM anyway, not physical
> dedicated HW.  The current server is also a VM.
> 
> 
> On 15-Nov-16 10:55 PM, Zdeněk Henek wrote:
> > I would start with moving this tomcat to its own hw.
> >
> > Did you check swap? This long pauses could be because part of your
> > heap is swapped to hdd
> >
> > Regards,
> > Zdenek Henek
> >
> > On Tue, Nov 15, 2016, 21:37 George I. Develekos
> > <gd...@omilia.com>
> > wrote:
> >
> >> On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
> >>> George,
> >>>
> >>> On 11/15/16 10:46 AM, George I. Develekos wrote:
> >>>> Hello guys,
> >>>>
> >>>> We are having problems on a production system with very long "full
> >>>> GC" times, as long as1200sec real time (!!!).
> >>>>
> >>>> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
> >>>> 7.0.64.
> >>>>
> >>>> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
> >>>> -XX:+CMSIncrementalMode
> >>>>
> >>>> No other custom memory-related settings are in place.
> >>>>
> >>>> Looking at the GC log, the last few Full-GC entries are:
> >>>>
> >>>> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
> >>>> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
> >>>> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
> >>>> user=1.96 sys=0.13, real=2.11 secs]
> >>>>
> >>>> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
> >>>> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
> >>>> real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
> >>>> [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
> >>>> user=6.12 sys=1.36, real=*23.21 secs*]
> >>>>
> >>>> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
> >>>> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
> >>>> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
> >>>> user=2.18 sys=0.14, real=2.34 secs]
> >>>>
> >>>>
> >>>> What can we do?
> >>> 1367.020 Full GC duration=2.11s
> >>> 2579.317 Full GC duration=1212.49s
> >>>
> >>> So your full GC immediately started another full GC that took 20
> >>> minutes ?
> >>>
> >>> Are you only showing certain FULL GC activity from your log, or is
> >>> that everything?
> >>>
> >>> CMS should have a mark and then a sweep each time, but your times
> >>> don't seem to add up.
> >>>
> >>> also note that the whole point of CMS is that there isn't any
> >>> stop-the-world during the mark portion of the process.
> >>>
> >>> Are you actually experiencing a problem, or are you just suffering
> >>> from instrumentor's remorse?
> >>>
> >>> - -chris
> >>>
> >> Chris,
> >>
> >> What I listed is the result of the command:
> >>
> >> grep "Full GC" gc.log
> >>
> >> So (obviously) I have skipped other GC activity, i.e. whatever GC
> >> activity didn't include the "Full GC" string.
> >>
> >> Yes we are having app trouble due to the GC delays so this is a real
> >> problem. Our application has real-time constraints so the GC delays
> >> cannot be tolerated. I selected those GC options _in order to avoid
> >> _long GC times.
> >>
> >> Additionally, these periods coincide with high CPU for that JVM
> >> process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
> >> Once GC is done, our app rushes to catch up with tasks that had to
> >> wait for GC to finish.
> >>
> >> Answering another question from a member who has kindly responded,
> >> yes the server is running other stuff. Basically it runs three
> >> tomcats, the main one being this one. It also runs a DB2 database
> >> that has close-to-zero activity.
> >>
> >> George
> >>
> >>

It might be helpful if you could post a larger chunk of your GC log, at least long enough to cover the start and end of the CMS phases and maybe even more.  Additionally, try using a tool like GCViewer to analyze the log.

How many CPUs do you have?  60% CPU usage isn't usually a big deal.

Like Chris already said, this is not a stop-the-world phase, so your application should continue working in parallel with the garbage collector.

Looks like your young generation is only 150MB (2080128k - 1926784k.)  That's very small for a 2-5GB heap.  Are you explicitly setting it somewhere or is the JVM choosing that for you?  It's so small that your old generation might be filling up faster that it should, leading to more frequent full collections.  You could try setting the young generation to something like 25-50% of the total heap.  You'd get a lot of small pauses as the young gen is collected but fewer long ones.

John

Re: Please help with Tomcat Garbage Collection

Posted by "George I. Develekos" <gd...@omilia.com>.
The system does very little swapping, both when it's GC'ing and when 
it's not. Less than 100MB worth of swap is taken.

Giving Tomcat its own HW is not an option at this time, especially as 
there's no guarantee it'll solve the problem. Besides it would be a VM 
anyway, not physical dedicated HW.  The current server is also a VM.


On 15-Nov-16 10:55 PM, Zden\u011bk Henek wrote:
> I would start with moving this tomcat to its own hw.
>
> Did you check swap? This long pauses could be because part of your heap is
> swapped to hdd
>
> Regards,
> Zdenek Henek
>
> On Tue, Nov 15, 2016, 21:37 George I. Develekos <gd...@omilia.com>
> wrote:
>
>> On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
>>> George,
>>>
>>> On 11/15/16 10:46 AM, George I. Develekos wrote:
>>>> Hello guys,
>>>>
>>>> We are having problems on a production system with very long "full
>>>> GC" times, as long as1200sec real time (!!!).
>>>>
>>>> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
>>>> 7.0.64.
>>>>
>>>> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
>>>> -XX:+CMSIncrementalMode
>>>>
>>>> No other custom memory-related settings are in place.
>>>>
>>>> Looking at the GC log, the last few Full-GC entries are:
>>>>
>>>> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
>>>> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
>>>> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
>>>> user=1.96 sys=0.13, real=2.11 secs]
>>>>
>>>> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
>>>> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
>>>> real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
>>>> [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
>>>> user=6.12 sys=1.36, real=*23.21 secs*]
>>>>
>>>> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
>>>> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
>>>> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
>>>> user=2.18 sys=0.14, real=2.34 secs]
>>>>
>>>>
>>>> What can we do?
>>> 1367.020 Full GC duration=2.11s
>>> 2579.317 Full GC duration=1212.49s
>>>
>>> So your full GC immediately started another full GC that took 20 minutes
>>> ?
>>>
>>> Are you only showing certain FULL GC activity from your log, or is
>>> that everything?
>>>
>>> CMS should have a mark and then a sweep each time, but your times
>>> don't seem to add up.
>>>
>>> also note that the whole point of CMS is that there isn't any
>>> stop-the-world during the mark portion of the process.
>>>
>>> Are you actually experiencing a problem, or are you just suffering
>>> from instrumentor's remorse?
>>>
>>> - -chris
>>>
>> Chris,
>>
>> What I listed is the result of the command:
>>
>> grep "Full GC" gc.log
>>
>> So (obviously) I have skipped other GC activity, i.e. whatever GC
>> activity didn't include the "Full GC" string.
>>
>> Yes we are having app trouble due to the GC delays so this is a real
>> problem. Our application has real-time constraints so the GC delays
>> cannot be tolerated. I selected those GC options _in order to avoid
>> _long GC times.
>>
>> Additionally, these periods coincide with high CPU for that JVM
>> process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
>> Once GC is done, our app rushes to catch up with tasks that had to wait
>> for GC to finish.
>>
>> Answering another question from a member who has kindly responded, yes
>> the server is running other stuff. Basically it runs three tomcats, the
>> main one being this one. It also runs a DB2 database that has
>> close-to-zero activity.
>>
>> George
>>
>>
>> ---
>> This email has been checked for viruses by Avast antivirus software.
>> https://www.avast.com/antivirus
>>


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Please help with Tomcat Garbage Collection

Posted by Zdeněk Henek <vr...@gmail.com>.
I would start with moving this tomcat to its own hw.

Did you check swap? This long pauses could be because part of your heap is
swapped to hdd

Regards,
Zdenek Henek

On Tue, Nov 15, 2016, 21:37 George I. Develekos <gd...@omilia.com>
wrote:

>
> On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
> >
> > George,
> >
> > On 11/15/16 10:46 AM, George I. Develekos wrote:
> >> Hello guys,
> >>
> >> We are having problems on a production system with very long "full
> >> GC" times, as long as1200sec real time (!!!).
> >>
> >> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
> >> 7.0.64.
> >>
> >> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
> >> -XX:+CMSIncrementalMode
> >>
> >> No other custom memory-related settings are in place.
> >>
> >> Looking at the GC log, the last few Full-GC entries are:
> >>
> >> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
> >> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
> >> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
> >> user=1.96 sys=0.13, real=2.11 secs]
> >>
> >> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
> >> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
> >> real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
> >> [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
> >> user=6.12 sys=1.36, real=*23.21 secs*]
> >>
> >> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
> >> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
> >> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
> >> user=2.18 sys=0.14, real=2.34 secs]
> >>
> >>
> >> What can we do?
> > 1367.020 Full GC duration=2.11s
> > 2579.317 Full GC duration=1212.49s
> >
> > So your full GC immediately started another full GC that took 20 minutes
> > ?
> >
> > Are you only showing certain FULL GC activity from your log, or is
> > that everything?
> >
> > CMS should have a mark and then a sweep each time, but your times
> > don't seem to add up.
> >
> > also note that the whole point of CMS is that there isn't any
> > stop-the-world during the mark portion of the process.
> >
> > Are you actually experiencing a problem, or are you just suffering
> > from instrumentor's remorse?
> >
> > - -chris
> >
>
> Chris,
>
> What I listed is the result of the command:
>
> grep "Full GC" gc.log
>
> So (obviously) I have skipped other GC activity, i.e. whatever GC
> activity didn't include the "Full GC" string.
>
> Yes we are having app trouble due to the GC delays so this is a real
> problem. Our application has real-time constraints so the GC delays
> cannot be tolerated. I selected those GC options _in order to avoid
> _long GC times.
>
> Additionally, these periods coincide with high CPU for that JVM
> process.  From 5-20% CPU where it is normally, it jumps to 60% ore more.
> Once GC is done, our app rushes to catch up with tasks that had to wait
> for GC to finish.
>
> Answering another question from a member who has kindly responded, yes
> the server is running other stuff. Basically it runs three tomcats, the
> main one being this one. It also runs a DB2 database that has
> close-to-zero activity.
>
> George
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>

Re: Please help with Tomcat Garbage Collection

Posted by "George I. Develekos" <gd...@omilia.com>.
On 15-Nov-16 10:22 PM, Christopher Schultz wrote:
>
> George,
>
> On 11/15/16 10:46 AM, George I. Develekos wrote:
>> Hello guys,
>>
>> We are having problems on a production system with very long "full
>> GC" times, as long as1200sec real time (!!!).
>>
>> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
>> 7.0.64.
>>
>> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
>> -XX:+CMSIncrementalMode
>>
>> No other custom memory-related settings are in place.
>>
>> Looking at the GC log, the last few Full-GC entries are:
>>
>> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
>> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
>> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
>> user=1.96 sys=0.13, real=2.11 secs]
>>
>> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
>> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
>> real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
>> [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
>> user=6.12 sys=1.36, real=*23.21 secs*]
>>
>> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
>> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
>> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
>> user=2.18 sys=0.14, real=2.34 secs]
>>
>>
>> What can we do?
> 1367.020 Full GC duration=2.11s
> 2579.317 Full GC duration=1212.49s
>
> So your full GC immediately started another full GC that took 20 minutes
> ?
>
> Are you only showing certain FULL GC activity from your log, or is
> that everything?
>
> CMS should have a mark and then a sweep each time, but your times
> don't seem to add up.
>
> also note that the whole point of CMS is that there isn't any
> stop-the-world during the mark portion of the process.
>
> Are you actually experiencing a problem, or are you just suffering
> from instrumentor's remorse?
>
> - -chris
>

Chris,

What I listed is the result of the command:

grep "Full GC" gc.log

So (obviously) I have skipped other GC activity, i.e. whatever GC 
activity didn't include the "Full GC" string.

Yes we are having app trouble due to the GC delays so this is a real 
problem. Our application has real-time constraints so the GC delays 
cannot be tolerated. I selected those GC options _in order to avoid 
_long GC times.

Additionally, these periods coincide with high CPU for that JVM 
process.  From 5-20% CPU where it is normally, it jumps to 60% ore more. 
Once GC is done, our app rushes to catch up with tasks that had to wait 
for GC to finish.

Answering another question from a member who has kindly responded, yes 
the server is running other stuff. Basically it runs three tomcats, the 
main one being this one. It also runs a DB2 database that has 
close-to-zero activity.

George


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: Please help with Tomcat Garbage Collection

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

George,

On 11/15/16 10:46 AM, George I. Develekos wrote:
> Hello guys,
> 
> We are having problems on a production system with very long "full
> GC" times, as long as1200sec real time (!!!).
> 
> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat
> 7.0.64.
> 
> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC 
> -XX:+CMSIncrementalMode
> 
> No other custom memory-related settings are in place.
> 
> Looking at the GC log, the last few Full-GC entries are:
> 
> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K), 
> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm : 
> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times:
> user=1.96 sys=0.13, real=2.11 secs]
> 
> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark: 
> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01,
> real=**1212.49 **secs] ** * 3539.969: [Full GC 3539.969:
> [CMS3540.056: [CMS-concurrent-sweep: 1.571/23.223 secs] [Times:
> user=6.12 sys=1.36, real=*23.21 secs*]
> 
> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K), 
> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm : 
> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times:
> user=2.18 sys=0.14, real=2.34 secs]
> 
> 
> What can we do?

1367.020 Full GC duration=2.11s
2579.317 Full GC duration=1212.49s

So your full GC immediately started another full GC that took 20 minutes
?

Are you only showing certain FULL GC activity from your log, or is
that everything?

CMS should have a mark and then a sweep each time, but your times
don't seem to add up.

also note that the whole point of CMS is that there isn't any
stop-the-world during the mark portion of the process.

Are you actually experiencing a problem, or are you just suffering
from instrumentor's remorse?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCAAGBQJYK25oAAoJEBzwKT+lPKRYQNEQAJ8xASbPXtJUHy7xzhdYpwnr
M6wpCnYcls/cihN1p5OgJaV5G4EyNDGzoDky/ijW6o6ywlAkwAw0TOZKiUD6KN3q
LPq/3nHCvUr0769TuzkLpCe+VjYKmD4k4JGq9qqBLrKeMhfEprKnod+BR2ZKVK03
WmI6mqYKfkdPi3iR5GBv95Wo4SXAONGSnG06ISo+mPPOrLqH8QSrGocFH63/+nU8
Vx3yESyWL4FiQxFe+BowEnhu26p5IgA76DtgXs8XV+ojKnuM1liZPd6GXTjT6lE/
HkVg7P30sMzVf9P+0naDzHzZw1Pg19c3L5CymF1+WQPUF7Bk7iaOJNX9ZvVvhtpP
aMmyNVp2GUWVXby2rHIS2cA1h0Nq9ZtUZECx5k8dLCYlMgmRlFdgM5W+Un2nP2qf
WrVh0YdsMc9PrP5JSt8MdgxRhRk/WAADC4gTi78hqS6Fuz6I8R5tkXu2xsQA5ALZ
Cd/dvX2hRY4yJ00A3ooYE9sRciQXoSrQ6VvfWfU1k4kI2ZC4C5mDXASQTOdyCKHx
KtGs9hK56hitZSCBhCrDD8OhzmnIJF1E+FmtKwuTEmkc7cGiCXiTJI+coO+sWzzV
AljVYpG8YIDSD+gHrWJbf4Bs9EzfjI7EwvAtszWBC1SM8pAHBbyguCDhFOAbLlp0
gHTCnOGA7/Qvgd+nLLxC
=UeZj
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Re: Please help with Tomcat Garbage Collection

Posted by Zdeněk Henek <vr...@gmail.com>.
Is there running something else in that server? Real time should be never
bigger than user+sys

Z

On Tue, Nov 15, 2016, 16:46 George I. Develekos <gd...@omilia.com>
wrote:

> Hello guys,
>
> We are having problems on a production system with very long "full GC"
> times, as long as1200sec real time (!!!).
>
> We are using Java 6 (stuck with CentOS 5.8 at this time) and Tomcat 7.0.64.
>
> Xmx is 5G, Xms is 2G, and GC options are -XX:+UseConcMarkSweepGC
> -XX:+CMSIncrementalMode
>
> No other custom memory-related settings are in place.
>
> Looking at the GC log, the last few Full-GC entries are:
>
> 1367.020: [Full GC 1367.020: [CMS: 1178831K->527456K(1926784K),
> 2.1117220 secs] 1250378K->527456K(2080128K), [CMS Perm :
> 169762K->56187K(169984K)] icms_dc=0 , 2.1118160 secs] [Times: user=1.96
> sys=0.13, real=2.11 secs]
>
> 2579.317: [Full GC 2579.317: [CMS2581.876: [CMS-concurrent-mark:
> 2.558/1212.733 secs] [*Times: user=113.05 sys=28.01, real=**1212.49
> **secs] **
> *
> 3539.969: [Full GC 3539.969: [CMS3540.056: [CMS-concurrent-sweep:
> 1.571/23.223 secs] [Times: user=6.12 sys=1.36, real=*23.21 secs*]
>
> 4070.456: [Full GC 4070.457: [CMS: 1252569K->591200K(1926784K),
> 2.3447040 secs] 1270617K->591200K(2080128K), [CMS Perm :
> 169983K->56598K(169984K)] icms_dc=0 , 2.3448140 secs] [Times: user=2.18
> sys=0.14, real=2.34 secs]
>
>
> What can we do?
>
> Thanks,
>
> George
>
>
>
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>