You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Pascal Alberty <pa...@albertyorban.be> on 2006/10/12 23:39:24 UTC
CPU level is not coming back to a normal level
Prod Box is:
- Debian kernel 2.6
- Bi-Xeon 3.2 Ghz - 1 GB Ram
- Apache 2.0 - prefork - maxclients = 256 - keealive = false - around
300 request/sec
- Tomcat 5.0 - mod_jk 1.2.18 - 512 Mb allocated.
- JDK 1.5
MySQL is on another separate box.
Web app is an ad server serving all requests through servlets giving
mainly HTML and gif.
At normal charge, around 50 threads are created on tomcat and around
40 are working, others are waiting (. CPUs are around 20% each mainly
for Tomcat.
We are developing a new servlet which goal is to generate a png file
from XHTML using Flying Saucer xhtmlrenderer. This one is using AWT
for the XHTML and CSS rendering.
When I'm testing the new servlet on my dev box, all is ok. When i'm
running the same servlet on the prod box, Tomcat is reaching almost 2
x 100% CPUs for about 4 to 7 seconds. During this time, many threads
are created (to the max, around 256 - see maxclients config). All
Apache sockets are busy and slowing down. After that, sockets are
released and Tomcat is working well again after few seconds.
BUT !! CPUs stay around 2 x 40%. When I dump JVM Tomcat, around 140
threads are still
remaining. 40 are working and around 100 are waiting in the thread pool.
My questions are:
- why CPUs stay busy whereas the same threads number are working in both case ?
- is it possible that's because the notifyall() on these 100 extra threads ?
- if yes, how can I ask Tomcat to kill these non working threads ?
- if no ... any idea ?
Thanks for your help.
--
Pascal Alberty
http://pascal.albertyorban.be
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: CPU level is not coming back to a normal level
Posted by David Smith <dn...@cornell.edu>.
I think you are going to have to get a profiler going to diagnose this.
See where the JVM is spending it's time when the CPU goes high like
this. Also try using the logs to see if it's coincident on specific
requests or other events in tomcat.
--David
Pascal Alberty wrote:
> Any other idea ?
>
> Thanks
>
> On 10/13/06, Pascal Alberty <pa...@albertyorban.be> wrote:
>
>> On 10/12/06, Caldarale, Charles R <Ch...@unisys.com> wrote:
>> > > From: alberty.pascal@gmail.com
>> > > [mailto:alberty.pascal@gmail.com] On Behalf Of Pascal Alberty
>> > > Subject: CPU level is not coming back to a normal level
>> > >
>> > > When I'm testing the new servlet on my dev box, all is ok. When i'm
>> > > running the same servlet on the prod box, Tomcat is reaching
>> almost 2
>> > > x 100% CPUs for about 4 to 7 seconds.
>> >
>> > Does this period coincide with garbage collection? (Although 4 - 7
>> > seconds is rather long for a GC.) Have you tried using a profiler to
>> > see where the threads are spending their time?
>>
>> First, thanks for your quick answer.
>>
>> I have set up GC information. While XHTML rendering, there is one Full
>> GC which spent 0.2 sec. All other GC operations are taking less then
>> 0.01 sec which seems "normal".
>>
>> When dumping the JVM through a kill -3, I can see threads working on
>> sockets talking to apache.
>>
>> > > how can I ask Tomcat to kill these non working threads ?
>> >
>> > What do you have maxSpareThreads for each of your connectors set to?
>> > Look here for the HTTP connector doc:
>> > http://tomcat.apache.org/tomcat-5.5-doc/config/http.html
>>
>> I'm not using this one
>>
>> > or here for AJP:
>> > http://tomcat.apache.org/tomcat-5.5-doc/config/ajp.html
>>
>> maxProcessors is 256 (same as Apache maxclients)
>> maxSpareThreads is not set because it's a tomcat-5.5 parameter and i'm
>> using tomcat 5.0
>>
>> Thanks again
>>
>> --
>> Pascal Alberty
>> http://pascal.albertyorban.be
>>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: CPU level is not coming back to a normal level
Posted by Pascal Alberty <pa...@albertyorban.be>.
Any other idea ?
Thanks
On 10/13/06, Pascal Alberty <pa...@albertyorban.be> wrote:
> On 10/12/06, Caldarale, Charles R <Ch...@unisys.com> wrote:
> > > From: alberty.pascal@gmail.com
> > > [mailto:alberty.pascal@gmail.com] On Behalf Of Pascal Alberty
> > > Subject: CPU level is not coming back to a normal level
> > >
> > > When I'm testing the new servlet on my dev box, all is ok. When i'm
> > > running the same servlet on the prod box, Tomcat is reaching almost 2
> > > x 100% CPUs for about 4 to 7 seconds.
> >
> > Does this period coincide with garbage collection? (Although 4 - 7
> > seconds is rather long for a GC.) Have you tried using a profiler to
> > see where the threads are spending their time?
>
> First, thanks for your quick answer.
>
> I have set up GC information. While XHTML rendering, there is one Full
> GC which spent 0.2 sec. All other GC operations are taking less then
> 0.01 sec which seems "normal".
>
> When dumping the JVM through a kill -3, I can see threads working on
> sockets talking to apache.
>
> > > how can I ask Tomcat to kill these non working threads ?
> >
> > What do you have maxSpareThreads for each of your connectors set to?
> > Look here for the HTTP connector doc:
> > http://tomcat.apache.org/tomcat-5.5-doc/config/http.html
>
> I'm not using this one
>
> > or here for AJP:
> > http://tomcat.apache.org/tomcat-5.5-doc/config/ajp.html
>
> maxProcessors is 256 (same as Apache maxclients)
> maxSpareThreads is not set because it's a tomcat-5.5 parameter and i'm
> using tomcat 5.0
>
> Thanks again
>
> --
> Pascal Alberty
> http://pascal.albertyorban.be
>
--
Pascal Alberty
http://pascal.albertyorban.be
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: CPU level is not coming back to a normal level
Posted by Pascal Alberty <pa...@albertyorban.be>.
I found something about this thread.
http://www.mail-archive.com/java2d-interest@capra.eng.sun.com/msg03812.html
Do you thing it could be related ?
I just make another charge test, threads growth to 200 without 2D
disposer ... and CPU stays around 20% ...
On 10/13/06, Pascal Alberty <pa...@albertyorban.be> wrote:
> There is also the following one, but in wait mode:
>
> "Java2D Disposer" daemon prio=1 tid=0x083a5858 nid=0x7443 in
> Object.wait() [0x853ff000..0x853ff1c0]
> at java.lang.Object.wait(Native Method)
> - waiting on <0x92d654d0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
> - locked <0x92d654d0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
> at sun.java2d.Disposer.run(Disposer.java:107)
> at java.lang.Thread.run(Thread.java:595)
>
> Complete dump could be found here:
> http://pascal.albertyorban.be/wp-content/dump.txt (i don't know if
> attached files are allowed by this mailinglist).
>
> Thanks for your help
>
> On 10/13/06, Leon Rosenberg <ro...@googlemail.com> wrote:
> > On 10/13/06, Pascal Alberty <pa...@albertyorban.be> wrote:
> > >
> > > I have set up GC information. While XHTML rendering, there is one Full
> > > GC which spent 0.2 sec. All other GC operations are taking less then
> > > 0.01 sec which seems "normal".
> > >
> > > When dumping the JVM through a kill -3, I can see threads working on
> > > sockets talking to apache.
> >
> > and there are no other suspicious working threads?
> >
> > Leon
> >
> > ---------------------------------------------------------------------
> > To start a new topic, e-mail: users@tomcat.apache.org
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
>
>
> --
> Pascal Alberty
> http://pascal.albertyorban.be
>
--
Pascal Alberty
http://pascal.albertyorban.be
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: CPU level is not coming back to a normal level
Posted by Pascal Alberty <pa...@albertyorban.be>.
There is also the following one, but in wait mode:
"Java2D Disposer" daemon prio=1 tid=0x083a5858 nid=0x7443 in
Object.wait() [0x853ff000..0x853ff1c0]
at java.lang.Object.wait(Native Method)
- waiting on <0x92d654d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x92d654d0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at sun.java2d.Disposer.run(Disposer.java:107)
at java.lang.Thread.run(Thread.java:595)
Complete dump could be found here:
http://pascal.albertyorban.be/wp-content/dump.txt (i don't know if
attached files are allowed by this mailinglist).
Thanks for your help
On 10/13/06, Leon Rosenberg <ro...@googlemail.com> wrote:
> On 10/13/06, Pascal Alberty <pa...@albertyorban.be> wrote:
> >
> > I have set up GC information. While XHTML rendering, there is one Full
> > GC which spent 0.2 sec. All other GC operations are taking less then
> > 0.01 sec which seems "normal".
> >
> > When dumping the JVM through a kill -3, I can see threads working on
> > sockets talking to apache.
>
> and there are no other suspicious working threads?
>
> Leon
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
Pascal Alberty
http://pascal.albertyorban.be
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: CPU level is not coming back to a normal level
Posted by Leon Rosenberg <ro...@googlemail.com>.
On 10/13/06, Pascal Alberty <pa...@albertyorban.be> wrote:
>
> I have set up GC information. While XHTML rendering, there is one Full
> GC which spent 0.2 sec. All other GC operations are taking less then
> 0.01 sec which seems "normal".
>
> When dumping the JVM through a kill -3, I can see threads working on
> sockets talking to apache.
and there are no other suspicious working threads?
Leon
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: CPU level is not coming back to a normal level
Posted by Pascal Alberty <pa...@albertyorban.be>.
On 10/12/06, Caldarale, Charles R <Ch...@unisys.com> wrote:
> > From: alberty.pascal@gmail.com
> > [mailto:alberty.pascal@gmail.com] On Behalf Of Pascal Alberty
> > Subject: CPU level is not coming back to a normal level
> >
> > When I'm testing the new servlet on my dev box, all is ok. When i'm
> > running the same servlet on the prod box, Tomcat is reaching almost 2
> > x 100% CPUs for about 4 to 7 seconds.
>
> Does this period coincide with garbage collection? (Although 4 - 7
> seconds is rather long for a GC.) Have you tried using a profiler to
> see where the threads are spending their time?
First, thanks for your quick answer.
I have set up GC information. While XHTML rendering, there is one Full
GC which spent 0.2 sec. All other GC operations are taking less then
0.01 sec which seems "normal".
When dumping the JVM through a kill -3, I can see threads working on
sockets talking to apache.
> > how can I ask Tomcat to kill these non working threads ?
>
> What do you have maxSpareThreads for each of your connectors set to?
> Look here for the HTTP connector doc:
> http://tomcat.apache.org/tomcat-5.5-doc/config/http.html
I'm not using this one
> or here for AJP:
> http://tomcat.apache.org/tomcat-5.5-doc/config/ajp.html
maxProcessors is 256 (same as Apache maxclients)
maxSpareThreads is not set because it's a tomcat-5.5 parameter and i'm
using tomcat 5.0
Thanks again
--
Pascal Alberty
http://pascal.albertyorban.be
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: CPU level is not coming back to a normal level
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: alberty.pascal@gmail.com
> [mailto:alberty.pascal@gmail.com] On Behalf Of Pascal Alberty
> Subject: CPU level is not coming back to a normal level
>
> When I'm testing the new servlet on my dev box, all is ok. When i'm
> running the same servlet on the prod box, Tomcat is reaching almost 2
> x 100% CPUs for about 4 to 7 seconds.
Does this period coincide with garbage collection? (Although 4 - 7
seconds is rather long for a GC.) Have you tried using a profiler to
see where the threads are spending their time?
> how can I ask Tomcat to kill these non working threads ?
What do you have maxSpareThreads for each of your connectors set to?
Look here for the HTTP connector doc:
http://tomcat.apache.org/tomcat-5.5-doc/config/http.html
or here for AJP:
http://tomcat.apache.org/tomcat-5.5-doc/config/ajp.html
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org