You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Guillaume Lahitette <GL...@abingdon.oilfield.slb.com> on 2005/03/15 02:43:02 UTC
Over 1000 threads running in production
Hello Tomcat'oids,
This is Tomcat 4.1.26 on Linux with JDK 1.4.1_03, load balanced accross 2 servers by a BigIP device. User base is around 50.
Our operations team discovered over 1000 java threads running on each production server. It appears that about 20 threads are added
to this count every hour.
Can someone clarify:
a.. how are threads open? Is it 1 thread per HTTP request? Can I monitor this / gather more details?
b.. how and when are threads "closed"? garbage collected?
c.. are threads re-used / pooled?
d.. how to tune the number of threads Tomcat uses appropriately?
e.. Any known issues with our setup?
Any other recommendations, best practices and war stories are greatly appreciated!
Many thanks in advance.
Guillaume
Re: Over 1000 threads running in production
Posted by Tim Funk <fu...@joedog.org>.
Sorry - I run such tests in adhoc manner.
-Tim
Guillaume Lahitette wrote:
> Tim,
>
> Thank you. This is very helpful.
>
> I've tried kill -3 <pid> (following a google hit:
> http://www.findarticles.com/p/articles/mi_m0MLW/is_3_3/ai_114495281/print)
> but can't see the thread dump in catalina.out nor localhost_log....txt
> files. What did I miss (this is Linux)?
>
> If you could share your perl / grep / wc scripts, that would surely
> jumpstart me and give me some ideas! Thanks in advance.
>
> Cheers,
> Guillaume
>
>
>
>>-----Original Message-----
>>From: Tim Funk [mailto:funkman@joedog.org]
>>Sent: 15 March 2005 11:03
>>To: Tomcat Users List
>>Subject: Re: Over 1000 threads running in production
>>
>>
>>Take periodic thread dumps and read them. perl, grep, and wc are
>>your friends
>>to parse the data without going insane.
>>
>>The thread dumps should give you a general idea of who created them.
>>
>>-Tim
>>
>>Guillaume Lahitette wrote:
>>
>>>Hello Tomcat'oids,
>>>
>>>This is Tomcat 4.1.26 on Linux with JDK 1.4.1_03, load balanced
>>
>>accross 2 servers by a BigIP device. User base is around 50.
>>
>>>Our operations team discovered over 1000 java threads running
>>
>>on each production server. It appears that about 20 threads are added
>>
>>>to this count every hour.
>>>
>>>Can someone clarify:
>>> a.. how are threads open? Is it 1 thread per HTTP request?
>>
>>Can I monitor this / gather more details?
>>
>>> b.. how and when are threads "closed"? garbage collected?
>>> c.. are threads re-used / pooled?
>>> d.. how to tune the number of threads Tomcat uses appropriately?
>>> e.. Any known issues with our setup?
>>>Any other recommendations, best practices and war stories are
>>
>>greatly appreciated!
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
RE: Over 1000 threads running in production
Posted by Guillaume Lahitette <GL...@abingdon.oilfield.slb.com>.
Found the solution: to know which PID to "kill -3" for, use ps -ef --forest
to detect the parent thread which has the *most* child threads. Then the
thread dump shows in catalina.out.
This is where I found the answer:
http://groups-beta.google.com/group/weblogic.developer.interest.performance/
browse_thread/thread/336d16199150720f/40be68d56007a1ea?q=periodic+%22thread+
dumps%22+linux+%22kill+-3%22#40be68d56007a1ea
Guillaume
> -----Original Message-----
> From: Guillaume Lahitette [mailto:glahitette@abingdon.oilfield.slb.com]
> Sent: 15 March 2005 11:34
> To: Tomcat Users List
> Subject: RE: Over 1000 threads running in production
>
>
> Tim,
>
> Thank you. This is very helpful.
>
> I've tried kill -3 <pid> (following a google hit:
> http://www.findarticles.com/p/articles/mi_m0MLW/is_3_3/ai_11449528
> 1/print) but can't see the thread dump in catalina.out nor
> localhost_log....txt files. What did I miss (this is Linux)?
>
> If you could share your perl / grep / wc scripts, that would
> surely jumpstart me and give me some ideas! Thanks in advance.
>
> Cheers,
> Guillaume
>
>
> > -----Original Message-----
> > From: Tim Funk [mailto:funkman@joedog.org]
> > Sent: 15 March 2005 11:03
> > To: Tomcat Users List
> > Subject: Re: Over 1000 threads running in production
> >
> >
> > Take periodic thread dumps and read them. perl, grep, and wc are
> > your friends
> > to parse the data without going insane.
> >
> > The thread dumps should give you a general idea of who created them.
> >
> > -Tim
> >
> > Guillaume Lahitette wrote:
> > > Hello Tomcat'oids,
> > >
> > > This is Tomcat 4.1.26 on Linux with JDK 1.4.1_03, load balanced
> > accross 2 servers by a BigIP device. User base is around 50.
> > >
> > > Our operations team discovered over 1000 java threads running
> > on each production server. It appears that about 20 threads are added
> > > to this count every hour.
> > >
> > > Can someone clarify:
> > > a.. how are threads open? Is it 1 thread per HTTP request?
> > Can I monitor this / gather more details?
> > > b.. how and when are threads "closed"? garbage collected?
> > > c.. are threads re-used / pooled?
> > > d.. how to tune the number of threads Tomcat uses appropriately?
> > > e.. Any known issues with our setup?
> > > Any other recommendations, best practices and war stories are
> > greatly appreciated!
> > >
> > > Many thanks in advance.
> > > Guillaume
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
RE: Over 1000 threads running in production
Posted by Guillaume Lahitette <GL...@abingdon.oilfield.slb.com>.
Tim,
Thank you. This is very helpful.
I've tried kill -3 <pid> (following a google hit:
http://www.findarticles.com/p/articles/mi_m0MLW/is_3_3/ai_114495281/print)
but can't see the thread dump in catalina.out nor localhost_log....txt
files. What did I miss (this is Linux)?
If you could share your perl / grep / wc scripts, that would surely
jumpstart me and give me some ideas! Thanks in advance.
Cheers,
Guillaume
> -----Original Message-----
> From: Tim Funk [mailto:funkman@joedog.org]
> Sent: 15 March 2005 11:03
> To: Tomcat Users List
> Subject: Re: Over 1000 threads running in production
>
>
> Take periodic thread dumps and read them. perl, grep, and wc are
> your friends
> to parse the data without going insane.
>
> The thread dumps should give you a general idea of who created them.
>
> -Tim
>
> Guillaume Lahitette wrote:
> > Hello Tomcat'oids,
> >
> > This is Tomcat 4.1.26 on Linux with JDK 1.4.1_03, load balanced
> accross 2 servers by a BigIP device. User base is around 50.
> >
> > Our operations team discovered over 1000 java threads running
> on each production server. It appears that about 20 threads are added
> > to this count every hour.
> >
> > Can someone clarify:
> > a.. how are threads open? Is it 1 thread per HTTP request?
> Can I monitor this / gather more details?
> > b.. how and when are threads "closed"? garbage collected?
> > c.. are threads re-used / pooled?
> > d.. how to tune the number of threads Tomcat uses appropriately?
> > e.. Any known issues with our setup?
> > Any other recommendations, best practices and war stories are
> greatly appreciated!
> >
> > Many thanks in advance.
> > Guillaume
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org
Re: Over 1000 threads running in production
Posted by Tim Funk <fu...@joedog.org>.
Take periodic thread dumps and read them. perl, grep, and wc are your friends
to parse the data without going insane.
The thread dumps should give you a general idea of who created them.
-Tim
Guillaume Lahitette wrote:
> Hello Tomcat'oids,
>
> This is Tomcat 4.1.26 on Linux with JDK 1.4.1_03, load balanced accross 2 servers by a BigIP device. User base is around 50.
>
> Our operations team discovered over 1000 java threads running on each production server. It appears that about 20 threads are added
> to this count every hour.
>
> Can someone clarify:
> a.. how are threads open? Is it 1 thread per HTTP request? Can I monitor this / gather more details?
> b.. how and when are threads "closed"? garbage collected?
> c.. are threads re-used / pooled?
> d.. how to tune the number of threads Tomcat uses appropriately?
> e.. Any known issues with our setup?
> Any other recommendations, best practices and war stories are greatly appreciated!
>
> Many thanks in advance.
> Guillaume
>
---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org