You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Chetan Chheda <ch...@yahoo.com> on 2009/05/12 18:03:49 UTC
Tomcat not closing threads
Hello,
I am having an issue where I could use some clarity.
We are using apache 2.0.59 and tomcat 5.5. We are having some growing pains with vendor specified modjk and tomcat settings.
1. Tomcat opens up a number of threads during some transactions but does not close them. Is this because the connectionTimeout parameter is missing in the AJP connector defination?
2. I do have the cache_timeout (deprecated , I know ) set to 900 secs. What impact does that have on closing idle threads?
3. Today one of the Tomcat's froze and thru jkstatus I saw 409 connections open to it. If my connection pool size is 256 (from modjk.log) why are more connections being opened?
Thanks,
Chetan
My workers.properties file looks like this.
ps=/
worker.list=ajp13App31,ajp13App32,loadbalancer,mystatus
workers.tomcat_home=/coach/app/ptc/tomcat1
worker.ajp13App31.port=8012
worker.ajp13App31.host=localhost
worker.ajp13App31.type=ajp13
worker.ajp13App31.lbfactor=1
worker.ajp13App31.cache_timeout=900
worker.ajp13App31.socket_timeout=0
worker.ajp13App31.retries=3
worker.ajp13App31.recycle_timeout=900
worker.ajp13App32.port=8013
worker.ajp13App32.host=localhost
worker.ajp13App32.type=ajp13
worker.ajp13App32.lbfactor=1
worker.ajp13App32.cache_timeout=900
worker.ajp13App32.socket_timeout=0
worker.ajp13App32.retries=3
worker.ajp13App32.recycle_timeout=900
worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=ajp13App31,ajp13App32
worker.loadbalancer.method=Request
worker.loadbalancer.sticky_session=true
worker.inprocess.class_path=$(workers.tomcat_home)$(ps)lib$(ps)tomcat.jar
worker.inprocess.cmd_line=start
worker.inprocess.stdout=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stdout
worker.inprocess.stderr=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stderr
worker.mystatus.type=status
worker.mystatus.mount=/admin/status/jk
And here is the AJP13 connector from one of the tomcat's server.xml :
<Connector URIEncoding="UTF-8" acceptCount="100" debug="1" port="8012" protocol="AJP/1.3" protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler" redirectPort="8445" tomcatAuthentication="false" useBodyEncodingForURI="true">
Apache MPM
# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
ThreadLimit 512
StartServers 100
MaxClients 512
MinSpareThreads 129
MaxSpareThreads 256
ThreadsPerChild 256
MaxRequestsPerChild 0
</IfModule>
Re: Tomcat not closing threads
Posted by Rainer Jung <ra...@kippdata.de>.
On 18.05.2009 16:43, Chetan Chheda wrote:
> Our network team is already engaged and looking into this. We had
> several instances of tomcats locking up today. I have attached one such
> thread dump here ..
>
> Can some of you pls take a look and point me into the right direction?
There are 149 Tomcat AJP pool threads connected to the web server and
waiting for the next request in their connection (and one additional
thread waiting for the next connection).
I assume the size of your Tomcat AJP connection pool is 150 and so your
pool is exhausted. See my coming up answer to your other posts.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads
Posted by Rainer Jung <ra...@kippdata.de>.
On 18.05.2009 20:03, Chetan Chheda wrote:
> HTTPd.conf :
> Timeout 300
> KeepAlive On
> MaxKeepAliveRequests 100
> KeepAliveTimeout 30
HTTP Keep-Alive is nice to achieve lower latency, but it will drive up
the number of httpd threads you need. 100 requests and 20 seconds is for
most use cases to expensive (in terms of needed threads), I would
suggest a KeepAliveTimeout closer to e.g. 5 seconds.
> <IfModule worker.c>
> ThreadLimit 512
> StartServers 100
> MaxClients 512
> MinSpareThreads 129
> MaxSpareThreads 256
> ThreadsPerChild 256
> MaxRequestsPerChild 0
> </IfModule>
Apache scales via processes. You chose to size the number of thread per
process to a much higher value than default. That means whenever Apache
starts an additional process, you add a lot of additional threads at
once. That's not a very fine granular scaling. Furthermore the
combination of 100 processes to start with, each with 256 threads, but a
total limit of 512 parallel connections does not make sense. Read
carefully through the httpd documentation what these parameters mean and
how they interact. Also be sure to have anough delta between Min and Max
SpareThrads in terms of multiples of ThreadsPerChild. Otherwise you'll
be very often starting and stopping Apache processes.
>From now on I Assume you find better values, but still go for a maximum
concurrency of 512 http connections.
Which version of mod_jk do you use (strings mod_jk.so | grep 1.2)? Does
HP now provide a recent one? The below attribute names indicate a very
old mod_jk.
Comments below the lines.
> Workers.properties:
> njipuplmapp01:/opt/hpws/apache/conf#>cat workers.properties
> ps=/
Delete
> workers.tomcat_home=/coach/app/ptc/tomcat1
Delete
> worker.list=ajp13App11,ajp13App12,loadbalancer,mystatus
> worker.ajp13App11.port=8010
> worker.ajp13App11.host=localhost
> worker.ajp13App11.type=ajp13
> worker.ajp13App11.lbfactor=1
> worker.ajp13App11.cache_timeout=900
Read the docs page about timeouts and set your Tomcat connectionTimeout
consistently.
> worker.ajp13App11.socket_timeout=0
I personally don't like the socket_timeout, but much more cping/cpong
timeouts.
> worker.ajp13App11.retries=3
> worker.ajp13App11.recycle_timeout=900
>
> worker.ajp13App12.port=8011
> worker.ajp13App12.host=localhost
> worker.ajp13App12.type=ajp13
> worker.ajp13App12.lbfactor=1
> worker.ajp13App12.cache_timeout=900
> worker.ajp13App12.socket_timeout=0
> worker.ajp13App12.retries=3
> worker.ajp13App12.recycle_timeout=900
>
> worker.loadbalancer.type=lb
> worker.loadbalancer.balanced_workers=ajp13App11,ajp13App12
> worker.loadbalancer.method=Request
> worker.loadbalancer.sticky_session=true
>
> worker.inprocess.class_path=$(workers.tomcat_home)$(ps)lib$(ps)tomcat.jar
> worker.inprocess.cmd_line=start
> worker.inprocess.stdout=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stdout
> worker.inprocess.stderr=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stderr
Delete those three attributes.
> worker.mystatus.type=status
> worker.mystatus.mount=/admin/status/jk
>
> mod_jk
> # Configure mod_jk for basic operation
> <IfModule mod_jk.c>
> JkWorkersFile conf/workers.properties
> JkLogFile logs/mod_jk.log
> JkLogLevel info
> JkShmFile logs/JkShmFile.log
>
> # Must use this for proper % escaping (cannot use with Tomcat 3.2.x)
> JkOptions +ForwardURIEscaped
> # Uncomment the line below once HP builds with mod_jk 1.2.13 or
> # higher to ensure that the 'X-Do-not-compress-this' header
> # behaves as expected.
> #JkOptions +FlushPackets
>
> # Mount all JSP pages with our default worker
> JkMount /*.jsp loadbalancer
> JkMount /*.jsp/* loadbalancer
> </IfModule>
>
> Connector :
> <Connector acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true" port="8082" redirectPort="8445" maxSpareThreads="75" maxThreads="150" minSpareThreads="25">
> </Connector>
> <Connector URIEncoding="UTF-8" acceptCount="100" debug="1" port="8012" protocol="AJP/1.3" protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler" redirectPort="8445" tomcatAuthentication="false" useBodyEncodingForURI="true">
Set number of threads (you'll need a maximum of 512, if your web servers
are allowed with 512 connections forwarding to Tomcat)
and also connectionTimeout!
> </Connector>
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads
Posted by Chetan Chheda <ch...@yahoo.com>.
HTTPd.conf :
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 30
<IfModule worker.c>
ThreadLimit 512
StartServers 100
MaxClients 512
MinSpareThreads 129
MaxSpareThreads 256
ThreadsPerChild 256
MaxRequestsPerChild 0
</IfModule>
Workers.properties:
njipuplmapp01:/opt/hpws/apache/conf#>cat workers.properties
ps=/
workers.tomcat_home=/coach/app/ptc/tomcat1
worker.list=ajp13App11,ajp13App12,loadbalancer,mystatus
worker.ajp13App11.port=8010
worker.ajp13App11.host=localhost
worker.ajp13App11.type=ajp13
worker.ajp13App11.lbfactor=1
worker.ajp13App11.cache_timeout=900
worker.ajp13App11.socket_timeout=0
worker.ajp13App11.retries=3
worker.ajp13App11.recycle_timeout=900
worker.ajp13App12.port=8011
worker.ajp13App12.host=localhost
worker.ajp13App12.type=ajp13
worker.ajp13App12.lbfactor=1
worker.ajp13App12.cache_timeout=900
worker.ajp13App12.socket_timeout=0
worker.ajp13App12.retries=3
worker.ajp13App12.recycle_timeout=900
worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=ajp13App11,ajp13App12
worker.loadbalancer.method=Request
worker.loadbalancer.sticky_session=true
worker.inprocess.class_path=$(workers.tomcat_home)$(ps)lib$(ps)tomcat.jar
worker.inprocess.cmd_line=start
worker.inprocess.stdout=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stdout
worker.inprocess.stderr=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stderr
worker.mystatus.type=status
worker.mystatus.mount=/admin/status/jk
mod_jk
# Configure mod_jk for basic operation
<IfModule mod_jk.c>
JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel info
JkShmFile logs/JkShmFile.log
# Must use this for proper % escaping (cannot use with Tomcat 3.2.x)
JkOptions +ForwardURIEscaped
# Uncomment the line below once HP builds with mod_jk 1.2.13 or
# higher to ensure that the 'X-Do-not-compress-this' header
# behaves as expected.
#JkOptions +FlushPackets
# Mount all JSP pages with our default worker
JkMount /*.jsp loadbalancer
JkMount /*.jsp/* loadbalancer
</IfModule>
Connector :
<Connector acceptCount="100" connectionTimeout="20000" disableUploadTimeout="true" port="8082" redirectPort="8445" maxSpareThreads="75" maxThreads="150" minSpareThreads="25">
</Connector>
<Connector URIEncoding="UTF-8" acceptCount="100" debug="1" port="8012" protocol="AJP/1.3" protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler" redirectPort="8445" tomcatAuthentication="false" useBodyEncodingForURI="true">
</Connector>
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Monday, May 18, 2009 12:18:22 PM
Subject: Re: Tomcat not closing threads
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chetan,
On 5/18/2009 10:43 AM, Chetan Chheda wrote:
> Our network team is already engaged and looking into this. We had
> several instances of tomcats locking up today. I have attached one such
> thread dump here ..
It looks like all your request processors are waiting for data to be
read from the client. Can you post your current:
workers.properties
httpd.conf (portions relevant to the worker MPM and mod_jk)
<Connector>
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkoRik4ACgkQ9CaO5/Lv0PDktACfZRY7Eu5R/0nvnyGdOiXD2tID
9RQAoKCgluYtV8ht5/DAvWc1nP4b+yeQ
=UycA
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chetan,
On 5/18/2009 10:43 AM, Chetan Chheda wrote:
> Our network team is already engaged and looking into this. We had
> several instances of tomcats locking up today. I have attached one such
> thread dump here ..
It looks like all your request processors are waiting for data to be
read from the client. Can you post your current:
workers.properties
httpd.conf (portions relevant to the worker MPM and mod_jk)
<Connector>
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkoRik4ACgkQ9CaO5/Lv0PDktACfZRY7Eu5R/0nvnyGdOiXD2tID
9RQAoKCgluYtV8ht5/DAvWc1nP4b+yeQ
=UycA
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads -- SOLVED ??
Posted by Chetan Chheda <ch...@yahoo.com>.
This is a vendor supported architecture. They deliver a set of pre-packaged Apache conf files. So apache is a part of their design.
On our side, we have multiple httpd's load balancing to multiple tomcats. Also the application is AD integrated and some paths of access lie in the DMZ where only a web tier can reside.
Chetan
________________________________
From: "Caldarale, Charles R" <Ch...@unisys.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Wednesday, May 27, 2009 11:38:11 AM
Subject: RE: Tomcat not closing threads -- SOLVED ??
> From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> Subject: Re: Tomcat not closing threads -- SOLVED ??
>
> Pardon my ignorance .. but browsing thru the application directories I
> found a number of other file types like css, javascript, ser etc and
> wasnt sure whether these need to go to tomcat or not. Hence -> JkMount
> /LCSW/* ajp13
Let's go back to another of André's questions: why are you using httpd at all? Can you simplify your life by simply removing it and letting Tomcat handle everything directly?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat not closing threads -- SOLVED ??
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> Subject: Re: Tomcat not closing threads -- SOLVED ??
>
> Pardon my ignorance .. but browsing thru the application directories I
> found a number of other file types like css, javascript, ser etc and
> wasnt sure whether these need to go to tomcat or not. Hence -> JkMount
> /LCSW/* ajp13
Let's go back to another of André's questions: why are you using httpd at all? Can you simplify your life by simply removing it and letting Tomcat handle everything directly?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads -- SOLVED ??
Posted by Chetan Chheda <ch...@yahoo.com>.
Pardon my ignorance .. but browsing thru the application directories I found a number of other file types like css, javascript, ser etc and wasnt sure whether these need to go to tomcat or not. Hence -> JkMount /LCSW/* ajp13
So I went with with I am sure about, which is image files and html pages. Nobody in my shop knows this side of the application and vendor support is terrible.
________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Wednesday, May 27, 2009 10:31:20 AM
Subject: Re: Tomcat not closing threads -- SOLVED ??
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chetan,
On 5/26/2009 2:47 PM, Chetan Chheda wrote:
> After some digging thru config files setup by the vendor, I think I
> might have found the root cause ..Correlating the access_log and tomcat
> logs, I found out that tomcat threads were shooting up whenever a large
> number of GIF files were being requested.
>
> These are the modjk settings in our application config files that
> are included in httpd.conf
> JkMount /LCSW/* ajp13
> JkMount /LCSW/*.jsp ajp13
> JkMount /LCSW/*.jsp/* ajp13
As André points out, the second and third line are superfluous, given
the first line.
> So when an image is accessed via the following URL,
> http://blahbhal/LCSW/images/header.gif , the above routes were sending
> it to tomcat. I verified this by shutting down tomcat and accessing the
> above URL, and was unsuccessful.
>
> Now I plan to add the following excludes
> JkUnMount /*.htm ajp13
> JkUnMount /*.html ajp13
> JkUnMount /*.png ajp13
> JkUnMount /*.gif ajp13
> JkUnMount /*.jpg ajp13
Instead of including everything, then excluding certain things, why not
just include /only/ the things you /actually/ want Tomcat to handle.
Something like:
JkMount /LCSW/*.jsp ajp13
> I tested the above in a test environment and was able to access a
> gif file with tomcat down.
Try the above. I think you'll like it better, since you won't have to
add exceptions for every type of static file you want to serve.
> my question is, how does the ajp13 connector interpret a gif
> file/static content request as opposed to a jsp request?
mod_jk does not do any interpretation at all. The JkMount directives
just set up a table of URL pattern matchers. No content is analyzed or
anything like that. In combination with another module, you might be
able to sniff some of the /request/ and then conditionally invoke Tomcat
(by using André's favorite "SetHandler jakarta-servlet" trick) but that
would be considered super-human efforts to do something that ought to be
simple.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkodTrgACgkQ9CaO5/Lv0PAZLACgwBc8vKC66xVRYrd+HJoYtmcf
sKwAn0bx7DlKzROMeIV0zhjQ/VSXB0zc
=Sxs5
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads -- SOLVED ??
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chetan,
On 5/26/2009 2:47 PM, Chetan Chheda wrote:
> After some digging thru config files setup by the vendor, I think I
> might have found the root cause ..Correlating the access_log and tomcat
> logs, I found out that tomcat threads were shooting up whenever a large
> number of GIF files were being requested.
>
> These are the modjk settings in our application config files that
> are included in httpd.conf
> JkMount /LCSW/* ajp13
> JkMount /LCSW/*.jsp ajp13
> JkMount /LCSW/*.jsp/* ajp13
As André points out, the second and third line are superfluous, given
the first line.
> So when an image is accessed via the following URL,
> http://blahbhal/LCSW/images/header.gif , the above routes were sending
> it to tomcat. I verified this by shutting down tomcat and accessing the
> above URL, and was unsuccessful.
>
> Now I plan to add the following excludes
> JkUnMount /*.htm ajp13
> JkUnMount /*.html ajp13
> JkUnMount /*.png ajp13
> JkUnMount /*.gif ajp13
> JkUnMount /*.jpg ajp13
Instead of including everything, then excluding certain things, why not
just include /only/ the things you /actually/ want Tomcat to handle.
Something like:
JkMount /LCSW/*.jsp ajp13
> I tested the above in a test environment and was able to access a
> gif file with tomcat down.
Try the above. I think you'll like it better, since you won't have to
add exceptions for every type of static file you want to serve.
> my question is, how does the ajp13 connector interpret a gif
> file/static content request as opposed to a jsp request?
mod_jk does not do any interpretation at all. The JkMount directives
just set up a table of URL pattern matchers. No content is analyzed or
anything like that. In combination with another module, you might be
able to sniff some of the /request/ and then conditionally invoke Tomcat
(by using André's favorite "SetHandler jakarta-servlet" trick) but that
would be considered super-human efforts to do something that ought to be
simple.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkodTrgACgkQ9CaO5/Lv0PAZLACgwBc8vKC66xVRYrd+HJoYtmcf
sKwAn0bx7DlKzROMeIV0zhjQ/VSXB0zc
=Sxs5
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads -- SOLVED ??
Posted by André Warnier <aw...@ice-sa.com>.
Chetan Chheda wrote:
> Great thanks! I will test out the alternate method ...
>
> What I meant when I asked about gif files and tomcat was basically is it an expensive transaction for tomcat ajp13 vs apache vs tomcat's http connector? I guess in other words I am trying to assess the impact my proposed changes in our environment.
>
Maybe you should look up the thread, on this same list, entitled "Apache
httpd vs Tomcat static content performance".
The first question you should be asking yourself is, if you only serve
some static content + java stuff, do you need Apache httpd and mod_jk in
front, or can you use just Tomcat directly as a httpd server ?
Tomcat and Apache are in many scenarios very close in performance when
it comes to serving static content, and you would spare yourself some
complication and overhead by using just Tomcat.
If you do need to use Apache httpd for some reason (and there may be
valid reasons), then the overhead of mod_jk itself is probably not going
to kill you. It will add mere fractional milliseconds to the whole cycle.
There are a lot of threads on this list that discuss the same subject
generally.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads -- SOLVED ??
Posted by Chetan Chheda <ch...@yahoo.com>.
Great thanks! I will test out the alternate method ...
What I meant when I asked about gif files and tomcat was basically is it an expensive transaction for tomcat ajp13 vs apache vs tomcat's http connector? I guess in other words I am trying to assess the impact my proposed changes in our environment.
________________________________
From: André Warnier <aw...@ice-sa.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Tuesday, May 26, 2009 3:20:06 PM
Subject: Re: Tomcat not closing threads -- SOLVED ??
André Warnier wrote:
> Chetan Chheda wrote:
>> After some digging thru config files setup by the vendor, I think I might have found the root cause ..Correlating the access_log and tomcat logs, I found out that tomcat threads were shooting up whenever a large number of GIF files were being requested.
>> These are the modjk settings in our application config files that are included in httpd.conf JkMount /LCSW/* ajp13 JkMount /LCSW/*.jsp ajp13
>> JkMount /LCSW/*.jsp/* ajp13
>
> The above seems totally redundant to me. The first JkMount will "capture" anything starting with /LCSW/, no ?
> If yes, then the following two are totally redundant.
>
>> So when an image is accessed via the following URL, http://blahbhal/LCSW/images/header.gif , the above routes were sending it to tomcat. I verified this by shutting down tomcat and accessing the above URL, and was unsuccessful.
>> Now I plan to add the following excludes JkUnMount /*.htm ajp13
>> JkUnMount /*.html ajp13
>> JkUnMount /*.png ajp13
>> JkUnMount /*.gif ajp13
>> JkUnMount /*.jpg ajp13
>
> That will work.
>
>>
>> I tested the above in a test environment and was able to access a gif file with tomcat down.
>> my question is, how does the ajp13 connector interpret a gif file/static content request as opposed to a jsp request?
> As far as I know, it does not see the difference, nor care about it. It only takes into account the match/unmatch of the request URL. It has no idea of what "static" or "dynamic" content is.
>
> Separately :
>
> For the kind of thing you are doing above, there exists an alternative syntax to the JkMount/JkUnMount pair, which I personally prefer because I find that it better "fits" in the Apache configuration logic.
>
> Here is an example :
>
> <Location /LCSW>
> SetHandler jakarta-servlet
> SetEnvIf REQUEST_URI \.(html?|png|gif|jpg)$ no-jk
> ...
> </Location>
>
> (You can of course easily insert other Apache statements inside the same Location block, for example authentication etc..
>
> You find this toward the end of this documentation page :
> http://tomcat.apache.org/connectors-doc/reference/apache.html
>
> The "SetHandler jakarta-servlet" does essentially the same as the JkMount, for everything that fits under /LCSW.
> The SetEnvIf then selectively sets the Apache environment variable "no-jk" (here, whenever the request URI ends in one of the listed extensions).
> The mod_jk module is always called (because it is the content handler for this section), but it declines to process this URL because the no-jk variable is set.
> Apache thus processes the request with its own default handler, which serves the content.
>
> To see exactly what happens, step by step, set the JkLogLevel to "debug", make a request, and look at the jk logfile.
>
>
Erratum :
replace
<Location /LCSW>
by
<Location /LCSW/>
in the above. Otherwise, it would also match /LCSW1/ for instance.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads -- SOLVED ??
Posted by André Warnier <aw...@ice-sa.com>.
André Warnier wrote:
> Chetan Chheda wrote:
>> After some digging thru config files setup by the vendor, I think I
>> might have found the root cause ..Correlating the access_log and
>> tomcat logs, I found out that tomcat threads were shooting up whenever
>> a large number of GIF files were being requested.
>> These are the modjk settings in our application config files that are
>> included in httpd.conf JkMount /LCSW/* ajp13 JkMount /LCSW/*.jsp ajp13
>> JkMount /LCSW/*.jsp/* ajp13
>
> The above seems totally redundant to me. The first JkMount will
> "capture" anything starting with /LCSW/, no ?
> If yes, then the following two are totally redundant.
>
>>
>> So when an image is accessed via the following URL,
>> http://blahbhal/LCSW/images/header.gif , the above routes were sending
>> it to tomcat. I verified this by shutting down tomcat and accessing
>> the above URL, and was unsuccessful.
>>
>> Now I plan to add the following excludes JkUnMount /*.htm ajp13
>> JkUnMount /*.html ajp13
>> JkUnMount /*.png ajp13
>> JkUnMount /*.gif ajp13
>> JkUnMount /*.jpg ajp13
>
> That will work.
>
>>
>> I tested the above in a test environment and was able to access a gif
>> file with tomcat down.
>> my question is, how does the ajp13 connector interpret a gif
>> file/static content request as opposed to a jsp request?
> As far as I know, it does not see the difference, nor care about it. It
> only takes into account the match/unmatch of the request URL. It has no
> idea of what "static" or "dynamic" content is.
>
> Separately :
>
> For the kind of thing you are doing above, there exists an alternative
> syntax to the JkMount/JkUnMount pair, which I personally prefer because
> I find that it better "fits" in the Apache configuration logic.
>
> Here is an example :
>
> <Location /LCSW>
> SetHandler jakarta-servlet
> SetEnvIf REQUEST_URI \.(html?|png|gif|jpg)$ no-jk
> ...
> </Location>
>
> (You can of course easily insert other Apache statements inside the same
> Location block, for example authentication etc..
>
> You find this toward the end of this documentation page :
> http://tomcat.apache.org/connectors-doc/reference/apache.html
>
> The "SetHandler jakarta-servlet" does essentially the same as the
> JkMount, for everything that fits under /LCSW.
> The SetEnvIf then selectively sets the Apache environment variable
> "no-jk" (here, whenever the request URI ends in one of the listed
> extensions).
> The mod_jk module is always called (because it is the content handler
> for this section), but it declines to process this URL because the no-jk
> variable is set.
> Apache thus processes the request with its own default handler, which
> serves the content.
>
> To see exactly what happens, step by step, set the JkLogLevel to
> "debug", make a request, and look at the jk logfile.
>
>
Erratum :
replace
<Location /LCSW>
by
<Location /LCSW/>
in the above. Otherwise, it would also match /LCSW1/ for instance.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads -- SOLVED ??
Posted by André Warnier <aw...@ice-sa.com>.
Chetan Chheda wrote:
> After some digging thru config files setup by the vendor, I think I might have found the root cause ..Correlating the access_log and tomcat logs, I found out that tomcat threads were shooting up whenever a large number of GIF files were being requested.
>
> These are the modjk settings in our application config files that are included in httpd.conf
> JkMount /LCSW/* ajp13
> JkMount /LCSW/*.jsp ajp13
> JkMount /LCSW/*.jsp/* ajp13
The above seems totally redundant to me. The first JkMount will
"capture" anything starting with /LCSW/, no ?
If yes, then the following two are totally redundant.
>
> So when an image is accessed via the following URL, http://blahbhal/LCSW/images/header.gif , the above routes were sending it to tomcat. I verified this by shutting down tomcat and accessing the above URL, and was unsuccessful.
>
> Now I plan to add the following excludes
> JkUnMount /*.htm ajp13
> JkUnMount /*.html ajp13
> JkUnMount /*.png ajp13
> JkUnMount /*.gif ajp13
> JkUnMount /*.jpg ajp13
That will work.
>
> I tested the above in a test environment and was able to access a gif file with tomcat down.
>
> my question is, how does the ajp13 connector interpret a gif file/static content request as opposed to a jsp request?
>
As far as I know, it does not see the difference, nor care about it. It
only takes into account the match/unmatch of the request URL. It has no
idea of what "static" or "dynamic" content is.
Separately :
For the kind of thing you are doing above, there exists an alternative
syntax to the JkMount/JkUnMount pair, which I personally prefer because
I find that it better "fits" in the Apache configuration logic.
Here is an example :
<Location /LCSW>
SetHandler jakarta-servlet
SetEnvIf REQUEST_URI \.(html?|png|gif|jpg)$ no-jk
...
</Location>
(You can of course easily insert other Apache statements inside the same
Location block, for example authentication etc..
You find this toward the end of this documentation page :
http://tomcat.apache.org/connectors-doc/reference/apache.html
The "SetHandler jakarta-servlet" does essentially the same as the
JkMount, for everything that fits under /LCSW.
The SetEnvIf then selectively sets the Apache environment variable
"no-jk" (here, whenever the request URI ends in one of the listed
extensions).
The mod_jk module is always called (because it is the content handler
for this section), but it declines to process this URL because the no-jk
variable is set.
Apache thus processes the request with its own default handler, which
serves the content.
To see exactly what happens, step by step, set the JkLogLevel to
"debug", make a request, and look at the jk logfile.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads -- SOLVED ??
Posted by Chetan Chheda <ch...@yahoo.com>.
After some digging thru config files setup by the vendor, I think I might have found the root cause ..Correlating the access_log and tomcat logs, I found out that tomcat threads were shooting up whenever a large number of GIF files were being requested.
These are the modjk settings in our application config files that are included in httpd.conf
JkMount /LCSW/* ajp13
JkMount /LCSW/*.jsp ajp13
JkMount /LCSW/*.jsp/* ajp13
So when an image is accessed via the following URL, http://blahbhal/LCSW/images/header.gif , the above routes were sending it to tomcat. I verified this by shutting down tomcat and accessing the above URL, and was unsuccessful.
Now I plan to add the following excludes
JkUnMount /*.htm ajp13
JkUnMount /*.html ajp13
JkUnMount /*.png ajp13
JkUnMount /*.gif ajp13
JkUnMount /*.jpg ajp13
I tested the above in a test environment and was able to access a gif file with tomcat down.
my question is, how does the ajp13 connector interpret a gif file/static content request as opposed to a jsp request?
Chetan
________________________________
From: Chetan Chheda <ch...@yahoo.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Monday, May 18, 2009 5:53:07 PM
Subject: Re: Tomcat not closing threads
Hi martin,
Thanks for your reply but you have left me confused
Since Apache is setup as below
Maxclients = 512
ThreadsPerChild = 256
Wont it restrict the processes to 2 processes with 256 threads each instead of the ServerLimit(16) ? Looking at the OS , I see 2 httpd processes each with 258 threads
Thanks,
Chetan
________________________________
From: Martin Gainty <mg...@hotmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Monday, May 18, 2009 4:00:16 PM
Subject: RE: Tomcat not closing threads
good to know
With worker mpm the connection_pool_size defaults to ThreadsPerChild
ThreadsPerChild that in the final gives the MaxClients connections to the Tomcat.
So the default value for connection_pool_size is always ThreadsPerChild
For threaded and hybrid servers (e.g. beos
or worker) MaxClients restricts
the total number of threads that will be available to serve clients.
The default value for beos is 50.
For
hybrid MPMs the default value is 16 (ServerLimit) multiplied by the value of
25 (ThreadsPerChild). Therefore, to increase MaxClients to a value that requires more than 16 processes,
you must also raise ServerLimit.
/*******in your case*******/
(MaxClients)MaxClients=16(ServerLimit)*256(ThreadsPerChild)=4096
onto thread calculation:
Description:Minimum number of idle threads available to handle request
spikes
Syntax:MinSpareThreads number
Default:See usage for details
Context:server config
Status:MPM
Module:beos, mpm_netware, mpmt_os2, worker
Minimum number of idle threads to handle request spikes.
Different MPMs deal with this directive
differently.
worker
uses a default of MinSpareThreads
75 and deal with idle threads on a server-wide basis. If
there aren't enough idle threads in the server then child
processes are created until the number of idle threads is greater
than number.
mpm_netware uses a default of
MinSpareThreads 10 and, since it is a single-process
MPM, tracks this on a server-wide bases.
beos and mpmt_os2 work
similar to mpm_netware. The default for
beos is MinSpareThreads 1. For
mpmt_os2 the default value is 5.
/*********lets assume you havent touched this and MinSpareThreads stays at 75***********/
MaxSpareThreads(this is the parameter that is plaguing your environment)
The range of the MaxSpareThreads value
is restricted. Apache will correct the given value automatically
according to the following rules:
perchild requires MaxSpareThreads to be less or equal than ThreadLimit.mpm_netware wants the value to be greater than
MinSpareThreads.For leader, threadpool and
worker the value must be greater or equal than
the sum of MinSpareThreads
and ThreadsPerChild.
MaxSpareThreads >= MinSpareThreads+ThreadsPerChild
MaxSpareThreads >=75+256
MaxSpareThreads must be >= 331
after all that if you still dont have enough SpareThreads i would up (master thread limit) ThreadLimit to the max
The default value for ThreadLimit is
1920 when used with mpm_winnt and
64 when used with the others.
There is a hard limit of ThreadLimit 20000 (or
ThreadLimit 15000 with mpm_winnt)
compiled into the server. This is intended to avoid nasty effects
caused by typos.
/****conceivable to start with 1920 and increase by increments of 500 until number of threads sufficient ***/
BTW: the ceiling on ThreadLimit=15000
In your case
ThreadLimit=1920
HTH
Martin Gainty
______________________________________________
Jogi és Bizalmassági kinyilatkoztatás/Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Ez az
üzenet bizalmas. Ha nem ön az akinek szánva volt, akkor kérjük, hogy
jelentse azt nekünk vissza. Semmiféle továbbítása vagy másolatának
készítése nem megengedett. Ez az üzenet csak ismeret cserét szolgál és
semmiféle jogi alkalmazhatósága sincs. Mivel az electronikus üzenetek
könnyen megváltoztathatóak, ezért minket semmi felelöség nem terhelhet
ezen üzenet tartalma miatt.
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> Date: Mon, 18 May 2009 12:03:35 -0700
> From: chetan_chheda@yahoo.com
> Subject: Re: Tomcat not closing threads
> To: users@tomcat.apache.org
>
> Apache is using worker mpm
> Server version: Apache/2.0.59 HP-UX_Apache-based_Web_Server
> Server built: Aug 21 2007 13:59:06
> Server's Module Magic Number: 20020903:12
> Server loaded: APR 0.9.12, APR-UTIL 0.9.12
> Compiled using: APR 0.9.12, APR-UTIL 0.9.12
> Architecture: 64-bit
> Server compiled with....
> -D APACHE_MPM_DIR="server/mpm/worker"
>
>
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Monday, May 18, 2009 2:44:45 PM
> Subject: RE: Tomcat not closing threads
>
>
> need to know if Apache MPM prefork or Apache worker is configured
> $APACHE_HOME/bin/Apache -l
>
> *any compiled-in modules with 'mpm' in the module-name is prefork*
> *any compiled-in modules with 'worker' in the module-name is worker*
>
> Martin Gainty
> ______________________________________________
> Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
>
> Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
> Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
>
>
>
>
> > Date: Mon, 18 May 2009 08:45:43 -0700
> > From: chetan_chheda@yahoo.com
> > Subject: Re: Tomcat not closing threads
> > To: users@tomcat.apache.org
> >
> > So what you are suggesting is my mod_jk and tomcat parameters in server.xml are messed up ..
> >
> > Here is my current setup I have one apache load balancing to 2 tomcat workers
> > Maxclients = 512
> > ThreadsPerChild = 256
> > Connection_pool_size = 256
> > Tomcat MaxThreads = 150
> >
> > I will suggest my team to change this to ...
> > connection_pool_size = 120 - which means that I will have 120*2(2 apache server procs) = 240 threads per tomcat
> > Tomcat maxthreads = 300
> >
> > So out of total 512 worker threads , a max of 480 threads will be used by mod_jk leaving the rest for serving static content.
> >
> > is my math correct?
> >
> > Thanks!
> >
> >
> >
> > ________________________________
> > From: Martin Gainty <mg...@hotmail.com>
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Sent: Monday, May 18, 2009 11:23:58 AM
> > Subject: RE: Tomcat not closing threads
> >
> >
> > some documentation:
> >
> > http://httpd.apache.org/docs/2.0/mod/mpm_common.html
> > MaxClients:
> > The MaxClients directive sets the limit on the number of simultaneous requests that will be served.
> > Any connection attempts over the MaxClients limit will normally be queued, up to a number based on the ListenBacklog directive.
> > Once a child process is freed at the end of a different request, the connection will then be serviced.
> > For non-threaded servers (i.e., prefork), MaxClients translates into the maximum number of child processes that will be launched to serve requests. The default value is 256; to increase it, you must also raise ServerLimit.
> > For threaded and hybrid servers (e.g. beos or worker) MaxClients restricts the total number of threads that will be available to serve clients.
> > The default value for beos is 50.
> > For hybrid MPMs (your situation) the default value is 16 (ServerLimit) multiplied by the value of ThreadsPerChild.
> > Therefore, to increase MaxClients to a value that requires more than 16 processes, you must also raise ServerLimit.
> >
> > ThreadsPerChild
> > This directive sets the number of threads created by each child process.
> > The child creates these threads at startup and never creates more.
> > If using an MPM like mpm_winnt, where there is only one child process, this number should be high enough to handle the entire load of the server.
> > If using an MPM like worker, where there are multiple child processes, the total number of threads should be high enough to handle the common load on the server.
> > The default value for ThreadsPerChild is 64 when used with mpm_winnt
> > The default value is 25 when used with other configurations such as worker
> >
> > also from mladens discussion of connection_pool_size calcs based on Apache HTTPD Prefork vs Worker
> > http://www.mail-archive.com/users@tomcat.apache.org/msg14767.html
> > Apache Prefork configuration
> > Apache 1.3 and Apache 2.x-prefork create a separate child process for
> > each client connection, meaning that if you have default
> > 250 MaxClients, you can end up with MaxClients * connection_pool_size
> > connections to the Tomcat, that would in your case be 250000.
> > (and in the case of ThreadsPerChild parameter MaxClients*64*connection_pool_size)
> >
> > Apache Worker configuration
> > With worker mpm the connection_pool_size defaults to ThreadsPerChild,
> > that in the final gives the MaxClients connections to the Tomcat.
> > So the default value for connection_pool_size is always ThreadsPerChild
> > that with prefork mpm is by design always 1.
> > (in the case of ThreadsPerChild parameter not being null MaxClients*ThreadsPerChild)
> >
> > HTH
> > Martin
> > ______________________________________________
> > Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
> >
> > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
> > Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> >
> >
> >
> >
> > Date: Mon, 18 May 2009 07:43:04 -0700
> > From: chetan_chheda@yahoo.com
> > Subject: Re: Tomcat not closing threads
> > To: users@tomcat.apache.org
> >
> >
> >
> > Our network team is already engaged and looking into this. We had several instances of tomcats locking up today. I have attached one such thread dump here ..
> >
> > Can some of you pls take a look and point me into the right direction?
> >
> >
> >
> >
> >
> >
> > From: rad muthu <rm...@gmail.com>
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Sent: Wednesday, May 13, 2009 8:05:08 PM
> > Subject: Re: Tomcat not closing threads
> >
> > Chetan,
> >
> >
> > We had similar issue with Sun solaris and weblogic.
> >
> > My system admin found out there is a socket read errors are going on . He
> > used snoop command in solaris. After tuning the TCP parameters the issue got
> > resolved.
> >
> > You might also need to something like that.
> >
> > Thanks
> > Krish
> >
> >
> >
> > On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda <ch...@yahoo.com>wrote:
> >
> > > I guess I need to ask my question again. Im primarily an
> > HP-UX
> > > administrator and recently inherited this web based application ...
> > >
> > > We are having some issues in accomodating additional user
> > > load/functionality. While a part of the team is looking at optimizing code,
> > > I am responsible for infrastructure componants.
> > >
> > > One particular focus area for me is the mod_jk and tomcat configurations.
> > > 1. Last of April saw one of the tomcat's stop processing requests. Looking
> > > at the mod_jk.log I saw the following errors
> > > Unable to get the free endpoint for worker XXX from 37 slots ....
> > > I have setup the status servlet for mod_jk in which I saw that max slots
> > > were opened. Thinking that mod_jk needs more number of connections, I
> > > basically set it to ThreadsPerChild from worker MPM
> > >
> > > 2. Now we are randomly seeing that even a simple activity as user login
> > > causes a spike in used tomcat connector threads.
> > Eventually all tomcat
> > > threads are used and no new requests can be serviced.
> > >
> > > So, long story short, is my apache, mod_jk and tomcat configuration in
> > > sync?
> > > Maxclients = 512
> > > ThreadsPerChild = 256
> > > So that means 2 server processes with 256 threads each.
> > >
> > > I have 2 worker threads in my worker.properties file with
> > > connection_pool_size=256.
> > > Does this mean that total number of connections into tomcat = 256 *
> > > 2(number of workers) * 2(number of server procs) = 1024 ??
> > >
> > > Do I need to change the connection_pool_size to 120 . Which means 120*2 *2
> > > = 480 and that leaves the remainder of apache threads for static content?
> > >
> > > 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in
> > > server.xml . Is this why connections remain open?
> > >
> > > Thanks,
> > > Chetan
> > >
> > >
> > >
> > ________________________________
> > > From: "Caldarale, Charles R" <Ch...@unisys.com>
> > > To: Tomcat Users List <us...@tomcat.apache.org>
> > > Sent: Tuesday, May 12, 2009 10:47:00 PM
> > > Subject: RE: Tomcat not closing threads
> > >
> > > > From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> > > > Subject: Re: Tomcat not closing threads
> > > >
> > > > I managed to get a thread dump during one such tomcat hangs. Most of
> > > > the threads are in the following status ...
> > >
> > > The ones you show are simply waiting for input from httpd; that's a pretty
> > > normal state. When looking at a thread dump, it's often
> > the oddball thread
> > > that's not doing what the rest are that is causing a problem. However, if
> > > all of the threads are just waiting to receive something from httpd, then
> > > you may have to look outside of Tomcat for the cause.
> > >
> > > - Chuck
> > >
> > >
> > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> > > MATERIAL and is thus for use only by the intended recipient. If you received
> > > this in error, please contact the sender and delete the e-mail and its
> > > attachments from all computers.
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > > For additional commands, e-mail: users-help@tomcat.apache.org
> > >
> > >
> > >
> > >
> >
> >
> > _________________________________________________________________
> > Windows Live™: Keep your life in sync.
> > http://windowslive.com/explore?ocid=TXT_TAGLM_BR_life_in_synch_052009
> >
> >
> >
>
> _________________________________________________________________
> Hotmail® goes with you.
> http://windowslive.com/Tutorial/Hotmail/Mobile?ocid=TXT_TAGLM_WL_HM_Tutorial_Mobile1_052009
>
>
>
_________________________________________________________________
Hotmail® has ever-growing storage! Don’t worry about storage limits.
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutorial_Storage1_052009
RE: JNDI
Posted by Martin Gainty <mg...@hotmail.com>.
http://www.mbaworld.com/docs/realm-howto.html#JNDIRealm
When a standard realm authenticates by retrieving the stored
password and comparing it with the value presented by the user, you
can select digested passwords by specifying the digest
attribute on your <Realm> element. The value for
this attribute must be one of the digest algorithms supported by the
java.security.MessageDigest class (SHA, MD2, or MD5).
When you select this option, the contents of the password that is
stored in the Realm must be the cleartext version of the
password, as digested by the specified algorithm.
MD5 is not yet supported in JNDIRealm
http://www.devdaily.com/java/jwarehouse/apache-tomcat-6.0.16/java/org/apache/catalina/realm/JNDIRealm.java.shtml
Similarly the MessageDigest has not yet implemented MD5
http://www.docjar.com/html/api/java/security/MessageDigest.java.html
you could have the MD5 functions added If you place your request into Bugzilla
bugzilla-admin@apache.org
Martin Gainty
______________________________________________
Jogi és Bizalmassági kinyilatkoztatás/Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Ez az
üzenet bizalmas. Ha nem ön az akinek szánva volt, akkor kérjük, hogy
jelentse azt nekünk vissza. Semmiféle továbbítása vagy másolatának
készítése nem megengedett. Ez az üzenet csak ismeret cserét szolgál és
semmiféle jogi alkalmazhatósága sincs. Mivel az electronikus üzenetek
könnyen megváltoztathatóak, ezért minket semmi felelöség nem terhelhet
ezen üzenet tartalma miatt.
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> Date: Mon, 18 May 2009 16:19:53 -0700
> From: secc77@yahoo.com
> Subject: JNDI
> To: users@tomcat.apache.org
>
> hi all,
> i am pretty sure this has been hashed out many times but i could not get clear guidelines on how to accomplish JNDI Realm + tomcat 6 + windows AD.
>
> I have got JNDI Realm + tomcat 6 + windows AD integration with basic authentication. as domain password is passed in clear text from browser to tomcat server, i would like to use digest http authentication. But digest auth is one way hash and tomcat is unable to authenticate the users against windows AD.
>
> Windows AD doesn't give out password ( at least from search) and so can't use comparitive mode in JNDI look up.
>
> So how do send user password from browser to tomcat server hashed such a way that i could authenticate against windows AD?
>
> One way i was thinking of MD5 the pass word on browser and tomcat authenticates with windows AD
> with MD5 scheme. So is there any setting that suggests to tomcat that do not Md5 the password but inform the LDAP provider the auth scheme is http://www.mbaworld.com/docs/realm-howto.html#JNDIRealm.
>
> thanks
> d.
>
>
>
>
_________________________________________________________________
Windows Live™: Keep your life in sync.
http://windowslive.com/explore?ocid=TXT_TAGLM_BR_life_in_synch_052009
JNDI
Posted by daniel steel <se...@yahoo.com>.
hi all,
i am pretty sure this has been hashed out many times but i could not get clear guidelines on how to accomplish JNDI Realm + tomcat 6 + windows AD.
I have got JNDI Realm + tomcat 6 + windows AD integration with basic authentication. as domain password is passed in clear text from browser to tomcat server, i would like to use digest http authentication. But digest auth is one way hash and tomcat is unable to authenticate the users against windows AD.
Windows AD doesn't give out password ( at least from search) and so can't use comparitive mode in JNDI look up.
So how do send user password from browser to tomcat server hashed such a way that i could authenticate against windows AD?
One way i was thinking of MD5 the pass word on browser and tomcat authenticates with windows AD
with MD5 scheme. So is there any setting that suggests to tomcat that do not Md5 the password but inform the LDAP provider the auth scheme is MD5.
thanks
d.
Re: Tomcat not closing threads
Posted by Chetan Chheda <ch...@yahoo.com>.
Hi martin,
Thanks for your reply but you have left me confused
Since Apache is setup as below
Maxclients = 512
ThreadsPerChild = 256
Wont it restrict the processes to 2 processes with 256 threads each instead of the ServerLimit(16) ? Looking at the OS , I see 2 httpd processes each with 258 threads
Thanks,
Chetan
________________________________
From: Martin Gainty <mg...@hotmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Monday, May 18, 2009 4:00:16 PM
Subject: RE: Tomcat not closing threads
good to know
With worker mpm the connection_pool_size defaults to ThreadsPerChild
ThreadsPerChild that in the final gives the MaxClients connections to the Tomcat.
So the default value for connection_pool_size is always ThreadsPerChild
For threaded and hybrid servers (e.g. beos
or worker) MaxClients restricts
the total number of threads that will be available to serve clients.
The default value for beos is 50.
For
hybrid MPMs the default value is 16 (ServerLimit) multiplied by the value of
25 (ThreadsPerChild). Therefore, to increase MaxClients to a value that requires more than 16 processes,
you must also raise ServerLimit.
/*******in your case*******/
(MaxClients)MaxClients=16(ServerLimit)*256(ThreadsPerChild)=4096
onto thread calculation:
Description:Minimum number of idle threads available to handle request
spikes
Syntax:MinSpareThreads number
Default:See usage for details
Context:server config
Status:MPM
Module:beos, mpm_netware, mpmt_os2, worker
Minimum number of idle threads to handle request spikes.
Different MPMs deal with this directive
differently.
worker
uses a default of MinSpareThreads
75 and deal with idle threads on a server-wide basis. If
there aren't enough idle threads in the server then child
processes are created until the number of idle threads is greater
than number.
mpm_netware uses a default of
MinSpareThreads 10 and, since it is a single-process
MPM, tracks this on a server-wide bases.
beos and mpmt_os2 work
similar to mpm_netware. The default for
beos is MinSpareThreads 1. For
mpmt_os2 the default value is 5.
/*********lets assume you havent touched this and MinSpareThreads stays at 75***********/
MaxSpareThreads(this is the parameter that is plaguing your environment)
The range of the MaxSpareThreads value
is restricted. Apache will correct the given value automatically
according to the following rules:
perchild requires MaxSpareThreads to be less or equal than ThreadLimit.mpm_netware wants the value to be greater than
MinSpareThreads.For leader, threadpool and
worker the value must be greater or equal than
the sum of MinSpareThreads
and ThreadsPerChild.
MaxSpareThreads >= MinSpareThreads+ThreadsPerChild
MaxSpareThreads >=75+256
MaxSpareThreads must be >= 331
after all that if you still dont have enough SpareThreads i would up (master thread limit) ThreadLimit to the max
The default value for ThreadLimit is
1920 when used with mpm_winnt and
64 when used with the others.
There is a hard limit of ThreadLimit 20000 (or
ThreadLimit 15000 with mpm_winnt)
compiled into the server. This is intended to avoid nasty effects
caused by typos.
/****conceivable to start with 1920 and increase by increments of 500 until number of threads sufficient ***/
BTW: the ceiling on ThreadLimit=15000
In your case
ThreadLimit=1920
HTH
Martin Gainty
______________________________________________
Jogi és Bizalmassági kinyilatkoztatás/Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Ez az
üzenet bizalmas. Ha nem ön az akinek szánva volt, akkor kérjük, hogy
jelentse azt nekünk vissza. Semmiféle továbbítása vagy másolatának
készítése nem megengedett. Ez az üzenet csak ismeret cserét szolgál és
semmiféle jogi alkalmazhatósága sincs. Mivel az electronikus üzenetek
könnyen megváltoztathatóak, ezért minket semmi felelöség nem terhelhet
ezen üzenet tartalma miatt.
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> Date: Mon, 18 May 2009 12:03:35 -0700
> From: chetan_chheda@yahoo.com
> Subject: Re: Tomcat not closing threads
> To: users@tomcat.apache.org
>
> Apache is using worker mpm
> Server version: Apache/2.0.59 HP-UX_Apache-based_Web_Server
> Server built: Aug 21 2007 13:59:06
> Server's Module Magic Number: 20020903:12
> Server loaded: APR 0.9.12, APR-UTIL 0.9.12
> Compiled using: APR 0.9.12, APR-UTIL 0.9.12
> Architecture: 64-bit
> Server compiled with....
> -D APACHE_MPM_DIR="server/mpm/worker"
>
>
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Monday, May 18, 2009 2:44:45 PM
> Subject: RE: Tomcat not closing threads
>
>
> need to know if Apache MPM prefork or Apache worker is configured
> $APACHE_HOME/bin/Apache -l
>
> *any compiled-in modules with 'mpm' in the module-name is prefork*
> *any compiled-in modules with 'worker' in the module-name is worker*
>
> Martin Gainty
> ______________________________________________
> Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
>
> Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
> Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
>
>
>
>
> > Date: Mon, 18 May 2009 08:45:43 -0700
> > From: chetan_chheda@yahoo.com
> > Subject: Re: Tomcat not closing threads
> > To: users@tomcat.apache.org
> >
> > So what you are suggesting is my mod_jk and tomcat parameters in server.xml are messed up ..
> >
> > Here is my current setup I have one apache load balancing to 2 tomcat workers
> > Maxclients = 512
> > ThreadsPerChild = 256
> > Connection_pool_size = 256
> > Tomcat MaxThreads = 150
> >
> > I will suggest my team to change this to ...
> > connection_pool_size = 120 - which means that I will have 120*2(2 apache server procs) = 240 threads per tomcat
> > Tomcat maxthreads = 300
> >
> > So out of total 512 worker threads , a max of 480 threads will be used by mod_jk leaving the rest for serving static content.
> >
> > is my math correct?
> >
> > Thanks!
> >
> >
> >
> > ________________________________
> > From: Martin Gainty <mg...@hotmail.com>
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Sent: Monday, May 18, 2009 11:23:58 AM
> > Subject: RE: Tomcat not closing threads
> >
> >
> > some documentation:
> >
> > http://httpd.apache.org/docs/2.0/mod/mpm_common.html
> > MaxClients:
> > The MaxClients directive sets the limit on the number of simultaneous requests that will be served.
> > Any connection attempts over the MaxClients limit will normally be queued, up to a number based on the ListenBacklog directive.
> > Once a child process is freed at the end of a different request, the connection will then be serviced.
> > For non-threaded servers (i.e., prefork), MaxClients translates into the maximum number of child processes that will be launched to serve requests. The default value is 256; to increase it, you must also raise ServerLimit.
> > For threaded and hybrid servers (e.g. beos or worker) MaxClients restricts the total number of threads that will be available to serve clients.
> > The default value for beos is 50.
> > For hybrid MPMs (your situation) the default value is 16 (ServerLimit) multiplied by the value of ThreadsPerChild.
> > Therefore, to increase MaxClients to a value that requires more than 16 processes, you must also raise ServerLimit.
> >
> > ThreadsPerChild
> > This directive sets the number of threads created by each child process.
> > The child creates these threads at startup and never creates more.
> > If using an MPM like mpm_winnt, where there is only one child process, this number should be high enough to handle the entire load of the server.
> > If using an MPM like worker, where there are multiple child processes, the total number of threads should be high enough to handle the common load on the server.
> > The default value for ThreadsPerChild is 64 when used with mpm_winnt
> > The default value is 25 when used with other configurations such as worker
> >
> > also from mladens discussion of connection_pool_size calcs based on Apache HTTPD Prefork vs Worker
> > http://www.mail-archive.com/users@tomcat.apache.org/msg14767.html
> > Apache Prefork configuration
> > Apache 1.3 and Apache 2.x-prefork create a separate child process for
> > each client connection, meaning that if you have default
> > 250 MaxClients, you can end up with MaxClients * connection_pool_size
> > connections to the Tomcat, that would in your case be 250000.
> > (and in the case of ThreadsPerChild parameter MaxClients*64*connection_pool_size)
> >
> > Apache Worker configuration
> > With worker mpm the connection_pool_size defaults to ThreadsPerChild,
> > that in the final gives the MaxClients connections to the Tomcat.
> > So the default value for connection_pool_size is always ThreadsPerChild
> > that with prefork mpm is by design always 1.
> > (in the case of ThreadsPerChild parameter not being null MaxClients*ThreadsPerChild)
> >
> > HTH
> > Martin
> > ______________________________________________
> > Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
> >
> > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
> > Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> >
> >
> >
> >
> > Date: Mon, 18 May 2009 07:43:04 -0700
> > From: chetan_chheda@yahoo.com
> > Subject: Re: Tomcat not closing threads
> > To: users@tomcat.apache.org
> >
> >
> >
> > Our network team is already engaged and looking into this. We had several instances of tomcats locking up today. I have attached one such thread dump here ..
> >
> > Can some of you pls take a look and point me into the right direction?
> >
> >
> >
> >
> >
> >
> > From: rad muthu <rm...@gmail.com>
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Sent: Wednesday, May 13, 2009 8:05:08 PM
> > Subject: Re: Tomcat not closing threads
> >
> > Chetan,
> >
> >
> > We had similar issue with Sun solaris and weblogic.
> >
> > My system admin found out there is a socket read errors are going on . He
> > used snoop command in solaris. After tuning the TCP parameters the issue got
> > resolved.
> >
> > You might also need to something like that.
> >
> > Thanks
> > Krish
> >
> >
> >
> > On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda <ch...@yahoo.com>wrote:
> >
> > > I guess I need to ask my question again. Im primarily an
> > HP-UX
> > > administrator and recently inherited this web based application ...
> > >
> > > We are having some issues in accomodating additional user
> > > load/functionality. While a part of the team is looking at optimizing code,
> > > I am responsible for infrastructure componants.
> > >
> > > One particular focus area for me is the mod_jk and tomcat configurations.
> > > 1. Last of April saw one of the tomcat's stop processing requests. Looking
> > > at the mod_jk.log I saw the following errors
> > > Unable to get the free endpoint for worker XXX from 37 slots ....
> > > I have setup the status servlet for mod_jk in which I saw that max slots
> > > were opened. Thinking that mod_jk needs more number of connections, I
> > > basically set it to ThreadsPerChild from worker MPM
> > >
> > > 2. Now we are randomly seeing that even a simple activity as user login
> > > causes a spike in used tomcat connector threads.
> > Eventually all tomcat
> > > threads are used and no new requests can be serviced.
> > >
> > > So, long story short, is my apache, mod_jk and tomcat configuration in
> > > sync?
> > > Maxclients = 512
> > > ThreadsPerChild = 256
> > > So that means 2 server processes with 256 threads each.
> > >
> > > I have 2 worker threads in my worker.properties file with
> > > connection_pool_size=256.
> > > Does this mean that total number of connections into tomcat = 256 *
> > > 2(number of workers) * 2(number of server procs) = 1024 ??
> > >
> > > Do I need to change the connection_pool_size to 120 . Which means 120*2 *2
> > > = 480 and that leaves the remainder of apache threads for static content?
> > >
> > > 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in
> > > server.xml . Is this why connections remain open?
> > >
> > > Thanks,
> > > Chetan
> > >
> > >
> > >
> > ________________________________
> > > From: "Caldarale, Charles R" <Ch...@unisys.com>
> > > To: Tomcat Users List <us...@tomcat.apache.org>
> > > Sent: Tuesday, May 12, 2009 10:47:00 PM
> > > Subject: RE: Tomcat not closing threads
> > >
> > > > From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> > > > Subject: Re: Tomcat not closing threads
> > > >
> > > > I managed to get a thread dump during one such tomcat hangs. Most of
> > > > the threads are in the following status ...
> > >
> > > The ones you show are simply waiting for input from httpd; that's a pretty
> > > normal state. When looking at a thread dump, it's often
> > the oddball thread
> > > that's not doing what the rest are that is causing a problem. However, if
> > > all of the threads are just waiting to receive something from httpd, then
> > > you may have to look outside of Tomcat for the cause.
> > >
> > > - Chuck
> > >
> > >
> > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> > > MATERIAL and is thus for use only by the intended recipient. If you received
> > > this in error, please contact the sender and delete the e-mail and its
> > > attachments from all computers.
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > > For additional commands, e-mail: users-help@tomcat.apache.org
> > >
> > >
> > >
> > >
> >
> >
> > _________________________________________________________________
> > Windows Live™: Keep your life in sync.
> > http://windowslive.com/explore?ocid=TXT_TAGLM_BR_life_in_synch_052009
> >
> >
> >
>
> _________________________________________________________________
> Hotmail® goes with you.
> http://windowslive.com/Tutorial/Hotmail/Mobile?ocid=TXT_TAGLM_WL_HM_Tutorial_Mobile1_052009
>
>
>
_________________________________________________________________
Hotmail® has ever-growing storage! Don’t worry about storage limits.
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutorial_Storage1_052009
RE: Tomcat not closing threads
Posted by Martin Gainty <mg...@hotmail.com>.
good to know
With worker mpm the connection_pool_size defaults to ThreadsPerChild
ThreadsPerChild that in the final gives the MaxClients connections to the Tomcat.
So the default value for connection_pool_size is always ThreadsPerChild
For threaded and hybrid servers (e.g. beos
or worker) MaxClients restricts
the total number of threads that will be available to serve clients.
The default value for beos is 50.
For
hybrid MPMs the default value is 16 (ServerLimit) multiplied by the value of
25 (ThreadsPerChild). Therefore, to increase MaxClients to a value that requires more than 16 processes,
you must also raise ServerLimit.
/*******in your case*******/
(MaxClients)MaxClients=16(ServerLimit)*256(ThreadsPerChild)=4096
onto thread calculation:
Description:Minimum number of idle threads available to handle request
spikes
Syntax:MinSpareThreads number
Default:See usage for details
Context:server config
Status:MPM
Module:beos, mpm_netware, mpmt_os2, worker
Minimum number of idle threads to handle request spikes.
Different MPMs deal with this directive
differently.
worker
uses a default of MinSpareThreads
75 and deal with idle threads on a server-wide basis. If
there aren't enough idle threads in the server then child
processes are created until the number of idle threads is greater
than number.
mpm_netware uses a default of
MinSpareThreads 10 and, since it is a single-process
MPM, tracks this on a server-wide bases.
beos and mpmt_os2 work
similar to mpm_netware. The default for
beos is MinSpareThreads 1. For
mpmt_os2 the default value is 5.
/*********lets assume you havent touched this and MinSpareThreads stays at 75***********/
MaxSpareThreads(this is the parameter that is plaguing your environment)
The range of the MaxSpareThreads value
is restricted. Apache will correct the given value automatically
according to the following rules:
perchild requires MaxSpareThreads to be less or equal than ThreadLimit.mpm_netware wants the value to be greater than
MinSpareThreads.For leader, threadpool and
worker the value must be greater or equal than
the sum of MinSpareThreads
and ThreadsPerChild.
MaxSpareThreads >= MinSpareThreads+ThreadsPerChild
MaxSpareThreads >=75+256
MaxSpareThreads must be >= 331
after all that if you still dont have enough SpareThreads i would up (master thread limit) ThreadLimit to the max
The default value for ThreadLimit is
1920 when used with mpm_winnt and
64 when used with the others.
There is a hard limit of ThreadLimit 20000 (or
ThreadLimit 15000 with mpm_winnt)
compiled into the server. This is intended to avoid nasty effects
caused by typos.
/****conceivable to start with 1920 and increase by increments of 500 until number of threads sufficient ***/
BTW: the ceiling on ThreadLimit=15000
In your case
ThreadLimit=1920
HTH
Martin Gainty
______________________________________________
Jogi és Bizalmassági kinyilatkoztatás/Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Ez az
üzenet bizalmas. Ha nem ön az akinek szánva volt, akkor kérjük, hogy
jelentse azt nekünk vissza. Semmiféle továbbítása vagy másolatának
készítése nem megengedett. Ez az üzenet csak ismeret cserét szolgál és
semmiféle jogi alkalmazhatósága sincs. Mivel az electronikus üzenetek
könnyen megváltoztathatóak, ezért minket semmi felelöség nem terhelhet
ezen üzenet tartalma miatt.
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> Date: Mon, 18 May 2009 12:03:35 -0700
> From: chetan_chheda@yahoo.com
> Subject: Re: Tomcat not closing threads
> To: users@tomcat.apache.org
>
> Apache is using worker mpm
> Server version: Apache/2.0.59 HP-UX_Apache-based_Web_Server
> Server built: Aug 21 2007 13:59:06
> Server's Module Magic Number: 20020903:12
> Server loaded: APR 0.9.12, APR-UTIL 0.9.12
> Compiled using: APR 0.9.12, APR-UTIL 0.9.12
> Architecture: 64-bit
> Server compiled with....
> -D APACHE_MPM_DIR="server/mpm/worker"
>
>
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Monday, May 18, 2009 2:44:45 PM
> Subject: RE: Tomcat not closing threads
>
>
> need to know if Apache MPM prefork or Apache worker is configured
> $APACHE_HOME/bin/Apache -l
>
> *any compiled-in modules with 'mpm' in the module-name is prefork*
> *any compiled-in modules with 'worker' in the module-name is worker*
>
> Martin Gainty
> ______________________________________________
> Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
>
> Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
> Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
>
>
>
>
> > Date: Mon, 18 May 2009 08:45:43 -0700
> > From: chetan_chheda@yahoo.com
> > Subject: Re: Tomcat not closing threads
> > To: users@tomcat.apache.org
> >
> > So what you are suggesting is my mod_jk and tomcat parameters in server.xml are messed up ..
> >
> > Here is my current setup I have one apache load balancing to 2 tomcat workers
> > Maxclients = 512
> > ThreadsPerChild = 256
> > Connection_pool_size = 256
> > Tomcat MaxThreads = 150
> >
> > I will suggest my team to change this to ...
> > connection_pool_size = 120 - which means that I will have 120*2(2 apache server procs) = 240 threads per tomcat
> > Tomcat maxthreads = 300
> >
> > So out of total 512 worker threads , a max of 480 threads will be used by mod_jk leaving the rest for serving static content.
> >
> > is my math correct?
> >
> > Thanks!
> >
> >
> >
> > ________________________________
> > From: Martin Gainty <mg...@hotmail.com>
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Sent: Monday, May 18, 2009 11:23:58 AM
> > Subject: RE: Tomcat not closing threads
> >
> >
> > some documentation:
> >
> > http://httpd.apache.org/docs/2.0/mod/mpm_common.html
> > MaxClients:
> > The MaxClients directive sets the limit on the number of simultaneous requests that will be served.
> > Any connection attempts over the MaxClients limit will normally be queued, up to a number based on the ListenBacklog directive.
> > Once a child process is freed at the end of a different request, the connection will then be serviced.
> > For non-threaded servers (i.e., prefork), MaxClients translates into the maximum number of child processes that will be launched to serve requests. The default value is 256; to increase it, you must also raise ServerLimit.
> > For threaded and hybrid servers (e.g. beos or worker) MaxClients restricts the total number of threads that will be available to serve clients.
> > The default value for beos is 50.
> > For hybrid MPMs (your situation) the default value is 16 (ServerLimit) multiplied by the value of ThreadsPerChild.
> > Therefore, to increase MaxClients to a value that requires more than 16 processes, you must also raise ServerLimit.
> >
> > ThreadsPerChild
> > This directive sets the number of threads created by each child process.
> > The child creates these threads at startup and never creates more.
> > If using an MPM like mpm_winnt, where there is only one child process, this number should be high enough to handle the entire load of the server.
> > If using an MPM like worker, where there are multiple child processes, the total number of threads should be high enough to handle the common load on the server.
> > The default value for ThreadsPerChild is 64 when used with mpm_winnt
> > The default value is 25 when used with other configurations such as worker
> >
> > also from mladens discussion of connection_pool_size calcs based on Apache HTTPD Prefork vs Worker
> > http://www.mail-archive.com/users@tomcat.apache.org/msg14767.html
> > Apache Prefork configuration
> > Apache 1.3 and Apache 2.x-prefork create a separate child process for
> > each client connection, meaning that if you have default
> > 250 MaxClients, you can end up with MaxClients * connection_pool_size
> > connections to the Tomcat, that would in your case be 250000.
> > (and in the case of ThreadsPerChild parameter MaxClients*64*connection_pool_size)
> >
> > Apache Worker configuration
> > With worker mpm the connection_pool_size defaults to ThreadsPerChild,
> > that in the final gives the MaxClients connections to the Tomcat.
> > So the default value for connection_pool_size is always ThreadsPerChild
> > that with prefork mpm is by design always 1.
> > (in the case of ThreadsPerChild parameter not being null MaxClients*ThreadsPerChild)
> >
> > HTH
> > Martin
> > ______________________________________________
> > Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
> >
> > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
> > Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> >
> >
> >
> >
> > Date: Mon, 18 May 2009 07:43:04 -0700
> > From: chetan_chheda@yahoo.com
> > Subject: Re: Tomcat not closing threads
> > To: users@tomcat.apache.org
> >
> >
> >
> > Our network team is already engaged and looking into this. We had several instances of tomcats locking up today. I have attached one such thread dump here ..
> >
> > Can some of you pls take a look and point me into the right direction?
> >
> >
> >
> >
> >
> >
> > From: rad muthu <rm...@gmail.com>
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Sent: Wednesday, May 13, 2009 8:05:08 PM
> > Subject: Re: Tomcat not closing threads
> >
> > Chetan,
> >
> >
> > We had similar issue with Sun solaris and weblogic.
> >
> > My system admin found out there is a socket read errors are going on . He
> > used snoop command in solaris. After tuning the TCP parameters the issue got
> > resolved.
> >
> > You might also need to something like that.
> >
> > Thanks
> > Krish
> >
> >
> >
> > On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda <ch...@yahoo.com>wrote:
> >
> > > I guess I need to ask my question again. Im primarily an
> > HP-UX
> > > administrator and recently inherited this web based application ...
> > >
> > > We are having some issues in accomodating additional user
> > > load/functionality. While a part of the team is looking at optimizing code,
> > > I am responsible for infrastructure componants.
> > >
> > > One particular focus area for me is the mod_jk and tomcat configurations.
> > > 1. Last of April saw one of the tomcat's stop processing requests. Looking
> > > at the mod_jk.log I saw the following errors
> > > Unable to get the free endpoint for worker XXX from 37 slots ....
> > > I have setup the status servlet for mod_jk in which I saw that max slots
> > > were opened. Thinking that mod_jk needs more number of connections, I
> > > basically set it to ThreadsPerChild from worker MPM
> > >
> > > 2. Now we are randomly seeing that even a simple activity as user login
> > > causes a spike in used tomcat connector threads.
> > Eventually all tomcat
> > > threads are used and no new requests can be serviced.
> > >
> > > So, long story short, is my apache, mod_jk and tomcat configuration in
> > > sync?
> > > Maxclients = 512
> > > ThreadsPerChild = 256
> > > So that means 2 server processes with 256 threads each.
> > >
> > > I have 2 worker threads in my worker.properties file with
> > > connection_pool_size=256.
> > > Does this mean that total number of connections into tomcat = 256 *
> > > 2(number of workers) * 2(number of server procs) = 1024 ??
> > >
> > > Do I need to change the connection_pool_size to 120 . Which means 120*2 *2
> > > = 480 and that leaves the remainder of apache threads for static content?
> > >
> > > 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in
> > > server.xml . Is this why connections remain open?
> > >
> > > Thanks,
> > > Chetan
> > >
> > >
> > >
> > ________________________________
> > > From: "Caldarale, Charles R" <Ch...@unisys.com>
> > > To: Tomcat Users List <us...@tomcat.apache.org>
> > > Sent: Tuesday, May 12, 2009 10:47:00 PM
> > > Subject: RE: Tomcat not closing threads
> > >
> > > > From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> > > > Subject: Re: Tomcat not closing threads
> > > >
> > > > I managed to get a thread dump during one such tomcat hangs. Most of
> > > > the threads are in the following status ...
> > >
> > > The ones you show are simply waiting for input from httpd; that's a pretty
> > > normal state. When looking at a thread dump, it's often
> > the oddball thread
> > > that's not doing what the rest are that is causing a problem. However, if
> > > all of the threads are just waiting to receive something from httpd, then
> > > you may have to look outside of Tomcat for the cause.
> > >
> > > - Chuck
> > >
> > >
> > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> > > MATERIAL and is thus for use only by the intended recipient. If you received
> > > this in error, please contact the sender and delete the e-mail and its
> > > attachments from all computers.
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > > For additional commands, e-mail: users-help@tomcat.apache.org
> > >
> > >
> > >
> > >
> >
> >
> > _________________________________________________________________
> > Windows Live™: Keep your life in sync.
> > http://windowslive.com/explore?ocid=TXT_TAGLM_BR_life_in_synch_052009
> >
> >
> >
>
> _________________________________________________________________
> Hotmail® goes with you.
> http://windowslive.com/Tutorial/Hotmail/Mobile?ocid=TXT_TAGLM_WL_HM_Tutorial_Mobile1_052009
>
>
>
_________________________________________________________________
Hotmail® has ever-growing storage! Don’t worry about storage limits.
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutorial_Storage1_052009
Re: Tomcat not closing threads
Posted by Chetan Chheda <ch...@yahoo.com>.
Apache is using worker mpm
Server version: Apache/2.0.59 HP-UX_Apache-based_Web_Server
Server built: Aug 21 2007 13:59:06
Server's Module Magic Number: 20020903:12
Server loaded: APR 0.9.12, APR-UTIL 0.9.12
Compiled using: APR 0.9.12, APR-UTIL 0.9.12
Architecture: 64-bit
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/worker"
________________________________
From: Martin Gainty <mg...@hotmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Monday, May 18, 2009 2:44:45 PM
Subject: RE: Tomcat not closing threads
need to know if Apache MPM prefork or Apache worker is configured
$APACHE_HOME/bin/Apache -l
*any compiled-in modules with 'mpm' in the module-name is prefork*
*any compiled-in modules with 'worker' in the module-name is worker*
Martin Gainty
______________________________________________
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> Date: Mon, 18 May 2009 08:45:43 -0700
> From: chetan_chheda@yahoo.com
> Subject: Re: Tomcat not closing threads
> To: users@tomcat.apache.org
>
> So what you are suggesting is my mod_jk and tomcat parameters in server.xml are messed up ..
>
> Here is my current setup I have one apache load balancing to 2 tomcat workers
> Maxclients = 512
> ThreadsPerChild = 256
> Connection_pool_size = 256
> Tomcat MaxThreads = 150
>
> I will suggest my team to change this to ...
> connection_pool_size = 120 - which means that I will have 120*2(2 apache server procs) = 240 threads per tomcat
> Tomcat maxthreads = 300
>
> So out of total 512 worker threads , a max of 480 threads will be used by mod_jk leaving the rest for serving static content.
>
> is my math correct?
>
> Thanks!
>
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Monday, May 18, 2009 11:23:58 AM
> Subject: RE: Tomcat not closing threads
>
>
> some documentation:
>
> http://httpd.apache.org/docs/2.0/mod/mpm_common.html
> MaxClients:
> The MaxClients directive sets the limit on the number of simultaneous requests that will be served.
> Any connection attempts over the MaxClients limit will normally be queued, up to a number based on the ListenBacklog directive.
> Once a child process is freed at the end of a different request, the connection will then be serviced.
> For non-threaded servers (i.e., prefork), MaxClients translates into the maximum number of child processes that will be launched to serve requests. The default value is 256; to increase it, you must also raise ServerLimit.
> For threaded and hybrid servers (e.g. beos or worker) MaxClients restricts the total number of threads that will be available to serve clients.
> The default value for beos is 50.
> For hybrid MPMs (your situation) the default value is 16 (ServerLimit) multiplied by the value of ThreadsPerChild.
> Therefore, to increase MaxClients to a value that requires more than 16 processes, you must also raise ServerLimit.
>
> ThreadsPerChild
> This directive sets the number of threads created by each child process.
> The child creates these threads at startup and never creates more.
> If using an MPM like mpm_winnt, where there is only one child process, this number should be high enough to handle the entire load of the server.
> If using an MPM like worker, where there are multiple child processes, the total number of threads should be high enough to handle the common load on the server.
> The default value for ThreadsPerChild is 64 when used with mpm_winnt
> The default value is 25 when used with other configurations such as worker
>
> also from mladens discussion of connection_pool_size calcs based on Apache HTTPD Prefork vs Worker
> http://www.mail-archive.com/users@tomcat.apache.org/msg14767.html
> Apache Prefork configuration
> Apache 1.3 and Apache 2.x-prefork create a separate child process for
> each client connection, meaning that if you have default
> 250 MaxClients, you can end up with MaxClients * connection_pool_size
> connections to the Tomcat, that would in your case be 250000.
> (and in the case of ThreadsPerChild parameter MaxClients*64*connection_pool_size)
>
> Apache Worker configuration
> With worker mpm the connection_pool_size defaults to ThreadsPerChild,
> that in the final gives the MaxClients connections to the Tomcat.
> So the default value for connection_pool_size is always ThreadsPerChild
> that with prefork mpm is by design always 1.
> (in the case of ThreadsPerChild parameter not being null MaxClients*ThreadsPerChild)
>
> HTH
> Martin
> ______________________________________________
> Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
>
> Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
> Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
>
>
>
>
> Date: Mon, 18 May 2009 07:43:04 -0700
> From: chetan_chheda@yahoo.com
> Subject: Re: Tomcat not closing threads
> To: users@tomcat.apache.org
>
>
>
> Our network team is already engaged and looking into this. We had several instances of tomcats locking up today. I have attached one such thread dump here ..
>
> Can some of you pls take a look and point me into the right direction?
>
>
>
>
>
>
> From: rad muthu <rm...@gmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Wednesday, May 13, 2009 8:05:08 PM
> Subject: Re: Tomcat not closing threads
>
> Chetan,
>
>
> We had similar issue with Sun solaris and weblogic.
>
> My system admin found out there is a socket read errors are going on . He
> used snoop command in solaris. After tuning the TCP parameters the issue got
> resolved.
>
> You might also need to something like that.
>
> Thanks
> Krish
>
>
>
> On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda <ch...@yahoo.com>wrote:
>
> > I guess I need to ask my question again. Im primarily an
> HP-UX
> > administrator and recently inherited this web based application ...
> >
> > We are having some issues in accomodating additional user
> > load/functionality. While a part of the team is looking at optimizing code,
> > I am responsible for infrastructure componants.
> >
> > One particular focus area for me is the mod_jk and tomcat configurations.
> > 1. Last of April saw one of the tomcat's stop processing requests. Looking
> > at the mod_jk.log I saw the following errors
> > Unable to get the free endpoint for worker XXX from 37 slots ....
> > I have setup the status servlet for mod_jk in which I saw that max slots
> > were opened. Thinking that mod_jk needs more number of connections, I
> > basically set it to ThreadsPerChild from worker MPM
> >
> > 2. Now we are randomly seeing that even a simple activity as user login
> > causes a spike in used tomcat connector threads.
> Eventually all tomcat
> > threads are used and no new requests can be serviced.
> >
> > So, long story short, is my apache, mod_jk and tomcat configuration in
> > sync?
> > Maxclients = 512
> > ThreadsPerChild = 256
> > So that means 2 server processes with 256 threads each.
> >
> > I have 2 worker threads in my worker.properties file with
> > connection_pool_size=256.
> > Does this mean that total number of connections into tomcat = 256 *
> > 2(number of workers) * 2(number of server procs) = 1024 ??
> >
> > Do I need to change the connection_pool_size to 120 . Which means 120*2 *2
> > = 480 and that leaves the remainder of apache threads for static content?
> >
> > 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in
> > server.xml . Is this why connections remain open?
> >
> > Thanks,
> > Chetan
> >
> >
> >
> ________________________________
> > From: "Caldarale, Charles R" <Ch...@unisys.com>
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Sent: Tuesday, May 12, 2009 10:47:00 PM
> > Subject: RE: Tomcat not closing threads
> >
> > > From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> > > Subject: Re: Tomcat not closing threads
> > >
> > > I managed to get a thread dump during one such tomcat hangs. Most of
> > > the threads are in the following status ...
> >
> > The ones you show are simply waiting for input from httpd; that's a pretty
> > normal state. When looking at a thread dump, it's often
> the oddball thread
> > that's not doing what the rest are that is causing a problem. However, if
> > all of the threads are just waiting to receive something from httpd, then
> > you may have to look outside of Tomcat for the cause.
> >
> > - Chuck
> >
> >
> > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> > MATERIAL and is thus for use only by the intended recipient. If you received
> > this in error, please contact the sender and delete the e-mail and its
> > attachments from all computers.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
> >
> >
>
>
> _________________________________________________________________
> Windows Live™: Keep your life in sync.
> http://windowslive.com/explore?ocid=TXT_TAGLM_BR_life_in_synch_052009
>
>
>
_________________________________________________________________
Hotmail® goes with you.
http://windowslive.com/Tutorial/Hotmail/Mobile?ocid=TXT_TAGLM_WL_HM_Tutorial_Mobile1_052009
RE: Tomcat not closing threads
Posted by Martin Gainty <mg...@hotmail.com>.
need to know if Apache MPM prefork or Apache worker is configured
$APACHE_HOME/bin/Apache -l
*any compiled-in modules with 'mpm' in the module-name is prefork*
*any compiled-in modules with 'worker' in the module-name is worker*
Martin Gainty
______________________________________________
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
> Date: Mon, 18 May 2009 08:45:43 -0700
> From: chetan_chheda@yahoo.com
> Subject: Re: Tomcat not closing threads
> To: users@tomcat.apache.org
>
> So what you are suggesting is my mod_jk and tomcat parameters in server.xml are messed up ..
>
> Here is my current setup I have one apache load balancing to 2 tomcat workers
> Maxclients = 512
> ThreadsPerChild = 256
> Connection_pool_size = 256
> Tomcat MaxThreads = 150
>
> I will suggest my team to change this to ...
> connection_pool_size = 120 - which means that I will have 120*2(2 apache server procs) = 240 threads per tomcat
> Tomcat maxthreads = 300
>
> So out of total 512 worker threads , a max of 480 threads will be used by mod_jk leaving the rest for serving static content.
>
> is my math correct?
>
> Thanks!
>
>
>
> ________________________________
> From: Martin Gainty <mg...@hotmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Monday, May 18, 2009 11:23:58 AM
> Subject: RE: Tomcat not closing threads
>
>
> some documentation:
>
> http://httpd.apache.org/docs/2.0/mod/mpm_common.html
> MaxClients:
> The MaxClients directive sets the limit on the number of simultaneous requests that will be served.
> Any connection attempts over the MaxClients limit will normally be queued, up to a number based on the ListenBacklog directive.
> Once a child process is freed at the end of a different request, the connection will then be serviced.
> For non-threaded servers (i.e., prefork), MaxClients translates into the maximum number of child processes that will be launched to serve requests. The default value is 256; to increase it, you must also raise ServerLimit.
> For threaded and hybrid servers (e.g. beos or worker) MaxClients restricts the total number of threads that will be available to serve clients.
> The default value for beos is 50.
> For hybrid MPMs (your situation) the default value is 16 (ServerLimit) multiplied by the value of ThreadsPerChild.
> Therefore, to increase MaxClients to a value that requires more than 16 processes, you must also raise ServerLimit.
>
> ThreadsPerChild
> This directive sets the number of threads created by each child process.
> The child creates these threads at startup and never creates more.
> If using an MPM like mpm_winnt, where there is only one child process, this number should be high enough to handle the entire load of the server.
> If using an MPM like worker, where there are multiple child processes, the total number of threads should be high enough to handle the common load on the server.
> The default value for ThreadsPerChild is 64 when used with mpm_winnt
> The default value is 25 when used with other configurations such as worker
>
> also from mladens discussion of connection_pool_size calcs based on Apache HTTPD Prefork vs Worker
> http://www.mail-archive.com/users@tomcat.apache.org/msg14767.html
> Apache Prefork configuration
> Apache 1.3 and Apache 2.x-prefork create a separate child process for
> each client connection, meaning that if you have default
> 250 MaxClients, you can end up with MaxClients * connection_pool_size
> connections to the Tomcat, that would in your case be 250000.
> (and in the case of ThreadsPerChild parameter MaxClients*64*connection_pool_size)
>
> Apache Worker configuration
> With worker mpm the connection_pool_size defaults to ThreadsPerChild,
> that in the final gives the MaxClients connections to the Tomcat.
> So the default value for connection_pool_size is always ThreadsPerChild
> that with prefork mpm is by design always 1.
> (in the case of ThreadsPerChild parameter not being null MaxClients*ThreadsPerChild)
>
> HTH
> Martin
> ______________________________________________
> Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
>
> Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
> Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
>
>
>
>
> Date: Mon, 18 May 2009 07:43:04 -0700
> From: chetan_chheda@yahoo.com
> Subject: Re: Tomcat not closing threads
> To: users@tomcat.apache.org
>
>
>
> Our network team is already engaged and looking into this. We had several instances of tomcats locking up today. I have attached one such thread dump here ..
>
> Can some of you pls take a look and point me into the right direction?
>
>
>
>
>
>
> From: rad muthu <rm...@gmail.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Wednesday, May 13, 2009 8:05:08 PM
> Subject: Re: Tomcat not closing threads
>
> Chetan,
>
>
> We had similar issue with Sun solaris and weblogic.
>
> My system admin found out there is a socket read errors are going on . He
> used snoop command in solaris. After tuning the TCP parameters the issue got
> resolved.
>
> You might also need to something like that.
>
> Thanks
> Krish
>
>
>
> On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda <ch...@yahoo.com>wrote:
>
> > I guess I need to ask my question again. Im primarily an
> HP-UX
> > administrator and recently inherited this web based application ...
> >
> > We are having some issues in accomodating additional user
> > load/functionality. While a part of the team is looking at optimizing code,
> > I am responsible for infrastructure componants.
> >
> > One particular focus area for me is the mod_jk and tomcat configurations.
> > 1. Last of April saw one of the tomcat's stop processing requests. Looking
> > at the mod_jk.log I saw the following errors
> > Unable to get the free endpoint for worker XXX from 37 slots ....
> > I have setup the status servlet for mod_jk in which I saw that max slots
> > were opened. Thinking that mod_jk needs more number of connections, I
> > basically set it to ThreadsPerChild from worker MPM
> >
> > 2. Now we are randomly seeing that even a simple activity as user login
> > causes a spike in used tomcat connector threads.
> Eventually all tomcat
> > threads are used and no new requests can be serviced.
> >
> > So, long story short, is my apache, mod_jk and tomcat configuration in
> > sync?
> > Maxclients = 512
> > ThreadsPerChild = 256
> > So that means 2 server processes with 256 threads each.
> >
> > I have 2 worker threads in my worker.properties file with
> > connection_pool_size=256.
> > Does this mean that total number of connections into tomcat = 256 *
> > 2(number of workers) * 2(number of server procs) = 1024 ??
> >
> > Do I need to change the connection_pool_size to 120 . Which means 120*2 *2
> > = 480 and that leaves the remainder of apache threads for static content?
> >
> > 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in
> > server.xml . Is this why connections remain open?
> >
> > Thanks,
> > Chetan
> >
> >
> >
> ________________________________
> > From: "Caldarale, Charles R" <Ch...@unisys.com>
> > To: Tomcat Users List <us...@tomcat.apache.org>
> > Sent: Tuesday, May 12, 2009 10:47:00 PM
> > Subject: RE: Tomcat not closing threads
> >
> > > From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> > > Subject: Re: Tomcat not closing threads
> > >
> > > I managed to get a thread dump during one such tomcat hangs. Most of
> > > the threads are in the following status ...
> >
> > The ones you show are simply waiting for input from httpd; that's a pretty
> > normal state. When looking at a thread dump, it's often
> the oddball thread
> > that's not doing what the rest are that is causing a problem. However, if
> > all of the threads are just waiting to receive something from httpd, then
> > you may have to look outside of Tomcat for the cause.
> >
> > - Chuck
> >
> >
> > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> > MATERIAL and is thus for use only by the intended recipient. If you received
> > this in error, please contact the sender and delete the e-mail and its
> > attachments from all computers.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
> >
> >
>
>
> _________________________________________________________________
> Windows Live™: Keep your life in sync.
> http://windowslive.com/explore?ocid=TXT_TAGLM_BR_life_in_synch_052009
>
>
>
_________________________________________________________________
Hotmail® goes with you.
http://windowslive.com/Tutorial/Hotmail/Mobile?ocid=TXT_TAGLM_WL_HM_Tutorial_Mobile1_052009
Re: Tomcat not closing threads
Posted by Chetan Chheda <ch...@yahoo.com>.
So what you are suggesting is my mod_jk and tomcat parameters in server.xml are messed up ..
Here is my current setup I have one apache load balancing to 2 tomcat workers
Maxclients = 512
ThreadsPerChild = 256
Connection_pool_size = 256
Tomcat MaxThreads = 150
I will suggest my team to change this to ...
connection_pool_size = 120 - which means that I will have 120*2(2 apache server procs) = 240 threads per tomcat
Tomcat maxthreads = 300
So out of total 512 worker threads , a max of 480 threads will be used by mod_jk leaving the rest for serving static content.
is my math correct?
Thanks!
________________________________
From: Martin Gainty <mg...@hotmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Monday, May 18, 2009 11:23:58 AM
Subject: RE: Tomcat not closing threads
some documentation:
http://httpd.apache.org/docs/2.0/mod/mpm_common.html
MaxClients:
The MaxClients directive sets the limit on the number of simultaneous requests that will be served.
Any connection attempts over the MaxClients limit will normally be queued, up to a number based on the ListenBacklog directive.
Once a child process is freed at the end of a different request, the connection will then be serviced.
For non-threaded servers (i.e., prefork), MaxClients translates into the maximum number of child processes that will be launched to serve requests. The default value is 256; to increase it, you must also raise ServerLimit.
For threaded and hybrid servers (e.g. beos or worker) MaxClients restricts the total number of threads that will be available to serve clients.
The default value for beos is 50.
For hybrid MPMs (your situation) the default value is 16 (ServerLimit) multiplied by the value of ThreadsPerChild.
Therefore, to increase MaxClients to a value that requires more than 16 processes, you must also raise ServerLimit.
ThreadsPerChild
This directive sets the number of threads created by each child process.
The child creates these threads at startup and never creates more.
If using an MPM like mpm_winnt, where there is only one child process, this number should be high enough to handle the entire load of the server.
If using an MPM like worker, where there are multiple child processes, the total number of threads should be high enough to handle the common load on the server.
The default value for ThreadsPerChild is 64 when used with mpm_winnt
The default value is 25 when used with other configurations such as worker
also from mladens discussion of connection_pool_size calcs based on Apache HTTPD Prefork vs Worker
http://www.mail-archive.com/users@tomcat.apache.org/msg14767.html
Apache Prefork configuration
Apache 1.3 and Apache 2.x-prefork create a separate child process for
each client connection, meaning that if you have default
250 MaxClients, you can end up with MaxClients * connection_pool_size
connections to the Tomcat, that would in your case be 250000.
(and in the case of ThreadsPerChild parameter MaxClients*64*connection_pool_size)
Apache Worker configuration
With worker mpm the connection_pool_size defaults to ThreadsPerChild,
that in the final gives the MaxClients connections to the Tomcat.
So the default value for connection_pool_size is always ThreadsPerChild
that with prefork mpm is by design always 1.
(in the case of ThreadsPerChild parameter not being null MaxClients*ThreadsPerChild)
HTH
Martin
______________________________________________
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
Date: Mon, 18 May 2009 07:43:04 -0700
From: chetan_chheda@yahoo.com
Subject: Re: Tomcat not closing threads
To: users@tomcat.apache.org
Our network team is already engaged and looking into this. We had several instances of tomcats locking up today. I have attached one such thread dump here ..
Can some of you pls take a look and point me into the right direction?
From: rad muthu <rm...@gmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Wednesday, May 13, 2009 8:05:08 PM
Subject: Re: Tomcat not closing threads
Chetan,
We had similar issue with Sun solaris and weblogic.
My system admin found out there is a socket read errors are going on . He
used snoop command in solaris. After tuning the TCP parameters the issue got
resolved.
You might also need to something like that.
Thanks
Krish
On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda <ch...@yahoo.com>wrote:
> I guess I need to ask my question again. Im primarily an
HP-UX
> administrator and recently inherited this web based application ...
>
> We are having some issues in accomodating additional user
> load/functionality. While a part of the team is looking at optimizing code,
> I am responsible for infrastructure componants.
>
> One particular focus area for me is the mod_jk and tomcat configurations.
> 1. Last of April saw one of the tomcat's stop processing requests. Looking
> at the mod_jk.log I saw the following errors
> Unable to get the free endpoint for worker XXX from 37 slots ....
> I have setup the status servlet for mod_jk in which I saw that max slots
> were opened. Thinking that mod_jk needs more number of connections, I
> basically set it to ThreadsPerChild from worker MPM
>
> 2. Now we are randomly seeing that even a simple activity as user login
> causes a spike in used tomcat connector threads.
Eventually all tomcat
> threads are used and no new requests can be serviced.
>
> So, long story short, is my apache, mod_jk and tomcat configuration in
> sync?
> Maxclients = 512
> ThreadsPerChild = 256
> So that means 2 server processes with 256 threads each.
>
> I have 2 worker threads in my worker.properties file with
> connection_pool_size=256.
> Does this mean that total number of connections into tomcat = 256 *
> 2(number of workers) * 2(number of server procs) = 1024 ??
>
> Do I need to change the connection_pool_size to 120 . Which means 120*2 *2
> = 480 and that leaves the remainder of apache threads for static content?
>
> 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in
> server.xml . Is this why connections remain open?
>
> Thanks,
> Chetan
>
>
>
________________________________
> From: "Caldarale, Charles R" <Ch...@unisys.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Tuesday, May 12, 2009 10:47:00 PM
> Subject: RE: Tomcat not closing threads
>
> > From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> > Subject: Re: Tomcat not closing threads
> >
> > I managed to get a thread dump during one such tomcat hangs. Most of
> > the threads are in the following status ...
>
> The ones you show are simply waiting for input from httpd; that's a pretty
> normal state. When looking at a thread dump, it's often
the oddball thread
> that's not doing what the rest are that is causing a problem. However, if
> all of the threads are just waiting to receive something from httpd, then
> you may have to look outside of Tomcat for the cause.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
_________________________________________________________________
Windows Live™: Keep your life in sync.
http://windowslive.com/explore?ocid=TXT_TAGLM_BR_life_in_synch_052009
RE: Tomcat not closing threads
Posted by Martin Gainty <mg...@hotmail.com>.
some documentation:
http://httpd.apache.org/docs/2.0/mod/mpm_common.html
MaxClients:
The MaxClients directive sets the limit on the number of simultaneous requests that will be served.
Any connection attempts over the MaxClients limit will normally be queued, up to a number based on the ListenBacklog directive.
Once a child process is freed at the end of a different request, the connection will then be serviced.
For non-threaded servers (i.e., prefork), MaxClients translates into the maximum number of child processes that will be launched to serve requests. The default value is 256; to increase it, you must also raise ServerLimit.
For threaded and hybrid servers (e.g. beos or worker) MaxClients restricts the total number of threads that will be available to serve clients.
The default value for beos is 50.
For hybrid MPMs (your situation) the default value is 16 (ServerLimit) multiplied by the value of ThreadsPerChild.
Therefore, to increase MaxClients to a value that requires more than 16 processes, you must also raise ServerLimit.
ThreadsPerChild
This directive sets the number of threads created by each child process.
The child creates these threads at startup and never creates more.
If using an MPM like mpm_winnt, where there is only one child process, this number should be high enough to handle the entire load of the server.
If using an MPM like worker, where there are multiple child processes, the total number of threads should be high enough to handle the common load on the server.
The default value for ThreadsPerChild is 64 when used with mpm_winnt
The default value is 25 when used with other configurations such as worker
also from mladens discussion of connection_pool_size calcs based on Apache HTTPD Prefork vs Worker
http://www.mail-archive.com/users@tomcat.apache.org/msg14767.html
Apache Prefork configuration
Apache 1.3 and Apache 2.x-prefork create a separate child process for
each client connection, meaning that if you have default
250 MaxClients, you can end up with MaxClients * connection_pool_size
connections to the Tomcat, that would in your case be 250000.
(and in the case of ThreadsPerChild parameter MaxClients*64*connection_pool_size)
Apache Worker configuration
With worker mpm the connection_pool_size defaults to ThreadsPerChild,
that in the final gives the MaxClients connections to the Tomcat.
So the default value for connection_pool_size is always ThreadsPerChild
that with prefork mpm is by design always 1.
(in the case of ThreadsPerChild parameter not being null MaxClients*ThreadsPerChild)
HTH
Martin
______________________________________________
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni.
Date: Mon, 18 May 2009 07:43:04 -0700
From: chetan_chheda@yahoo.com
Subject: Re: Tomcat not closing threads
To: users@tomcat.apache.org
Our network team is already engaged and looking into this. We had several instances of tomcats locking up today. I have attached one such thread dump here ..
Can some of you pls take a look and point me into the right direction?
From: rad muthu <rm...@gmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Wednesday, May 13, 2009 8:05:08 PM
Subject: Re: Tomcat not closing threads
Chetan,
We had similar issue with Sun solaris and weblogic.
My system admin found out there is a socket read errors are going on . He
used snoop command in solaris. After tuning the TCP parameters the issue got
resolved.
You might also need to something like that.
Thanks
Krish
On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda <ch...@yahoo.com>wrote:
> I guess I need to ask my question again. Im primarily an
HP-UX
> administrator and recently inherited this web based application ...
>
> We are having some issues in accomodating additional user
> load/functionality. While a part of the team is looking at optimizing code,
> I am responsible for infrastructure componants.
>
> One particular focus area for me is the mod_jk and tomcat configurations.
> 1. Last of April saw one of the tomcat's stop processing requests. Looking
> at the mod_jk.log I saw the following errors
> Unable to get the free endpoint for worker XXX from 37 slots ....
> I have setup the status servlet for mod_jk in which I saw that max slots
> were opened. Thinking that mod_jk needs more number of connections, I
> basically set it to ThreadsPerChild from worker MPM
>
> 2. Now we are randomly seeing that even a simple activity as user login
> causes a spike in used tomcat connector threads.
Eventually all tomcat
> threads are used and no new requests can be serviced.
>
> So, long story short, is my apache, mod_jk and tomcat configuration in
> sync?
> Maxclients = 512
> ThreadsPerChild = 256
> So that means 2 server processes with 256 threads each.
>
> I have 2 worker threads in my worker.properties file with
> connection_pool_size=256.
> Does this mean that total number of connections into tomcat = 256 *
> 2(number of workers) * 2(number of server procs) = 1024 ??
>
> Do I need to change the connection_pool_size to 120 . Which means 120*2 *2
> = 480 and that leaves the remainder of apache threads for static content?
>
> 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in
> server.xml . Is this why connections remain open?
>
> Thanks,
> Chetan
>
>
>
________________________________
> From: "Caldarale, Charles R" <Ch...@unisys.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Tuesday, May 12, 2009 10:47:00 PM
> Subject: RE: Tomcat not closing threads
>
> > From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> > Subject: Re: Tomcat not closing threads
> >
> > I managed to get a thread dump during one such tomcat hangs. Most of
> > the threads are in the following status ...
>
> The ones you show are simply waiting for input from httpd; that's a pretty
> normal state. When looking at a thread dump, it's often
the oddball thread
> that's not doing what the rest are that is causing a problem. However, if
> all of the threads are just waiting to receive something from httpd, then
> you may have to look outside of Tomcat for the cause.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
_________________________________________________________________
Windows Live™: Keep your life in sync.
http://windowslive.com/explore?ocid=TXT_TAGLM_BR_life_in_synch_052009
Re: Tomcat not closing threads
Posted by Chetan Chheda <ch...@yahoo.com>.
Our network team is already engaged and looking into this. We had several instances of tomcats locking up today. I have attached one such thread dump here ..
Can some of you pls take a look and point me into the right direction?
________________________________
From: rad muthu <rm...@gmail.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Wednesday, May 13, 2009 8:05:08 PM
Subject: Re: Tomcat not closing threads
Chetan,
We had similar issue with Sun solaris and weblogic.
My system admin found out there is a socket read errors are going on . He
used snoop command in solaris. After tuning the TCP parameters the issue got
resolved.
You might also need to something like that.
Thanks
Krish
On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda <ch...@yahoo.com>wrote:
> I guess I need to ask my question again. Im primarily an HP-UX
> administrator and recently inherited this web based application ...
>
> We are having some issues in accomodating additional user
> load/functionality. While a part of the team is looking at optimizing code,
> I am responsible for infrastructure componants.
>
> One particular focus area for me is the mod_jk and tomcat configurations.
> 1. Last of April saw one of the tomcat's stop processing requests. Looking
> at the mod_jk.log I saw the following errors
> Unable to get the free endpoint for worker XXX from 37 slots ....
> I have setup the status servlet for mod_jk in which I saw that max slots
> were opened. Thinking that mod_jk needs more number of connections, I
> basically set it to ThreadsPerChild from worker MPM
>
> 2. Now we are randomly seeing that even a simple activity as user login
> causes a spike in used tomcat connector threads. Eventually all tomcat
> threads are used and no new requests can be serviced.
>
> So, long story short, is my apache, mod_jk and tomcat configuration in
> sync?
> Maxclients = 512
> ThreadsPerChild = 256
> So that means 2 server processes with 256 threads each.
>
> I have 2 worker threads in my worker.properties file with
> connection_pool_size=256.
> Does this mean that total number of connections into tomcat = 256 *
> 2(number of workers) * 2(number of server procs) = 1024 ??
>
> Do I need to change the connection_pool_size to 120 . Which means 120*2 *2
> = 480 and that leaves the remainder of apache threads for static content?
>
> 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in
> server.xml . Is this why connections remain open?
>
> Thanks,
> Chetan
>
>
> ________________________________
> From: "Caldarale, Charles R" <Ch...@unisys.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Tuesday, May 12, 2009 10:47:00 PM
> Subject: RE: Tomcat not closing threads
>
> > From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> > Subject: Re: Tomcat not closing threads
> >
> > I managed to get a thread dump during one such tomcat hangs. Most of
> > the threads are in the following status ...
>
> The ones you show are simply waiting for input from httpd; that's a pretty
> normal state. When looking at a thread dump, it's often the oddball thread
> that's not doing what the rest are that is causing a problem. However, if
> all of the threads are just waiting to receive something from httpd, then
> you may have to look outside of Tomcat for the cause.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
Re: Tomcat not closing threads
Posted by rad muthu <rm...@gmail.com>.
Chetan,
We had similar issue with Sun solaris and weblogic.
My system admin found out there is a socket read errors are going on . He
used snoop command in solaris. After tuning the TCP parameters the issue got
resolved.
You might also need to something like that.
Thanks
Krish
On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda <ch...@yahoo.com>wrote:
> I guess I need to ask my question again. Im primarily an HP-UX
> administrator and recently inherited this web based application ...
>
> We are having some issues in accomodating additional user
> load/functionality. While a part of the team is looking at optimizing code,
> I am responsible for infrastructure componants.
>
> One particular focus area for me is the mod_jk and tomcat configurations.
> 1. Last of April saw one of the tomcat's stop processing requests. Looking
> at the mod_jk.log I saw the following errors
> Unable to get the free endpoint for worker XXX from 37 slots ....
> I have setup the status servlet for mod_jk in which I saw that max slots
> were opened. Thinking that mod_jk needs more number of connections, I
> basically set it to ThreadsPerChild from worker MPM
>
> 2. Now we are randomly seeing that even a simple activity as user login
> causes a spike in used tomcat connector threads. Eventually all tomcat
> threads are used and no new requests can be serviced.
>
> So, long story short, is my apache, mod_jk and tomcat configuration in
> sync?
> Maxclients = 512
> ThreadsPerChild = 256
> So that means 2 server processes with 256 threads each.
>
> I have 2 worker threads in my worker.properties file with
> connection_pool_size=256.
> Does this mean that total number of connections into tomcat = 256 *
> 2(number of workers) * 2(number of server procs) = 1024 ??
>
> Do I need to change the connection_pool_size to 120 . Which means 120*2 *2
> = 480 and that leaves the remainder of apache threads for static content?
>
> 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in
> server.xml . Is this why connections remain open?
>
> Thanks,
> Chetan
>
>
> ________________________________
> From: "Caldarale, Charles R" <Ch...@unisys.com>
> To: Tomcat Users List <us...@tomcat.apache.org>
> Sent: Tuesday, May 12, 2009 10:47:00 PM
> Subject: RE: Tomcat not closing threads
>
> > From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> > Subject: Re: Tomcat not closing threads
> >
> > I managed to get a thread dump during one such tomcat hangs. Most of
> > the threads are in the following status ...
>
> The ones you show are simply waiting for input from httpd; that's a pretty
> normal state. When looking at a thread dump, it's often the oddball thread
> that's not doing what the rest are that is causing a problem. However, if
> all of the threads are just waiting to receive something from httpd, then
> you may have to look outside of Tomcat for the cause.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you received
> this in error, please contact the sender and delete the e-mail and its
> attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
>
Re: Tomcat not closing threads
Posted by Chetan Chheda <ch...@yahoo.com>.
I guess I need to ask my question again. Im primarily an HP-UX administrator and recently inherited this web based application ...
We are having some issues in accomodating additional user load/functionality. While a part of the team is looking at optimizing code, I am responsible for infrastructure componants.
One particular focus area for me is the mod_jk and tomcat configurations.
1. Last of April saw one of the tomcat's stop processing requests. Looking at the mod_jk.log I saw the following errors
Unable to get the free endpoint for worker XXX from 37 slots ....
I have setup the status servlet for mod_jk in which I saw that max slots were opened. Thinking that mod_jk needs more number of connections, I basically set it to ThreadsPerChild from worker MPM
2. Now we are randomly seeing that even a simple activity as user login causes a spike in used tomcat connector threads. Eventually all tomcat threads are used and no new requests can be serviced.
So, long story short, is my apache, mod_jk and tomcat configuration in sync?
Maxclients = 512
ThreadsPerChild = 256
So that means 2 server processes with 256 threads each.
I have 2 worker threads in my worker.properties file with connection_pool_size=256.
Does this mean that total number of connections into tomcat = 256 * 2(number of workers) * 2(number of server procs) = 1024 ??
Do I need to change the connection_pool_size to 120 . Which means 120*2 *2 = 480 and that leaves the remainder of apache threads for static content?
3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in server.xml . Is this why connections remain open?
Thanks,
Chetan
________________________________
From: "Caldarale, Charles R" <Ch...@unisys.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Tuesday, May 12, 2009 10:47:00 PM
Subject: RE: Tomcat not closing threads
> From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> Subject: Re: Tomcat not closing threads
>
> I managed to get a thread dump during one such tomcat hangs. Most of
> the threads are in the following status ...
The ones you show are simply waiting for input from httpd; that's a pretty normal state. When looking at a thread dump, it's often the oddball thread that's not doing what the rest are that is causing a problem. However, if all of the threads are just waiting to receive something from httpd, then you may have to look outside of Tomcat for the cause.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat not closing threads
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Chetan Chheda [mailto:chetan_chheda@yahoo.com]
> Subject: Re: Tomcat not closing threads
>
> I managed to get a thread dump during one such tomcat hangs. Most of
> the threads are in the following status ...
The ones you show are simply waiting for input from httpd; that's a pretty normal state. When looking at a thread dump, it's often the oddball thread that's not doing what the rest are that is causing a problem. However, if all of the threads are just waiting to receive something from httpd, then you may have to look outside of Tomcat for the cause.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat not closing threads
Posted by Chetan Chheda <ch...@yahoo.com>.
I managed to get a thread dump during one such tomcat hangs. Most of the threads are in the following status ...
"TP-Processor7" daemon prio=10 tid=00093ab0 nid=86 lwp_id=4866290 runnable [53c00000..53c00dc0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:134)
at java.io.FilterInputStream.read(FilterInputStream.java:111)
at com.wily.introscope.agent.probe.net.ManagedSocketInputStream.read(ManagedSocketInputStream.java:214)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <8e0d3a50> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:598)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:535)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:663)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:534)
"TP-Processor6" daemon prio=10 tid=000938d0 nid=85 lwp_id=4866288 runnable [53cc0000..53cc0dc0]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:134)
at java.io.FilterInputStream.read(FilterInputStream.java:111)
at com.wily.introscope.agent.probe.net.ManagedSocketInputStream.read(ManagedSocketInputStream.java:214)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:183)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:222)
at java.io.BufferedInputStream.read(BufferedInputStream.java:277)
- locked <8e0d7ed8> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:598)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:535)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:663)
at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:534)
________________________________
From: Chetan Chheda <ch...@yahoo.com>
To: Tomcat Users List <us...@tomcat.apache.org>
Sent: Tuesday, May 12, 2009 1:32:16 PM
Subject: Re: Tomcat not closing threads
Just wondering if anyone got my message below?
________________________________
From: Chetan Chheda <ch...@yahoo.com>
To: users@tomcat.apache.org
Sent: Tuesday, May 12, 2009 12:03:49 PM
Subject: Tomcat not closing threads
Hello,
I am having an issue where I could use some clarity.
We are using apache 2.0.59 and tomcat 5.5. We are having some growing pains with vendor specified modjk and tomcat settings.
1. Tomcat opens up a number of threads during some transactions but does not close them. Is this because the connectionTimeout parameter is missing in the AJP connector defination?
2. I do have the cache_timeout (deprecated , I know ) set to 900 secs. What impact does that have on closing idle threads?
3. Today one of the Tomcat's froze and thru jkstatus I saw 409 connections open to it. If my connection pool size is 256 (from modjk.log) why are more connections being opened?
Thanks,
Chetan
My workers.properties file looks like this.
ps=/
worker.list=ajp13App31,ajp13App32,loadbalancer,mystatus
workers.tomcat_home=/coach/app/ptc/tomcat1
worker.ajp13App31.port=8012
worker.ajp13App31.host=localhost
worker.ajp13App31.type=ajp13
worker.ajp13App31.lbfactor=1
worker.ajp13App31.cache_timeout=900
worker.ajp13App31.socket_timeout=0
worker.ajp13App31.retries=3
worker.ajp13App31.recycle_timeout=900
worker.ajp13App32.port=8013
worker.ajp13App32.host=localhost
worker.ajp13App32.type=ajp13
worker.ajp13App32.lbfactor=1
worker.ajp13App32.cache_timeout=900
worker.ajp13App32.socket_timeout=0
worker.ajp13App32.retries=3
worker.ajp13App32.recycle_timeout=900
worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=ajp13App31,ajp13App32
worker.loadbalancer.method=Request
worker.loadbalancer.sticky_session=true
worker.inprocess.class_path=$(workers.tomcat_home)$(ps)lib$(ps)tomcat.jar
worker.inprocess.cmd_line=start
worker.inprocess.stdout=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stdout
worker.inprocess.stderr=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stderr
worker.mystatus.type=status
worker.mystatus.mount=/admin/status/jk
And here is the AJP13 connector from one of the tomcat's server.xml :
<Connector URIEncoding="UTF-8" acceptCount="100" debug="1" port="8012" protocol="AJP/1.3" protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler" redirectPort="8445" tomcatAuthentication="false" useBodyEncodingForURI="true">
Apache MPM
# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
ThreadLimit 512
StartServers 100
MaxClients 512
MinSpareThreads 129
MaxSpareThreads 256
ThreadsPerChild 256
MaxRequestsPerChild 0
</IfModule>
Re: Tomcat not closing threads
Posted by Chetan Chheda <ch...@yahoo.com>.
Just wondering if anyone got my message below?
________________________________
From: Chetan Chheda <ch...@yahoo.com>
To: users@tomcat.apache.org
Sent: Tuesday, May 12, 2009 12:03:49 PM
Subject: Tomcat not closing threads
Hello,
I am having an issue where I could use some clarity.
We are using apache 2.0.59 and tomcat 5.5. We are having some growing pains with vendor specified modjk and tomcat settings.
1. Tomcat opens up a number of threads during some transactions but does not close them. Is this because the connectionTimeout parameter is missing in the AJP connector defination?
2. I do have the cache_timeout (deprecated , I know ) set to 900 secs. What impact does that have on closing idle threads?
3. Today one of the Tomcat's froze and thru jkstatus I saw 409 connections open to it. If my connection pool size is 256 (from modjk.log) why are more connections being opened?
Thanks,
Chetan
My workers.properties file looks like this.
ps=/
worker.list=ajp13App31,ajp13App32,loadbalancer,mystatus
workers.tomcat_home=/coach/app/ptc/tomcat1
worker.ajp13App31.port=8012
worker.ajp13App31.host=localhost
worker.ajp13App31.type=ajp13
worker.ajp13App31.lbfactor=1
worker.ajp13App31.cache_timeout=900
worker.ajp13App31.socket_timeout=0
worker.ajp13App31.retries=3
worker.ajp13App31.recycle_timeout=900
worker.ajp13App32.port=8013
worker.ajp13App32.host=localhost
worker.ajp13App32.type=ajp13
worker.ajp13App32.lbfactor=1
worker.ajp13App32.cache_timeout=900
worker.ajp13App32.socket_timeout=0
worker.ajp13App32.retries=3
worker.ajp13App32.recycle_timeout=900
worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=ajp13App31,ajp13App32
worker.loadbalancer.method=Request
worker.loadbalancer.sticky_session=true
worker.inprocess.class_path=$(workers.tomcat_home)$(ps)lib$(ps)tomcat.jar
worker.inprocess.cmd_line=start
worker.inprocess.stdout=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stdout
worker.inprocess.stderr=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stderr
worker.mystatus.type=status
worker.mystatus.mount=/admin/status/jk
And here is the AJP13 connector from one of the tomcat's server.xml :
<Connector URIEncoding="UTF-8" acceptCount="100" debug="1" port="8012" protocol="AJP/1.3" protocolHandlerClassName="org.apache.jk.server.JkCoyoteHandler" redirectPort="8445" tomcatAuthentication="false" useBodyEncodingForURI="true">
Apache MPM
# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
ThreadLimit 512
StartServers 100
MaxClients 512
MinSpareThreads 129
MaxSpareThreads 256
ThreadsPerChild 256
MaxRequestsPerChild 0
</IfModule>