Re: Apache, Tomcat, Mod_JK Configurations

[CrystalCracker <su...@gmail.com>]

Ok.. adding some info to mod jk logs and doing some thread and heap dumps, I
figured that, during peak traffic hours, the garbage collection gets more
frequest and takes longer. This slows down the overall system and the
threads start piling up. Requests that used to take 2 seconds now take a
couple of hundred seconds. The application isn't using the swap space
though. It takes a long as 15 minutes for the system to recover completely
and sometimes I have to restart.

Now how can I intelligently use the timeout settings to kill the long
running requests and prevent threads from making it worse? 

In httpd.conf, I have:
Timeout 120

In worker.properties, I have: 
worker.node1.connection_pool_timeout=600

In the tomcat AJP connector, I have:
connectionTimeout="600000" 

How can I ensure that no request is taking longer than 3 minutes?

-- 
View this message in context: http://www.nabble.com/Apache%2C-Tomcat%2C-Mod_JK-Configurations-tp23751902p23817845.html
Sent from the Tomcat - User mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Apache, Tomcat, Mod_JK Configurations

[Rainer Jung <ra...@kippdata.de>]

On 01.06.2009 18:30, CrystalCracker wrote:
> Ok.. adding some info to mod jk logs and doing some thread and heap dumps, I
> figured that, during peak traffic hours, the garbage collection gets more
> frequest and takes longer. This slows down the overall system and the
> threads start piling up. Requests that used to take 2 seconds now take a
> couple of hundred seconds. The application isn't using the swap space
> though. It takes a long as 15 minutes for the system to recover completely
> and sometimes I have to restart.
> 
> Now how can I intelligently use the timeout settings to kill the long
> running requests and prevent threads from making it worse? 
> 
> In httpd.conf, I have:
> Timeout 120
> 
> In worker.properties, I have: 
> worker.node1.connection_pool_timeout=600
> 
> In the tomcat AJP connector, I have:
> connectionTimeout="600000" 
> 
> How can I ensure that no request is taking longer than 3 minutes?

First of all there is no direct way of "killing" the threads inside
Tomcat. Once your requests get there, you can't stop them from
processing by any kind of timeout set in Apache or mod_jk.

So most important: Analyze and fix your memory problem/sizing. That's
the only way that will help you really.

Apart from that, the question is, what kind of reaction do you want?

a) Apache should not wait longer than xxx seconds for a response and
then return an error to the client, thus freeing its own resources, but
Tomcat will proceed with the request until it starts sending back the
response, then noticing that the web server is gone.

b) Like a), but if this happens to often, Apache should send no more
requests to the node until it gets fast again, in order to shorten the
bad period. The requests which fail over to some other node will likely
produce an error, if they use a web session (that does not exist on the
other nodes).

c) Send all requests without sessions to some other node, once a node
goes bad.

Ad c): You can use the lb method "Busyness" in mod_jk. It will choose
the node with the fewest concurrent requests running for the next
request (request without session). Beware that a few users reported a
problem with Busyness counters slowly getting out of sync with real
concurrent requests over time.

Ad a) You can set a reply_timeout. If you do not want the complete node
being taken out of balancing after a single reply_timeout, you should
also set max_reply_timeouts to a big value. Use a recent mod_jk version,
if you are using those attributes.

Ad b): Like a, but fine tune max_reply_timeouts. Rule of thumb:
max_reply_timeout should be double the value of reply_timeouts you want
to accept, before taking a node fully our of balancing.

Note: with recent mod_jk you can set reply timeouts also per URL, e.g.
in case you know that some URLs run really long (reporting etc.) and you
want to have e.g. 30 seconds general timeout, but 2 minutes for some
well known special URLs. This can be done by using either the Apache
environment variable JK_REPLY_TIMEOUT, or alternatively the so-called
rule extensions of uriworkermap.properties:

http://tomcat.apache.org/connectors-doc/reference/uriworkermap.html#Rule%20extensions

Regards,

Rainer

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org