Problems with Tomcat in a high load environment

[pe...@kisstechnologies.co.uk]

Hi,
we are running Tomcat 4.1.18 on Windows 2000 under JDK1.4.1_01.  We are 
using IIS as a web server connecting using the ISAPI JK 2 connector.  We 
are currently experiencing 2 seperate problems when under high load (ie > 
30 requests per second):

1.  A huge number of log events (~30 sets per minute) are generated in the 
NT event log which show problems with the ISAPI connector.  Each time 
there is a problem, 7 lines are created:

Error: [jk_worker_ajp13.c (512)]: ajp13.service() Error  forwarding 
ajp13:localhost:8029 1 0
Error: [jk_worker_ajp13.c (416)]: ajp13.service() ajpGetReply recoverable 
error 3
Error: [jk_handler_response.c (178)]: handler.response() Error sending 
response 
Error: [jk_service_iis.c (157)]: jk_ws_service_t::head, 
ServerSupportFunction failed
Error: [jk_worker_ajp13.c (416)]: ajp13.service() ajpGetReply recoverable 
error 3
Error: [jk_handler_response.c (200)]: Error ajp_process_callback - write 
failed
Error: [jk_service_iis.c (247)]: jk_ws_service_t::write, WriteClient 
failed

These I think lead to the eventual crash of IIS however I also suspect 
that they may cause user errors though I havent actually seen any evidence 
of this.  Does anyone know anything about these errors or what I can do to 
reduce them?  Could it be a tuning of Tomcat issue?

2. The server is set up currently with -Xms512m and the same for Xmx. 
This, I would have thought, was OK for this application, though of course 
I could be wrong, but is in any case irrelevant to the problem as if I 
increase memory to 1Gb it makes no odds.
The server will run fine for a few minutes at about 128m of memory usage. 
At some (slightly random point with no obvious trigger), it will in a 
matter of seconds use up all available memory, thus triggering a huge rise 
in the processor usage which sits at roughly 50-60% (across 2 processors) 
constantly.  It is worth noting that 1 processor is not maxed out - the 
load is relatively evenly distributed.
After a further while, the young generation space runs out of memory and a 
process ensues of the processor load "bouncing" up to 100% and back to 60% 
over and over again as it GCs, reduces the momery used to ~ 15m less than 
it was, then it is used up again and so on.  This obviously seriously 
impacts the usage of the application.  I cant see why it is doing it; this 
is an ecommerce application and the loads are not that high - clearly a 
leak of some description is occuring somewhere but the speed with which 
these changes happen baffle me, and I dont thik there is much setup work I 
can do to change it.  I could install a profiler to find out whats going 
on but it is a live system and I am loath to do so.
Finally it is worth noting that I have only just made this system live and 
a functionally identical although architecturally simpler version ran 
quite happily under the same loads using no more than 256m of ram under 
JRun 3.1.

Any ideas?!

cheers
Pete

Kiss Technologies

http://www.kisstechnologies.co.uk/

4, Percy Street
London
W1T 1DF

Phone numbers:

Phone 020 7692 9922
Fax 020 7692 9923