ResourceLimitingPool and XMLSerializers

[Tor-Einar Jarnbjo <To...@mobile-consulting.net>]

Hi,

we are having a rather strange problem with a Cocoon based web application,
which seem to be load dependent and we think we have found the problem 
to be related to XMLSerializers not being handed back to the ResourceLimitingPool 
when processing requests.

We've added a few log outputs to the pool's get and put methods and 
found, that at some point, Cocoon seems to stop returning XMLSerializers 
to the pool (or at least does it only very rarely). As soon as this 
happens, the pool starts creating new instances of the XMLSerializer 
and this is obviously a very expensive operation, which among other 
things includes reading files from the harddisk. This causes Tomcat's 
thread pool to drain and in effect the system to stop working at 
all.


One such "crash" could look like this (with the pool size set to 1024)

At 14:52:49, the pool has 249 instances created, of which 190 are 
waiting in the pool.

Within the next 30 seconds, only 15 serializers are returned to the 
pool, but 205 are requested, causing the pool to be empty at 14:53:
20.

At 14:53:24 another three serializers are returned to the pool, but 
in the meantime 11 serializers have been requested, causing the pool 
to grow to a size of 257 instances.

The next serializers are now not returned to the pool until 14:54:
57 (90 seconds later). In the meantime, the pool has grown to 350 
instances and Tomcat's thread pool is already drained, as the server 
is not able to server the requests fast enough.

In the following three minutes until 14:57:10, there are 80 serializers 
returned to the pool, almost 400 are beeing requested and the pool 
has grown to 650 instances.

In the next two minutes until 14:59:00, the "tide" changes and there 
are about 150 serializers returned, but only 50 requested, so that 
the pool stays the same size and now has about 100 serializers cached.


>From now, there are again almost no serializers returned to the pool 
and it reaches a size of 1000 instances at 15:03:00. We restarted 
the server around 15:04:30 as it did not seem to recover from the 
situation.


Such episodes are now occuring about once an hour or so when the 
server is running under high load. We have a load balancing system 
in front of two servers, but as soon as one server "hangs" and all 
the requests are redirected to the other server, this one will also 
fail with the same symptomes within less than a minute.

To put my question short: Does someone have an idea why this is happening 
and what we could do to solve the problem?


Tor







===================================================================
EASY and FREE access to your email anywhere: http://Mailreader.com/
===================================================================



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org