You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Robin Wilson <rw...@kingsisle.com> on 2010/01/08 22:53:01 UTC
When running under high load (via load generating tools), Tomcat
6.0.20 stops expiring sessions until the load subsides; causing thrashing on
Garbage Collection
I am trying to get a clustered Tomcat 6.0.20 environment to operate in a high load environment. As part of the testing regime, we want to be able to run continuous high load against a QA setup - and validate that Tomcat operates continuously without a hiccup.
The SETUP:
1) Apache front-end (don't ask me why - I don't control this part) using AJP through to proxy to the Tomcat cluster.
2) 2 Tomcat hosts 'clustered' (although for the test, I'm only actually hitting one of the tomcats). Both are identically configured, 8-processor, 8GB systems running Tomcat 6.0.20. The Cluster is functioning proper (near as we can tell). The Tomcats have PermSize=1g, Xmx=6g, Xms=3g.
3) 2 "load generators" using JMeter to make requests against the Apache host. They are running 100 threads (each) to request a simple unauthenticated page. They are running 20 threads each to perform "registration" and "login" (the primary difference being that registration and login makes 5 requests per thread, and they end up creating a much larger 'session' object when they are done).
4) Watching the 'session count' using "/manager/html" on the Tomcat that we're hitting.
The "problem" we're seeing:
1) Watching the 'session' count we see the number of sessions increase to about 450,000 over the course of about 40 minutes. (Of those, they lean about 120-to-1 for simple "empty" sessions created by the single-page requests.) The sessions for the single-page requests have an expire after time of '1 second'. Near as we can tell, none of these sessions expire while we are under load.
2) We watch the garbage collection activity by tailing the log file, and notice that after we reach the 450,000 sessions limit, the Tomcat starts to perform Full GCs. The Full GCs take about 20 seconds (even a little more), and there is about 2-5 seconds between the Full GCs, so essentially 80+% of the time is spent doing Full GCs, the Tomcat is effectively 'hung' at this point.
3) EXCEPT, if we stop the load generator (and nothing else is changed), immediately the session count on the Tomcat begins to drop - and Full GCs stop within a couple more cycles. If we wait ~3 minutes, the sessions drop down to only the sessions that haven't reached their inactivity timeout - and we can restart the load generators and go again for another 40 mins (or so).
The question:
Is there a setting we're missing that tells Tomcat to make session expiration a higher priority?
--
Robin D. Wilson
Director of Web Development
KingsIsle Entertainment, Inc.
CELL: 512-426-3929
DESK: 512-623-5913
www.KingsIsle.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: When running under high load (via load generating tools), Tomcat
6.0.20 stops expiring sessions until the load subsides; causing thrashing
on Garbage Collection
Posted by Mark Thomas <ma...@apache.org>.
On 09/01/2010 15:50, Robin Wilson wrote:
> Well, we encounter this problem in our production environment every so often. If we run at sustained load for 2 hours we hit the wall in production. Keep in mind, our production environment has 4 clustered servers, so it takes about 1/4 the amount of load per server to encounter the problem.
Fair enough - if the load test is just a quick way of reproducing the
problem then it doesn't really matter if it is artificial.
> That being said, I think you are correct, we are creating sessions faster than they can be destroyed.
>
> Is there a way to setup the context so that it doesn't automatically create a session? (Keep in mind, we aren't using JSP, we are using Tapestry - and I am not sure how to use Tapestry without a session.)
Unfortunately, that will be under the control of the web application -
or in this case Tapestry. The Tapestry users list is probably you best
source of info on this.
You can configure JSPs not to create sessions by default in web.xml but
I'm not sure how much - if at all - that would help you.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: When running under high load (via load generating tools),
Tomcat 6.0.20 stops expiring sessions until the load subsides; causing
thrashing on Garbage Collection
Posted by Robin Wilson <rw...@kingsisle.com>.
Well, we encounter this problem in our production environment every so often. If we run at sustained load for 2 hours we hit the wall in production. Keep in mind, our production environment has 4 clustered servers, so it takes about 1/4 the amount of load per server to encounter the problem.
That being said, I think you are correct, we are creating sessions faster than they can be destroyed.
Is there a way to setup the context so that it doesn't automatically create a session? (Keep in mind, we aren't using JSP, we are using Tapestry - and I am not sure how to use Tapestry without a session.)
--
Robin D. Wilson
Director of Web Development
KingsIsle Entertainment, Inc.
CELL: 512-426-3929
DESK: 512-623-5913
www.KingsIsle.com
-----Original Message-----
From: Mark Thomas [mailto:markt@apache.org]
Sent: Saturday, January 09, 2010 5:13 AM
To: Tomcat Users List
Subject: Re: When running under high load (via load generating tools), Tomcat 6.0.20 stops expiring sessions until the load subsides; causing thrashing on Garbage Collection
On 08/01/2010 23:07, Robin Wilson wrote:
> OK, I made the following changes (1 at a time) to 'server.xml', and retested:
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> processExpiresFrequency="1"
> notifyListenersOnReplication="true"/>
>
> No discernable difference was noted on the retest (NOTE: I have measured the sessions vs. the number of 'threads requesting pages', and I think that the sessions are actually expiring - just _much_ slower than my load generator is creating them. So I'm creating 100,000 sessions in 'n' time period, and in that same time period only 6,000 are being expired and removed (the ratio seems pretty constant - ~94% of the sessions created are staying around).
>
> Then I made this change to 'server.xml':
>
> <Engine
> name="tomcat"
> backgroundProcessorDelay="1"
> defaultHost="localhost">
>
> Again, no discernable difference in behavior.
>
> So, new question - is there a way to tell Tomcat to clear out larger numbers of sessions at once? (Assuming they are past their "inactivity duration" period?) Even when I shut off the load generators, it only clears the sessions at a rate of a few thousand every second.
Given it works when the load is reduced, it looks like the expiration
logic is correct. One thing that would give you a bit more information
is setting the log level for org.apache.catalina.session.ManagerBase to
debug.
I wonder if you are simply creating sessions faster than the single
background thread can clear them out. If that is the case, how realistic
is your load? I'm happy to try a fix a genuine issue but I don't think
there is much benefit in making changes to address an artificial problem.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: When running under high load (via load generating tools),
Tomcat 6.0.20 stops expiring sessions until the load subsides; causing
thrashing on Garbage Collection
Posted by Robin Wilson <rw...@kingsisle.com>.
Thanks. I'll pursue that angle - and see if there is a way, in Tapestry to just not create the session.
(If anyone has any suggestions on where I can look for an answer to that - I'm all ears...)
--
Robin D. Wilson
Director of Web Development
KingsIsle Entertainment, Inc.
CELL: 512-426-3929
DESK: 512-623-5913
www.KingsIsle.com
-----Original Message-----
From: Rainer Jung [mailto:rainer.jung@kippdata.de]
Sent: Saturday, January 09, 2010 6:11 AM
To: Tomcat Users List
Subject: Re: When running under high load (via load generating tools), Tomcat 6.0.20 stops expiring sessions until the load subsides; causing thrashing on Garbage Collection
On 09.01.2010 12:13, Mark Thomas wrote:
> On 08/01/2010 23:07, Robin Wilson wrote:
>> OK, I made the following changes (1 at a time) to 'server.xml', and retested:
>>
>> <Manager className="org.apache.catalina.ha.session.DeltaManager"
>> expireSessionsOnShutdown="false"
>> processExpiresFrequency="1"
>> notifyListenersOnReplication="true"/>
>>
>> No discernable difference was noted on the retest (NOTE: I have measured the sessions vs. the number of 'threads requesting pages', and I think that the sessions are actually expiring - just _much_ slower than my load generator is creating them. So I'm creating 100,000 sessions in 'n' time period, and in that same time period only 6,000 are being expired and removed (the ratio seems pretty constant - ~94% of the sessions created are staying around).
>>
>> Then I made this change to 'server.xml':
>>
>> <Engine
>> name="tomcat"
>> backgroundProcessorDelay="1"
>> defaultHost="localhost">
>>
>> Again, no discernable difference in behavior.
>>
>> So, new question - is there a way to tell Tomcat to clear out larger numbers of sessions at once? (Assuming they are past their "inactivity duration" period?) Even when I shut off the load generators, it only clears the sessions at a rate of a few thousand every second.
>
> Given it works when the load is reduced, it looks like the expiration
> logic is correct. One thing that would give you a bit more information
> is setting the log level for org.apache.catalina.session.ManagerBase to
> debug.
>
> I wonder if you are simply creating sessions faster than the single
> background thread can clear them out. If that is the case, how realistic
> is your load? I'm happy to try a fix a genuine issue but I don't think
> there is much benefit in making changes to address an artificial problem.
Seconding Mark: it seems for those "1 second idle timeout" sessions you
are not using the session at all. So why are you generating a huge
number of sessions you are never going to use?
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: When running under high load (via load generating tools), Tomcat
6.0.20 stops expiring sessions until the load subsides; causing thrashing
on Garbage Collection
Posted by Rainer Jung <ra...@kippdata.de>.
On 09.01.2010 12:13, Mark Thomas wrote:
> On 08/01/2010 23:07, Robin Wilson wrote:
>> OK, I made the following changes (1 at a time) to 'server.xml', and retested:
>>
>> <Manager className="org.apache.catalina.ha.session.DeltaManager"
>> expireSessionsOnShutdown="false"
>> processExpiresFrequency="1"
>> notifyListenersOnReplication="true"/>
>>
>> No discernable difference was noted on the retest (NOTE: I have measured the sessions vs. the number of 'threads requesting pages', and I think that the sessions are actually expiring - just _much_ slower than my load generator is creating them. So I'm creating 100,000 sessions in 'n' time period, and in that same time period only 6,000 are being expired and removed (the ratio seems pretty constant - ~94% of the sessions created are staying around).
>>
>> Then I made this change to 'server.xml':
>>
>> <Engine
>> name="tomcat"
>> backgroundProcessorDelay="1"
>> defaultHost="localhost">
>>
>> Again, no discernable difference in behavior.
>>
>> So, new question - is there a way to tell Tomcat to clear out larger numbers of sessions at once? (Assuming they are past their "inactivity duration" period?) Even when I shut off the load generators, it only clears the sessions at a rate of a few thousand every second.
>
> Given it works when the load is reduced, it looks like the expiration
> logic is correct. One thing that would give you a bit more information
> is setting the log level for org.apache.catalina.session.ManagerBase to
> debug.
>
> I wonder if you are simply creating sessions faster than the single
> background thread can clear them out. If that is the case, how realistic
> is your load? I'm happy to try a fix a genuine issue but I don't think
> there is much benefit in making changes to address an artificial problem.
Seconding Mark: it seems for those "1 second idle timeout" sessions you
are not using the session at all. So why are you generating a huge
number of sessions you are never going to use?
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: When running under high load (via load generating tools), Tomcat
6.0.20 stops expiring sessions until the load subsides; causing thrashing
on Garbage Collection
Posted by Mark Thomas <ma...@apache.org>.
On 08/01/2010 23:07, Robin Wilson wrote:
> OK, I made the following changes (1 at a time) to 'server.xml', and retested:
>
> <Manager className="org.apache.catalina.ha.session.DeltaManager"
> expireSessionsOnShutdown="false"
> processExpiresFrequency="1"
> notifyListenersOnReplication="true"/>
>
> No discernable difference was noted on the retest (NOTE: I have measured the sessions vs. the number of 'threads requesting pages', and I think that the sessions are actually expiring - just _much_ slower than my load generator is creating them. So I'm creating 100,000 sessions in 'n' time period, and in that same time period only 6,000 are being expired and removed (the ratio seems pretty constant - ~94% of the sessions created are staying around).
>
> Then I made this change to 'server.xml':
>
> <Engine
> name="tomcat"
> backgroundProcessorDelay="1"
> defaultHost="localhost">
>
> Again, no discernable difference in behavior.
>
> So, new question - is there a way to tell Tomcat to clear out larger numbers of sessions at once? (Assuming they are past their "inactivity duration" period?) Even when I shut off the load generators, it only clears the sessions at a rate of a few thousand every second.
Given it works when the load is reduced, it looks like the expiration
logic is correct. One thing that would give you a bit more information
is setting the log level for org.apache.catalina.session.ManagerBase to
debug.
I wonder if you are simply creating sessions faster than the single
background thread can clear them out. If that is the case, how realistic
is your load? I'm happy to try a fix a genuine issue but I don't think
there is much benefit in making changes to address an artificial problem.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: When running under high load (via load generating tools),
Tomcat 6.0.20 stops expiring sessions until the load subsides; causing
thrashing on Garbage Collection
Posted by Robin Wilson <rw...@kingsisle.com>.
OK, I made the following changes (1 at a time) to 'server.xml', and retested:
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
processExpiresFrequency="1"
notifyListenersOnReplication="true"/>
No discernable difference was noted on the retest (NOTE: I have measured the sessions vs. the number of 'threads requesting pages', and I think that the sessions are actually expiring - just _much_ slower than my load generator is creating them. So I'm creating 100,000 sessions in 'n' time period, and in that same time period only 6,000 are being expired and removed (the ratio seems pretty constant - ~94% of the sessions created are staying around).
Then I made this change to 'server.xml':
<Engine
name="tomcat"
backgroundProcessorDelay="1"
defaultHost="localhost">
Again, no discernable difference in behavior.
So, new question - is there a way to tell Tomcat to clear out larger numbers of sessions at once? (Assuming they are past their "inactivity duration" period?) Even when I shut off the load generators, it only clears the sessions at a rate of a few thousand every second.
--
Robin D. Wilson
Director of Web Development
KingsIsle Entertainment, Inc.
CELL: 512-426-3929
DESK: 512-623-5913
www.KingsIsle.com
-----Original Message-----
From: Robin Wilson [mailto:rwilson@kingsisle.com]
Sent: Friday, January 08, 2010 4:10 PM
To: Tomcat Users List
Subject: RE: When running under high load (via load generating tools), Tomcat 6.0.20 stops expiring sessions until the load subsides; causing thrashing on Garbage Collection
Thanks, I'll give that a try.
Just out of curiosity, what is the downside of doing this? (Performance hit, ???)
--
Robin D. Wilson
Director of Web Development
KingsIsle Entertainment, Inc.
CELL: 512-426-3929
DESK: 512-623-5913
www.KingsIsle.com
-----Original Message-----
From: Mark Thomas [mailto:markt@apache.org]
Sent: Friday, January 08, 2010 4:07 PM
To: Tomcat Users List
Subject: Re: When running under high load (via load generating tools), Tomcat 6.0.20 stops expiring sessions until the load subsides; causing thrashing on Garbage Collection
On 08/01/2010 21:53, Robin Wilson wrote:
> Is there a setting we're missing that tells Tomcat to make session expiration a higher priority?
Session expirations are processed every Engine.backgroundProcessorDelay
* Manager.processExpiresFrequency seconds which by default is 10 * 6 =
60 seconds
I'd suggest reducing processExpiresFrequency first. If that is set to 1
and you still have issues then look at reducing backgroundProcessorDelay.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: When running under high load (via load generating tools),
Tomcat 6.0.20 stops expiring sessions until the load subsides; causing
thrashing on Garbage Collection
Posted by Robin Wilson <rw...@kingsisle.com>.
Thanks, I'll give that a try.
Just out of curiosity, what is the downside of doing this? (Performance hit, ???)
--
Robin D. Wilson
Director of Web Development
KingsIsle Entertainment, Inc.
CELL: 512-426-3929
DESK: 512-623-5913
www.KingsIsle.com
-----Original Message-----
From: Mark Thomas [mailto:markt@apache.org]
Sent: Friday, January 08, 2010 4:07 PM
To: Tomcat Users List
Subject: Re: When running under high load (via load generating tools), Tomcat 6.0.20 stops expiring sessions until the load subsides; causing thrashing on Garbage Collection
On 08/01/2010 21:53, Robin Wilson wrote:
> Is there a setting we're missing that tells Tomcat to make session expiration a higher priority?
Session expirations are processed every Engine.backgroundProcessorDelay
* Manager.processExpiresFrequency seconds which by default is 10 * 6 =
60 seconds
I'd suggest reducing processExpiresFrequency first. If that is set to 1
and you still have issues then look at reducing backgroundProcessorDelay.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: When running under high load (via load generating tools), Tomcat
6.0.20 stops expiring sessions until the load subsides; causing thrashing
on Garbage Collection
Posted by Mark Thomas <ma...@apache.org>.
On 08/01/2010 21:53, Robin Wilson wrote:
> Is there a setting we're missing that tells Tomcat to make session expiration a higher priority?
Session expirations are processed every Engine.backgroundProcessorDelay
* Manager.processExpiresFrequency seconds which by default is 10 * 6 =
60 seconds
I'd suggest reducing processExpiresFrequency first. If that is set to 1
and you still have issues then look at reducing backgroundProcessorDelay.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org