You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Rob G <ro...@gmail.com> on 2010/10/07 18:59:21 UTC
Error 503 ocurring when server under load
Hey all,
Recently migrated a production site (mixture of Servlets and JSPs)
from Oracle Application Server to Apache/Tomcat. Since then we have
seen numerous HTTP Error 503 - Service unavailable errors at peak
times when site is under load. mod_jk.log has the following error
message(s):
[2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
failed, no more workers left
I'm looking for help in trying to tweak settings to prevent this, or
confirmation that I've configured the setup correctly.
Many thanks
Rob
Platform:
Windows Server 2003 SP2
Setup:
Two tomcat instances with a single Apache front end, all on the same server
Versions
Tomcat: 6.0.24
Apache: 2.2.16
mod SSL: 2.2.16
Open SSL: 0.9.8
mod_JK:1.2.30
Conf:
httpd.conf:
<IfModule mod_jk.c>
JkWorkersFile conf/workers.properties
JkLogFile logs/mod_jk.log
JkLogLevel error
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
JkRequestLogFormat "%w %V %T"
<Location /*/WEB-INF/*>
AllowOverride None
deny from all
</Location>
JkMount /examples balancer
JkMount /examples/* balancer
JkMount /App1/* balancer
JkMount /App2/* balancer
</IfModule>
Workers.properties:
workers.tomcat_home="C:\tomcat1"
workers.java_home="C:\Program Files\Java\jdk1.6.0_18"
ps=\
# Define workers
#worker.list=worker1,worker2
worker.list=balancer
# Set properties for worker worker1 (ajp13)
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009
worker.worker1.lbfactor=1
worker.worker1.cachesize=10
worker.worker1.connection_pool_timeout=300
worker.worker1.socket_keepalive=FALSE
#worker.worker1.recycle_timeout=300
# Set properties for worker worker2 (ajp13)
worker.worker2.type=ajp13
worker.worker2.host=localhost
worker.worker2.port=8019
worker.worker2.lbfactor=1
worker.worker2.cachesize=10
worker.worker2.connection_pool_timeout=300
worker.worker2.socket_keepalive=FALSE
#worker.worker2.recycle_timeout=300
worker.balancer.type=lb
worker.balancer.balance_workers=worker1,worker2
worker.balancer.method=R
worker.balancer.sticky_session=True
Server.xml (same for both tomcat instances apart from jvmRoute):
<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />
<Listener className="org.apache.catalina.core.JasperListener" />
<Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
/>
<Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
<Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
/>
<GlobalNamingResources>
<Resource name="UserDatabase" auth="Container"
type="org.apache.catalina.UserDatabase"
description="User database that can be updated and saved"
factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
pathname="conf/tomcat-users.xml" />
</GlobalNamingResources>
<Service name="Catalina">
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="40000"
redirectPort="8443" />
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost" jvmRoute="worker1">
<Realm className="org.apache.catalina.realm.UserDatabaseRealm"
resourceName="UserDatabase"/>
<Host name="localhost" appBase="webapps"
unpackWARs="true" autoDeploy="true"
xmlValidation="false" xmlNamespaceAware="false">
</Host>
</Engine>
</Service>
</Server>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Error 503 ocurring when server under load
Posted by Pid <pi...@pidster.com>.
On 07/10/2010 17:59, Rob G wrote:
> Hey all,
>
> Recently migrated a production site (mixture of Servlets and JSPs)
> from Oracle Application Server to Apache/Tomcat. Since then we have
> seen numerous HTTP Error 503 - Service unavailable errors at peak
> times when site is under load. mod_jk.log has the following error
> message(s):
>
> [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
> failed, no more workers left
>
> I'm looking for help in trying to tweak settings to prevent this, or
> confirmation that I've configured the setup correctly.
>
> Many thanks
> Rob
>
> Platform:
> Windows Server 2003 SP2
>
> Setup:
> Two tomcat instances with a single Apache front end, all on the same server
>
> Versions
> Tomcat: 6.0.24
> Apache: 2.2.16
> mod SSL: 2.2.16
> Open SSL: 0.9.8
> mod_JK:1.2.30
>
> Conf:
> httpd.conf:
>
> <IfModule mod_jk.c>
>
> JkWorkersFile conf/workers.properties
> JkLogFile logs/mod_jk.log
> JkLogLevel error
> JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
> JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
> JkRequestLogFormat "%w %V %T"
>
> <Location /*/WEB-INF/*>
> AllowOverride None
> deny from all
> </Location>
>
> JkMount /examples balancer
> JkMount /examples/* balancer
> JkMount /App1/* balancer
> JkMount /App2/* balancer
> </IfModule>
>
> Workers.properties:
> workers.tomcat_home="C:\tomcat1"
> workers.java_home="C:\Program Files\Java\jdk1.6.0_18"
> ps=\
>
> # Define workers
> #worker.list=worker1,worker2
> worker.list=balancer
>
> # Set properties for worker worker1 (ajp13)
> worker.worker1.type=ajp13
> worker.worker1.host=localhost
> worker.worker1.port=8009
> worker.worker1.lbfactor=1
>
> worker.worker1.cachesize=10
> worker.worker1.connection_pool_timeout=300
> worker.worker1.socket_keepalive=FALSE
> #worker.worker1.recycle_timeout=300
>
> # Set properties for worker worker2 (ajp13)
> worker.worker2.type=ajp13
> worker.worker2.host=localhost
> worker.worker2.port=8019
> worker.worker2.lbfactor=1
>
> worker.worker2.cachesize=10
> worker.worker2.connection_pool_timeout=300
> worker.worker2.socket_keepalive=FALSE
> #worker.worker2.recycle_timeout=300
>
> worker.balancer.type=lb
> worker.balancer.balance_workers=worker1,worker2
> worker.balancer.method=R
> worker.balancer.sticky_session=True
>
> Server.xml (same for both tomcat instances apart from jvmRoute):
> <?xml version='1.0' encoding='utf-8'?>
> <Server port="8005" shutdown="SHUTDOWN">
>
> <Listener className="org.apache.catalina.core.AprLifecycleListener"
> SSLEngine="on" />
> <Listener className="org.apache.catalina.core.JasperListener" />
> <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener"
> />
> <Listener className="org.apache.catalina.mbeans.ServerLifecycleListener" />
> <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
> />
>
>
> <GlobalNamingResources>
>
> <Resource name="UserDatabase" auth="Container"
> type="org.apache.catalina.UserDatabase"
> description="User database that can be updated and saved"
> factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
> pathname="conf/tomcat-users.xml" />
> </GlobalNamingResources>
>
> <Service name="Catalina">
>
> <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="40000"
> redirectPort="8443" />
>
> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
>
> <Engine name="Catalina" defaultHost="localhost" jvmRoute="worker1">
>
> <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
> resourceName="UserDatabase"/>
>
> <Host name="localhost" appBase="webapps"
> unpackWARs="true" autoDeploy="true"
> xmlValidation="false" xmlNamespaceAware="false">
>
> </Host>
> </Engine>
> </Service>
> </Server>
[Answers own previous question] Weirdness.
In addition to the previous relevant questions:
Are both of the two Tomcat instances actually working? If the shutdown
port is identical on both (8005), then I think the second one won't
start up. The connectors certainly won't.
Also, if you're using mod_jk 1.2.30 there's an example config in the
install dir which uses the template config method and has good default
settings.
p
Re: Error 503 ocurring when server under load
Posted by Pid <pi...@pidster.com>.
On 08/10/2010 10:08, André Warnier wrote:
> Pid wrote:
> ...
>>
>> Is this info in the original email?
> Yes.
>
> (If so, I think my mailer isn't
>> behaving well).
>>
> It seems also somewhat spastic. We've been receiving multiple copies of
> what you send.
> Or did you just have too much coffee lately ?
Hmm. My apologies.
I was on a tiny & unreliable network connection; I think client & server
dropped & restarted transactions.
p
Re: Error 503 ocurring when server under load
Posted by André Warnier <aw...@ice-sa.com>.
Pid wrote:
...
>
> Is this info in the original email?
Yes.
(If so, I think my mailer isn't
> behaving well).
>
It seems also somewhat spastic. We've been receiving multiple copies of what you send.
Or did you just have too much coffee lately ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Error 503 ocurring when server under load
Posted by Pid <pi...@pidster.com>.
On 07/10/2010 23:17, Christopher Schultz wrote:
> Rob,
>
> On 10/7/2010 12:59 PM, Rob G wrote:
>> Recently migrated a production site (mixture of Servlets and JSPs)
>> from Oracle Application Server to Apache/Tomcat. Since then we have
>> seen numerous HTTP Error 503 - Service unavailable errors at peak
>> times when site is under load. mod_jk.log has the following error
>> message(s):
>
>> [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
>> failed, no more workers left
>
> Just to reinforce a comment by Andre, I think you might want to compare
> the request processor counts you have for Tomcat versus mod_jk: the
> default maxThreads for a Tomcat <Connector> is 200, while your
> MaxClients settings might be different on your Apache httpd configuration.
>
> If you have httpd.MaxClients=250 and connector.maxThreads=200 then you
> have a deficit of 50 who will get 503 errors if they try to access your
> webapp during peak load.
>
>> Apache: 2.2.16
>
> I believe that only the Worker MPM is available on Microsoft Windows.
> Have you explicitly configured an MPM?
>
>> worker.worker1.cachesize=10
Is this info in the original email? (If so, I think my mailer isn't
behaving well).
p
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Error 503 ocurring when server under load
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Rob,
On 10/7/2010 12:59 PM, Rob G wrote:
> Recently migrated a production site (mixture of Servlets and JSPs)
> from Oracle Application Server to Apache/Tomcat. Since then we have
> seen numerous HTTP Error 503 - Service unavailable errors at peak
> times when site is under load. mod_jk.log has the following error
> message(s):
>
> [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
> failed, no more workers left
Just to reinforce a comment by Andre, I think you might want to compare
the request processor counts you have for Tomcat versus mod_jk: the
default maxThreads for a Tomcat <Connector> is 200, while your
MaxClients settings might be different on your Apache httpd configuration.
If you have httpd.MaxClients=250 and connector.maxThreads=200 then you
have a deficit of 50 who will get 503 errors if they try to access your
webapp during peak load.
> Apache: 2.2.16
I believe that only the Worker MPM is available on Microsoft Windows.
Have you explicitly configured an MPM?
> worker.worker1.cachesize=10
You might want to read the documentation for worker.cachesize in
http://tomcat.apache.org/connectors-doc/reference/workers.html
Note that it's deprecated and you should use connection_pool_size instead.
Since you're using Apache 2.x, mod_jk should be able to auto-configure
this value for you.
- From my reading, you may be limiting yourself to 10 simultaneous
connections from Apache httpd to Tomcat, and others will get a 503.
Rainer or Mladen will have to comment to clarify: I am certainly no
expert in mod_jk configurations.
Hope that helps,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkyuRwUACgkQ9CaO5/Lv0PC3nACePwYgc2w4td8jSdRCM5rp9IB9
PB4AoKj4cR9F+FhsR20W4vIvcWkRevyL
=nuW/
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Error 503 ocurring when server under load
Posted by Pid <pi...@pidster.com>.
On 07/10/2010 18:31, André Warnier wrote:
> Rob G wrote:
>> Hey all,
>>
>> Recently migrated a production site (mixture of Servlets and JSPs)
>> from Oracle Application Server to Apache/Tomcat. Since then we have
>> seen numerous HTTP Error 503 - Service unavailable errors at peak
>> times when site is under load. mod_jk.log has the following error
>> message(s):
OK. Is there anything else different apart from the Servlet container?
>> [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
>> failed, no more workers left
Seems like your Tomcats are maxed out.
>> I'm looking for help in trying to tweak settings to prevent this, or
>> confirmation that I've configured the setup correctly.
>> Platform:
>> Windows Server 2003 SP2
>>
>> Setup:
>> Two tomcat instances with a single Apache front end, all on the same
>> server
Why do you have two Tomcat instances? (It's not a trick question, I'm
interested in your reasoning.)
>> Versions
>> Tomcat: 6.0.24
>> Apache: 2.2.16
>> mod SSL: 2.2.16
>> Open SSL: 0.9.8
There's newer OpenSSL available, with important security fixes, if I'm
not mistaken.
>> mod_JK:1.2.30
>
> Your configuration looks very clean to me (no unnecessary settings
> etc.), which in this case is a plus (a good base to start tuning).
Was there an attachment I didn't see?
> You may want to upgrade Tomcat to the latest version (6.0.29).
+1
> But before you start tuning, you should get some idea of what is
> actually going on.
+1
> For example, at the moment these errors happen, what are these Tomcats
> really doing ?
> Are they really busy each processing 200 requests, with 200 threads
> running and actually doing something ? (200 is the default for the
> "maxThreads" attribute of the AJP Connector).
Q: How many threads & server instances did you have before?
> If yes, then you may just need a leaner application, or a bigger system
> (more RAM, faster CPU), or more systems. What does the Task Manager
> tell you about the total system load ?
What were your Java -Xmx etc settings before, and now?
> If not, and many of these threads are waiting, then you may have an
> issue with a keepAlive that is too long.
Or several other things.
What is your Connector config?
Are you using an Executor?
Is there a database behind this, if so, what are the DataSource pool
size settings?
> See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the
> connectionTimeout and keepAliveTimeout attributes.
>
> Whatever you do, first get an idea of the starting situation. Then
> modify one setting at a time, and observe (and note) the effects.
Thread dumps from a maxed out Tomcat will tell you what each Thread is
waiting for. Collect a series of these during high load periods to find
out what's happening.
Enable JMX and check the Connector, (Executor if enabled), and
DataSources; I'd be looking at backlog of requests, active + idle pool
members, total pool size.
p
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: Error 503 ocurring when server under load
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Jeffrey,
On 10/12/2010 3:22 PM, Jeffrey Janner wrote:
> I just occurred to me that I don't think anyone's asked if these are
> net-mounted file systems. I've seen this timestamp-shifting before,
> but only on net-mounted filesystems. Usually the source and local
> systems are set to different timezones (or DST settings).
That's a very good thought.
I've never seen good things happen when using a network filesystem for
webapp deployments. Feel free to store your data on NFS, just don't use
NFS to store your WAR files ;)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAky0xJcACgkQ9CaO5/Lv0PCEQwCfdh1CXe7Z0MYqZPpgpsDdsoGA
838An1uHbLlpeQxNTqnF44zl04rqnE2d
=PgBy
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Error 503 ocurring when server under load
Posted by Jeffrey Janner <Je...@PolyDyne.com>.
I just realized that I somehow replied to the wrong thread on this one.
It was meant for Jane Muse's thread.
I'll repost there in case someone missed it.
Sorry to interrupt your thread with irrelevant information/spleculation.
Jeff
> -----Original Message-----
> From: Jeffrey Janner [mailto:Jeffrey.Janner@PolyDyne.com]
> Sent: Tuesday, October 12, 2010 2:23 PM
> To: Tomcat Users List
> Subject: RE: Error 503 ocurring when server under load
>
> I just occurred to me that I don't think anyone's asked if these are
> net-mounted file systems. I've seen this timestamp-shifting before,
> but only on net-mounted filesystems. Usually the source and local
> systems are set to different timezones (or DST settings).
>
> > -----Original Message-----
> > From: Christopher Schultz [mailto:chris@christopherschultz.net]
> > Sent: Tuesday, October 12, 2010 1:47 PM
> > To: Tomcat Users List
> > Subject: Re: Error 503 ocurring when server under load
> >
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Rob,
> >
> > On 10/11/2010 4:40 PM, Rob G wrote:
> > > So if I'm reading your email and the docs correctly. I should just
> > > comment out the cachesize=10 from the workers.properties.
> >
> > I would.
> >
> > > And since for connection_pool_size (that replaced it) JK will
> > > discover this number for the Apache web server automatically and
> set
> > > the pool size to this value, I don't need to add anything to the
> > > workers.properties file?
> >
> > I believe that is true.
> >
> > On the other hand, there is another case where you might have
> problems.
> > If you have, say, 512 worker threads in Apache httpd but you only
> have,
> > say, 200 request processor threads configured in Tomcat, then you
> will
> > get mod_jk connection failures on the httpd side.
> >
> > I would recommend that you have enough request processors configured
> in
> > Tomcat to handle the expected load.
> >
> > - -chris
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.10 (MingW32)
> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> >
> > iEYEARECAAYFAky0rToACgkQ9CaO5/Lv0PCvjACgwttZ9YfINxpWP+DI1+VlKfvI
> > OTAAoK+2RQRibL56GdYWlaWxx6obZVln
> > =hY3s
> > -----END PGP SIGNATURE-----
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
> _______________________________________________________________________
> ___
>
> Confidentiality Notice: This Transmission (including any attachments)
> may contain information that is privileged, confidential, and exempt
> from disclosure under applicable law. If the reader of this message is
> not the intended recipient you are hereby notified that any
> dissemination, distribution, or copying of this communication is
> strictly prohibited.
>
> If you have received this transmission in error, please immediately
> reply to the sender or telephone (512) 343-9100 and delete this
> transmission from your system.
__________________________________________________________________________
Confidentiality Notice: This Transmission (including any attachments) may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited.
If you have received this transmission in error, please immediately reply to the sender or telephone (512) 343-9100 and delete this transmission from your system.
RE: Error 503 ocurring when server under load
Posted by Jeffrey Janner <Je...@PolyDyne.com>.
I just occurred to me that I don't think anyone's asked if these are net-mounted file systems. I've seen this timestamp-shifting before, but only on net-mounted filesystems. Usually the source and local systems are set to different timezones (or DST settings).
> -----Original Message-----
> From: Christopher Schultz [mailto:chris@christopherschultz.net]
> Sent: Tuesday, October 12, 2010 1:47 PM
> To: Tomcat Users List
> Subject: Re: Error 503 ocurring when server under load
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Rob,
>
> On 10/11/2010 4:40 PM, Rob G wrote:
> > So if I'm reading your email and the docs correctly. I should just
> > comment out the cachesize=10 from the workers.properties.
>
> I would.
>
> > And since for connection_pool_size (that replaced it) JK will
> > discover this number for the Apache web server automatically and set
> > the pool size to this value, I don't need to add anything to the
> > workers.properties file?
>
> I believe that is true.
>
> On the other hand, there is another case where you might have problems.
> If you have, say, 512 worker threads in Apache httpd but you only have,
> say, 200 request processor threads configured in Tomcat, then you will
> get mod_jk connection failures on the httpd side.
>
> I would recommend that you have enough request processors configured in
> Tomcat to handle the expected load.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAky0rToACgkQ9CaO5/Lv0PCvjACgwttZ9YfINxpWP+DI1+VlKfvI
> OTAAoK+2RQRibL56GdYWlaWxx6obZVln
> =hY3s
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
__________________________________________________________________________
Confidentiality Notice: This Transmission (including any attachments) may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited.
If you have received this transmission in error, please immediately reply to the sender or telephone (512) 343-9100 and delete this transmission from your system.
Re: Error 503 ocurring when server under load
Posted by Rob G <ro...@gmail.com>.
On 12 October 2010 19:47, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> I would.
>
> I believe that is true.
>
> On the other hand, there is another case where you might have problems.
> If you have, say, 512 worker threads in Apache httpd but you only have,
> say, 200 request processor threads configured in Tomcat, then you will
> get mod_jk connection failures on the httpd side.
>
> I would recommend that you have enough request processors configured in
> Tomcat to handle the expected load.
>
> - -chris
Thanks Chris (and everyone else for their comments).
As an update: I updated Tomcat from 6.0.24 to 6.0.29 and commented out
the cachesize directive from workers.properties. The server ran for a
full day yesterday and not a single 503 error. :) So with a bit of
luck it will continue to work as expected. Of course I'll monitor for
any other issues for the next few days.
Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Error 503 ocurring when server under load
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Rob,
On 10/11/2010 4:40 PM, Rob G wrote:
> So if I'm reading your email and the docs correctly. I should just
> comment out the cachesize=10 from the workers.properties.
I would.
> And since for connection_pool_size (that replaced it) JK will
> discover this number for the Apache web server automatically and set
> the pool size to this value, I don't need to add anything to the
> workers.properties file?
I believe that is true.
On the other hand, there is another case where you might have problems.
If you have, say, 512 worker threads in Apache httpd but you only have,
say, 200 request processor threads configured in Tomcat, then you will
get mod_jk connection failures on the httpd side.
I would recommend that you have enough request processors configured in
Tomcat to handle the expected load.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAky0rToACgkQ9CaO5/Lv0PCvjACgwttZ9YfINxpWP+DI1+VlKfvI
OTAAoK+2RQRibL56GdYWlaWxx6obZVln
=hY3s
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Error 503 ocurring when server under load
Posted by Rob G <ro...@gmail.com>.
Christopher Schultz wrote> This directive has been deprecated since 1.2.16.
> Cachesize defines the number of connections made to the AJP backend that
> are maintained as a connection pool. It will limit the number of those
> connection that each web server child process can make.
>
> Cachesize property is used only for multi threaded web servers such as
> Apache 2.0 (worker), IIS and Netscape. The cachesize property should
> reflect the number of threads per child process. JK will discover the
> number of threads per child process on Apache 2 web server with
> worker-mpm and set its default value to match the ThreadsPerChild
> Apache directive. [...]
> "
>
> Since mpm_winnt (like mpm_worker) uses ThreadsPerChild to set the number
> of worker threads (and therefore simultaneous requests that can be
> accepted), you'll have to know what ThreadsPerChild is set to. That's
> easy: just search for that directive in httpd.conf. If it's not there,
> or commented-out, the default is 64 on mpm_winnt
> (http://httpd.apache.org/docs/2.0/mod/mpm_common.html#threadsperchild).
>
> If you've left it as the default (64) and you've set worker.cachesize
> (really worker.connection_pool_size) to 10, then you can accept 64
> connections from clients but only call-out to Tomcat over 10 of those
> connections: the others will have to wait. I'm not sure what the
> scheduling algorithm is for choosing which thread gets the next
> available connection from the pool (FIFO?), but it's very possible to
> starve your clients even after 10 simultaneous connections.
>
> - -chris
So if I'm reading your email and the docs correctly. I should just
comment out the cachesize=10 from the workers.properties. And since
for connection_pool_size (that replaced it) JK will discover this
number for the Apache web server automatically and set the pool size
to this value, I don't need to add anything to the workers.properties
file?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Error 503 ocurring when server under load
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Rob,
On 10/7/2010 7:07 PM, Rob G wrote:
> Christopher Schultz wrote
>> I think you might want to compare the request processor counts you
>> have for Tomcat versus mod_jk: the default maxThreads for a Tomcat
>> <Connector> is 200, while your MaxClients settings might be
>> different on your Apache httpd configuration.
>
> Excuse my ignorance, but can you clarify where I check these settings?
Obviously, the worker settings are in your Workers.properties file that
you've already posted. The settings for Apache httpd are usually in a
file called conf/httpd.conf in the Apache httpd installation directory.
On win32, that might be something like C:\Program Files\Apache
Foundation\Apache Web Server\conf\httpd.conf
Tomcat's server settings are in conf/server.xml in the Apache Tomcat
installation directory. On win32, that might be something like
C:\Program Files\Apache Foundation\Apache Tomcat\conf\server.xml
Since you have two copies of Tomcat working, you should check both of
them. In fact, if you could post the server.xml files from both
instances and tell us how you have them configured to run side-by-side,
that would be helpful.
Pid pointed out that you might not have both instances running due to a
shutdownPort conflict. That might also be the case if you have other
ports that conflict.
>> I believe that only the Worker MPM is available on Microsoft Windows. Have you explicitly configured an MPM?
>
> Since I don't know what an MPM is I'm going to say no.
:) That stands for "Multi-Processing Module" and it's how the Apache Web
Server serves more than one request at a time. On *NIX, you can choose
the "prefork" module which gives you one child process per simultaneous
request, or the "worker" module which gets you a single (child?) process
with many threads running within it. I just checked the documentation
and it looks like on windows, you get mpm_winnt, which is different from
even those, though it quite resembles the mpm_worker.
>> Since you're using Apache 2.x, mod_jk should be able to
>> auto-configure this value for you.
>
> Do I need to set anything to allow this auto config to happen?
No. If you don't set any value, mod_jk will compute a reasonable (and
usually correct) default. For a prefork MPM, having more than a single
connection in the connection pool is a huge waste, since a single httpd
request processor can't make more than one connection to Tomcat on the
backend. With the worker MPM, the connection pool should probably be the
same size as the number of worker threads you have configured.
Otherwise, you'll have a deficit that may cause the errors you are
encountering.
>> From my reading, you may be limiting yourself to 10 simultaneous
>> connections from Apache httpd to Tomcat, and others will get a
>> 503.
>
> What lead you to that conclusion?
Here's the text from the docs:
"
This directive has been deprecated since 1.2.16.
Cachesize defines the number of connections made to the AJP backend that
are maintained as a connection pool. It will limit the number of those
connection that each web server child process can make.
Cachesize property is used only for multi threaded web servers such as
Apache 2.0 (worker), IIS and Netscape. The cachesize property should
reflect the number of threads per child process. JK will discover the
number of threads per child process on Apache 2 web server with
worker-mpm and set its default value to match the ThreadsPerChild
Apache directive. [...]
"
Since mpm_winnt (like mpm_worker) uses ThreadsPerChild to set the number
of worker threads (and therefore simultaneous requests that can be
accepted), you'll have to know what ThreadsPerChild is set to. That's
easy: just search for that directive in httpd.conf. If it's not there,
or commented-out, the default is 64 on mpm_winnt
(http://httpd.apache.org/docs/2.0/mod/mpm_common.html#threadsperchild).
If you've left it as the default (64) and you've set worker.cachesize
(really worker.connection_pool_size) to 10, then you can accept 64
connections from clients but only call-out to Tomcat over 10 of those
connections: the others will have to wait. I'm not sure what the
scheduling algorithm is for choosing which thread gets the next
available connection from the pool (FIFO?), but it's very possible to
starve your clients even after 10 simultaneous connections.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkyvKLwACgkQ9CaO5/Lv0PBAqgCgh3/TNdasW5b6cnHXOP99BxrL
jCIAmgMt/kFdRWIut1m9QF8TmzbZ7zW3
=LXSf
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Error 503 ocurring when server under load
Posted by Pid <pi...@pidster.com>.
On 08/10/2010 00:07, Rob G wrote:
> Thanks to all for their replies to date. Much appreciated, I'll
> respond to some of the points raised. Apologies if I'm asking basic
> stuff but still getting my ahead around Tomcat and Apache integration.
>
> On 7 October 2010 18:31, André Warnier wrote:
>
>> Your configuration looks very clean to me (no unnecessary settings etc.),
>> which in this case is a plus (a good base to start tuning).
> Thank you
>
>> You may want to upgrade Tomcat to the latest version (6.0.29).
> Noted
>
>> For example, at the moment these errors happen, what are these Tomcats
>> really doing ?
>> Are they really busy each processing 200 requests, with 200 threads running
>> and actually doing something ? (200 is the default for the "maxThreads"
>> attribute of the AJP Connector).
> Unfortunately I'm not in a position to conduct realtime monitoring of
> the server and Tomcat. Is there a way to log this thread data?
How do you get log data?
>> If yes, then you may just need a leaner application, or a bigger system
>> (more RAM, faster CPU), or more systems. What does the Task Manager tell
>> you about the total system load ?
> Again, unfortunately not in a position to monitor in real time.
You can enable JMX and connect remotely on a specified port to get a
look inside the JVM>
>> If not, and many of these threads are waiting, then you may have an issue
>> with a keepAlive that is too long.
>> See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the
>> connectionTimeout and keepAliveTimeout attributes.
> Will do
>
>> Whatever you do, first get an idea of the starting situation. Then modify
>> one setting at a time, and observe (and note) the effects.
> Sensible advice
>
> Pid wrote:
>> OK. Is there anything else different apart from the Servlet container?
> I'm not sure what you mean?
Did you change DB too, or is was it a simple migration?
>> Why do you have two Tomcat instances? (It's not a trick question, I'm interested in your reasoning.)
> Load balancing. I thought/guessed that two instances would be
> sufficent. Perhaps this was incorrect and I need more?
Maybe, maybe not.
I was trying to understand why you had two instances, rather than one
with, say, more threads configured.
Load balancing two Tomcat instances which are both using the same
underlying computing resource will not offer a tangible benefit in terms
of throughput / response time compared to a suitably configured single
instance.
If you're on a 32bit system and are hitting process memory limits, you
might need two instances to cope with the load.
>> How many threads & server instances did you have before?
>> What were your Java -Xmx etc settings before, and now?
>> Are you using an Executor?
> I'll have to check, but I think it's safe to say that they are set to
> the default values as I don't recall changing anything relating to
> them.
>
>> Thread dumps from a maxed out Tomcat will tell you what each Thread is waiting for. Collect a series of these during high load periods to find
> out what's happening.
> I'll have to figure out how to do this, but sounds a good idea.
Do you have remote access to the machine?
There are command line tools with JDK 6 (6.0.21 is the latest) which are
useful. E.g. jstack, jmap.
If you have remote access to the GUI, JDK 6 also has JConsole and
VisualVM which provide metric tonnes of useful information.
Otherwise you can connect those tools, for slightly less output,
remotely via JMX. Be sure to use :
org.apache.catalina.mbeans.JmxRemoteLifecycleListener
http://tomcat.apache.org/tomcat-6.0-doc/config/listeners.html
Please also check the port numbers you have in use for each Tomcat
instance. Are the Tomcats installed and run as a service?
p
> Christopher Schultz wrote
>> I think you might want to compare the request processor counts you have for Tomcat versus mod_jk: the default maxThreads for a Tomcat <Connector> is 200, while your MaxClients settings might be different on your Apache httpd configuration.
> Excuse my ignorance, but can you clarify where I check these settings?
>
>> I believe that only the Worker MPM is available on Microsoft Windows. Have you explicitly configured an MPM?
> Since I don't know what an MPM is I'm going to say no.
>
>> Since you're using Apache 2.x, mod_jk should be able to auto-configure this value for you.
> Do I need to set anything to allow this auto config to happen?
>
>> From my reading, you may be limiting yourself to 10 simultaneous connections from Apache httpd to Tomcat, and others will get a 503.
> What lead you to that conclusion?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Re: Error 503 ocurring when server under load
Posted by Rob G <ro...@gmail.com>.
Thanks to all for their replies to date. Much appreciated, I'll
respond to some of the points raised. Apologies if I'm asking basic
stuff but still getting my ahead around Tomcat and Apache integration.
On 7 October 2010 18:31, André Warnier wrote:
> Your configuration looks very clean to me (no unnecessary settings etc.),
> which in this case is a plus (a good base to start tuning).
Thank you
> You may want to upgrade Tomcat to the latest version (6.0.29).
Noted
> For example, at the moment these errors happen, what are these Tomcats
> really doing ?
> Are they really busy each processing 200 requests, with 200 threads running
> and actually doing something ? (200 is the default for the "maxThreads"
> attribute of the AJP Connector).
Unfortunately I'm not in a position to conduct realtime monitoring of
the server and Tomcat. Is there a way to log this thread data?
> If yes, then you may just need a leaner application, or a bigger system
> (more RAM, faster CPU), or more systems. What does the Task Manager tell
> you about the total system load ?
Again, unfortunately not in a position to monitor in real time.
> If not, and many of these threads are waiting, then you may have an issue
> with a keepAlive that is too long.
> See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the
> connectionTimeout and keepAliveTimeout attributes.
Will do
> Whatever you do, first get an idea of the starting situation. Then modify
> one setting at a time, and observe (and note) the effects.
Sensible advice
Pid wrote:
> OK. Is there anything else different apart from the Servlet container?
I'm not sure what you mean?
> Why do you have two Tomcat instances? (It's not a trick question, I'm interested in your reasoning.)
Load balancing. I thought/guessed that two instances would be
sufficent. Perhaps this was incorrect and I need more?
> Was there an attachment I didn't see?
No, all details for .conf files and server.xml was posted inline.
> How many threads & server instances did you have before?
>What were your Java -Xmx etc settings before, and now?
> Are you using an Executor?
I'll have to check, but I think it's safe to say that they are set to
the default values as I don't recall changing anything relating to
them.
> Thread dumps from a maxed out Tomcat will tell you what each Thread is waiting for. Collect a series of these during high load periods to find
out what's happening.
I'll have to figure out how to do this, but sounds a good idea.
Christopher Schultz wrote
> I think you might want to compare the request processor counts you have for Tomcat versus mod_jk: the default maxThreads for a Tomcat <Connector> is 200, while your MaxClients settings might be different on your Apache httpd configuration.
Excuse my ignorance, but can you clarify where I check these settings?
> I believe that only the Worker MPM is available on Microsoft Windows. Have you explicitly configured an MPM?
Since I don't know what an MPM is I'm going to say no.
> Since you're using Apache 2.x, mod_jk should be able to auto-configure this value for you.
Do I need to set anything to allow this auto config to happen?
> From my reading, you may be limiting yourself to 10 simultaneous connections from Apache httpd to Tomcat, and others will get a 503.
What lead you to that conclusion?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Error 503 ocurring when server under load
Posted by André Warnier <aw...@ice-sa.com>.
Rob G wrote:
> Hey all,
>
> Recently migrated a production site (mixture of Servlets and JSPs)
> from Oracle Application Server to Apache/Tomcat. Since then we have
> seen numerous HTTP Error 503 - Service unavailable errors at peak
> times when site is under load. mod_jk.log has the following error
> message(s):
>
> [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances
> failed, no more workers left
>
> I'm looking for help in trying to tweak settings to prevent this, or
> confirmation that I've configured the setup correctly.
>
> Many thanks
> Rob
>
> Platform:
> Windows Server 2003 SP2
>
> Setup:
> Two tomcat instances with a single Apache front end, all on the same server
>
> Versions
> Tomcat: 6.0.24
> Apache: 2.2.16
> mod SSL: 2.2.16
> Open SSL: 0.9.8
> mod_JK:1.2.30
>
Your configuration looks very clean to me (no unnecessary settings etc.), which in this
case is a plus (a good base to start tuning).
You may want to upgrade Tomcat to the latest version (6.0.29).
But before you start tuning, you should get some idea of what is actually going on.
For example, at the moment these errors happen, what are these Tomcats really doing ?
Are they really busy each processing 200 requests, with 200 threads running and actually
doing something ? (200 is the default for the "maxThreads" attribute of the AJP Connector).
If yes, then you may just need a leaner application, or a bigger system (more RAM, faster
CPU), or more systems. What does the Task Manager tell you about the total system load ?
If not, and many of these threads are waiting, then you may have an issue with a keepAlive
that is too long.
See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the connectionTimeout and
keepAliveTimeout attributes.
Whatever you do, first get an idea of the starting situation. Then modify one setting at
a time, and observe (and note) the effects.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org