You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Ge...@gta-travel.com on 2007/08/29 12:47:13 UTC
apache getting in "sending reply" state when connecting to tomcat
Hi
I'm kind of between a rock and a hard place.
We have a problem in our production system that occurs quite regularly.
Apache's connections all get into a Sending Reply ( W ) state and which
makes the application unresponsive.
We have an apache 2.0.52 fronting 12 tomcat 5.5 all on CentOS 4.5 using
mod_jk with, dare is say it, default settings. :-(
We have a stripped down apache installed on each Tomcat blade and what
is interesting is that when we reach this stage of all connections in
"W" state we can access the application using the local apache on the
tomcat blade using port 8080 but not access it on port 8009 using the
local apache on the tomcat blade. This to me points to a connector
problem.
I believe that the problem is related to our mod_jk settings or lack
there off and also the version we are using.
What makes matters a bit difficult for me is that we are unable to
recreate the problems we are seeing in production on our test systems
which makes it very difficult to push out changes to production.
Management is quite strict in allowing production changes, which is
understandable because downtime is expensive.
We are using
mod_jk-1.2.22-2.0.52-linux-x86_64.so
httpd-2.0.52-28.ent prefork
Tomcat 5.5
Questions:
~~~~~~~~~~
* Do you agree that it is mod_jk settings?
* What more information do I need or should look at to determine
problems.( The developers regularly scrutinize thread dumps we make)
* mod_jk docs says: mod_jk-1.2.25-httpd-2.0.59.so is for Apache 2.0.x
and works with Apache 2.0.59 and later,
will using httpd-2.0.52-28.ent be a problem?
Settings
~~~~~~
httpd.conf
~~~~~~~~~~
<IfModule prefork.c>
StartServers 8
MinSpareServers 8
MaxSpareServers 300
ServerLimit 575
MaxClients 575
MaxRequestsPerChild 4000
workers.properties
~~~~~~~~~~~~~~~~~~
# Worker list
worker.list=xml-gta,jkstatus
# Worker definitions
worker.xml-gta.type=lb
worker.xml-gta.method=Busyness
worker.xml-gta.balanced_workers=
lonstct01agx,lonstct01bgx,lonstct01cgx,,lonstct01dgx,lonstct01egx,lonstc
t01fgx,lonstct01ggx,lonstct01hgx,lonstct01igx,lonstct01jgx,lonstct01kgx,
lonstct01lgx
worker.jkstatus.type=status
# Balance workers
worker.lonstct01agx.port=8009
worker.lonstct01agx.host=xx.xx.xx.xx
worker.lonstct01agx.type=ajp13
worker.lonstct01agx.lbfactor=1
...
worker.lonstct01lgx.port=8009
worker.lonstct01lgx.host=xx.xx.xx.xx
worker.lonstct01lgx.type=ajp13
worker.lonstct01lgx.lbfactor=1
server.xml
~~~~~~~~~~
<Connector port="8009"
enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />
Suggested Changes I want to make (but still need approval for)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Upgrade to mod_jk .25
Change workers.properties to:
workers.properties
~~~~~~~~~~~~~~~~~~
# Worker list
worker.list=xml-oct-gta,jkstatus
# Worker definitions
worker.xml-oct-gta.type=lb
worker.xml-oct-gta.method=Busyness
worker.xml-oct-gta.balance_workers=longtct02c,longtct02d
worker.xml-oct-gta.lock=Pessimistic
worker.xml-oct-gta.max_reply_timeouts=10
worker.jkstatus.type=status
# Worker Template
worker.reference.port=8009
worker.reference.type=ajp13
worker.reference.lbfactor=1
worker.reference.socket_timeout=60
worker.reference.socket_keepalive=true
worker.reference.connect_timeout=500
worker.reference.prepost_timeout=500
worker.reference.reply_timeout=32000
worker.reference.recovery_options=27
# 16 8 2 1
worker.reference.retries=12 #
# Balance workers
worker.longtct02c.reference=worker.reference
worker.longtct02c.host=xx.xx.xx.xx
... (there are 10 other servers not listed here for space saving
purposes)
worker.longtct02d.reference=worker.reference
worker.longtct02d.host=xx.xx.xx.xx
Regards
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache getting in "sending reply" state when connecting to tomcat
Posted by Rainer Jung <ra...@kippdata.de>.
Gerhardus.Geldenhuis@gta-travel.com wrote:
> Hi
> I'm going to be a real pain, but it make no sense now...
Let's see :)
> The email has been a team effort in our offices. We have included some
> diagrams to help illustrate our understanding or lack off.
>
> Using a simple example:
>
> 1/ Assume I have one httpd server (prefork) that can spawn a maximum of
> 200 children (through httpd Maxclients directive).
>
> 2/ Assume I have 1 tomcat servers that can handle 200 threads each.
>
> If I connect the apache to tomcat with mod_jk (lb) I can, in theory
> handle 200 concurrent connections.
>
> Now, if I change the figures
>
> 1/ Assume I have one httpd server (prefork) that can spawn a maximum of
> 200 children (through httpd Maxclients directive).
>
> 2/ Assume I have 4 tomcat servers that can handle 200 threads each.
>
> In this case each apache child opens a connection to each tomcat server
> so I have reached the maximum amount of connections each tomcat can
> handle. What I cannot understand is that by increasing the tomcats to 4
> I now have 800 possible connections but with the above config I can only
> access 200 of them. If I set apache to 800 (through httpd Maxclients
> directive) I will open more connection to each tomcat than they can
> handle.
>
> Is the above senario correct? and if it is then we are not getting more
> throughput by adding more tomcats and it would be better to access the
> tomcats directly.
Your considerations are correct. Since you can't influence, which apache
httpd process handles requests for which Tomcat instance, and since they
don't share the connections, this design doesn't scale to a huge farm
with a simple 1:N (1 httpd, N>>= Tomcats) setup.
What can you do:
0) For a relatively small farm the problem is usually not very big,
because if there is high load, the costly ressource is the CPU power,
and not the memory and switching overhead for having to many threads.
1) You can use the APR connector for Tomcat. This will decouple the
thread from the connection, as long as there's no request on it. That
way you'll only need threads for the real request paralellism concerning
each backend Tomcat. the number of connections will stay high though, so
you can't scale to a hundred of Tomcats with a thousand connections each..
2) You can use the worker MPM, because there a configurable number of
threads share the same connection pool. For N Tomcats, on average only
Threads_per_Process/N requests will need a connection for one Tomcat
instance. Of course in reality the number will be higher, but for bigger
N and enough thread per process you should notice a relevant decrease in
connections. Maybe not by 1/N but something like 2/N, depending on how
much session affinity breaks ideal balancing. If you get close to some
factor C/N for a not to large constant C, you are back in the scaling
busyness.
3) For huge designs you'll need to partition it into M:N (M apache
httpd, N Tomcat), where the quotient N/M doesn't get to big.
4) If your balancing breaks - or much more likely - if something in your
system gets slow then your expectation concerning the parallelism get
wrong. You can't fix that without fixing the slowness reason. What is
important though, is to configure the idleness timeouts for the Tomcat
thread pool and the jk connection pool, such that when the original
reason of slowness is gone, the connections and threads get down below
the critical level.
The situation you experienced was most likely coming from a slowness in
the backend applications (remember the need for a Java thread dump?).
Then any throughput system will soon get filled up from the back to the
front. The best you can do, is to answer the overload requests quickly
with an error, such that the backend systems have a chance to get stable
again. For this you need Timeouts and other load limitating configurations.
When doing sizing considerations, you always need to be clear for
yourself, if you are talking about the normal situation, or if you are
trying to find out, what will happen during times of overload.
> So using a ridiculous example, if you have 100 tomcat boxes connecting
> to one httpd server. The the limit for amount of spawned children would
> still only by 200. Even though you should be able to handle 100x200
> concurrent connections. Even if you take into account that for each
> request per second received the request will take 4 seconds to process
> it still does not seem effective use of the tomcat resources.
> A few other resulting questions:
> If child1, child2, child3 etc each have a connection to each tomcat,
> does each child also do its own load balancing or do all the children
> share information to do loadbalancing?
Fortunately they share the information state about the balancing. This
was introduced about 10 JK releases ago by means of a shared memory segment.
You could ask, why the processes don't share the connection pool. They
might do this some time in the future. Historically the pool came before
the shared memory for balancing. We prefer to stabilize JK 1.2.x now and
start working on a major next release. Switching to a shared pool will
likely lead to a couple of releases with a couple of bugs, so I don't
expect that to happen in 1.2.x.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache getting in "sending reply" state when connecting to tomcat
Posted by Ge...@gta-travel.com.
Hi
I'm going to be a real pain, but it make no sense now...
The email has been a team effort in our offices. We have included some
diagrams to help illustrate our understanding or lack off.
Using a simple example:
1/ Assume I have one httpd server (prefork) that can spawn a maximum of
200 children (through httpd Maxclients directive).
2/ Assume I have 1 tomcat servers that can handle 200 threads each.
If I connect the apache to tomcat with mod_jk (lb) I can, in theory
handle 200 concurrent connections.
Now, if I change the figures
1/ Assume I have one httpd server (prefork) that can spawn a maximum of
200 children (through httpd Maxclients directive).
2/ Assume I have 4 tomcat servers that can handle 200 threads each.
In this case each apache child opens a connection to each tomcat server
so I have reached the maximum amount of connections each tomcat can
handle. What I cannot understand is that by increasing the tomcats to 4
I now have 800 possible connections but with the above config I can only
access 200 of them. If I set apache to 800 (through httpd Maxclients
directive) I will open more connection to each tomcat than they can
handle.
Is the above senario correct? and if it is then we are not getting more
throughput by adding more tomcats and it would be better to access the
tomcats directly.
So using a ridiculous example, if you have 100 tomcat boxes connecting
to one httpd server. The the limit for amount of spawned children would
still only by 200. Even though you should be able to handle 100x200
concurrent connections. Even if you take into account that for each
request per second received the request will take 4 seconds to process
it still does not seem effective use of the tomcat resources.
A few other resulting questions:
If child1, child2, child3 etc each have a connection to each tomcat,
does each child also do its own load balancing or do all the children
share information to do loadbalancing?
Regards
>
> I don't know exactly, if that parameter changed is some
> versions, I simply checked one version of Tomcat. I would
> give it a 85% chance, that 200 is the default also for your version.
Sorry I should have bee more specific I did explicitely check this limit
so in our case it is definitely 200.
>
> You can: learn about the jmxproxy functionality inside the
> Tomcat manager webapp to look up the actual values used during runtime
>
> http://tomcat.apache.org/tomcat-5.5-doc/manager-howto.html
>
> use "qry=*:*" to get a dump of all MBeans, and then search
> for 8009 and you will find the right value.
>
> You can also use netstat -an in a more fine grained way, by
> not dropping the last column (connection state) to find out,
> if all connections are actually established, or maybe in SYN_SENT etc.
>
> > I am in the process of upgrading our production enviroment
> to mod_jk
> > .25 and setting relevant time out values which will
> hopefully improve
> > things or at least rule out one less potential problem.
> >
> > Regards
>
> Yes timeouts and increased thread numbers inside Tomcat will
> be good. To find out, why there are so many requests in
> progress you (resp. your webapp developers) really need to
> take a look at some Java thread dumps of your Tomcat processes.
>
> Regards,
>
> Rainer
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
Re: apache getting in "sending reply" state when connecting to tomcat
Posted by Rainer Jung <ra...@kippdata.de>.
Gerhardus.Geldenhuis@gta-travel.com wrote:
> If no value is specified then the maximum connectors is default 200.
>
> I assumed that Tomcat would not allow more connections than 200 to be
> made to port 8009. Why then do we see more than 200 connections on port
> 8009 on the httpd and tomcat side. Is this additional connections in
> some kind of "waiting pool" that gets establised by the OS but not
> honoured by Tomcat?
I don't know exactly, if that parameter changed is some versions, I
simply checked one version of Tomcat. I would give it a 85% chance, that
200 is the default also for your version.
You can: learn about the jmxproxy functionality inside the Tomcat
manager webapp to look up the actual values used during runtime
http://tomcat.apache.org/tomcat-5.5-doc/manager-howto.html
use "qry=*:*" to get a dump of all MBeans, and then search for 8009 and
you will find the right value.
You can also use netstat -an in a more fine grained way, by not dropping
the last column (connection state) to find out, if all connections are
actually established, or maybe in SYN_SENT etc.
> I am in the process of upgrading our production enviroment to mod_jk .25
> and setting relevant time out values which will hopefully improve things
> or at least rule out one less potential problem.
>
> Regards
Yes timeouts and increased thread numbers inside Tomcat will be good. To
find out, why there are so many requests in progress you (resp. your
webapp developers) really need to take a look at some Java thread dumps
of your Tomcat processes.
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache getting in "sending reply" state when connecting to tomcat
Posted by Ge...@gta-travel.com.
Thanks for your patience, things are starting to make more sense now.
>
> Tomcat associates one thread with each incoming connection
> (at least the default connector) independant of it's
> idleness, i.e. even if there is no request coming in. The
> connectionTimeout parameter in the connector tells tomcat,
> howe long it should wait for a request, before shutting down
> the connection and putting back the thread into the pool of
> idle threads.
>
> That's the reason, why you need to adjust the parameters
> between httpd and Tomcat.
>
> The reason, why you have so many connections will be in some
> part of the webapp being to slow. This you will be able to
> analyze by using a Java thread dump.
>
If no value is specified then the maximum connectors is default 200.
I assumed that Tomcat would not allow more connections than 200 to be
made to port 8009. Why then do we see more than 200 connections on port
8009 on the httpd and tomcat side. Is this additional connections in
some kind of "waiting pool" that gets establised by the OS but not
honoured by Tomcat?
I am in the process of upgrading our production enviroment to mod_jk .25
and setting relevant time out values which will hopefully improve things
or at least rule out one less potential problem.
Regards
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache getting in "sending reply" state when connecting to tomcat
Posted by Rainer Jung <ra...@kippdata.de>.
Gerhardus.Geldenhuis@gta-travel.com wrote:
> Hi
> This part is still sligtly confusing to me:
>
> We have 575 potential connections that we can except on the httpd
> server, according to my understanding mod_jk will load balance these
> connections to the tomcat servers. Thus typically 48 connections per
> tomcat. This does seem obviously wrong ...
>
> We are seeing avarage amount of connections of about 245 per tomcat
> server on the apache box when we reach a state were all servers are in
> "W" state. That is about 3000 connections in total over the 12 tomcats.
> This does not tally up with the default maximum of 200. Any ideas where
> this additional connections comes from??
Connection != Request
mod_jk uses persistant connections between Apache httpd and Tomcat.
Because it's server to server communication one usually prefers not to
establish a new connection for each new request.
By default mod_jk will open up at most as many connections from any
httpd process to any backend, as there are threads in the httpd process.
In your case you use the prefork MPM, which has only one thread per
httpd process, so mod_jk will create at most one connection from each
httpd process to each Tomcat backend. Connections are always httpd
process private.
Connections are closed after some time of idleness, depending on
configuration. By default they stay open forever. Check the
connection_cache_* parameters for details.
Tomcat associates one thread with each incoming connection (at least the
default connector) independant of it's idleness, i.e. even if there is
no request coming in. The connectionTimeout parameter in the connector
tells tomcat, howe long it should wait for a request, before shutting
down the connection and putting back the thread into the pool of idle
threads.
That's the reason, why you need to adjust the parameters between httpd
and Tomcat.
The reason, why you have so many connections will be in some part of the
webapp being to slow. This you will be able to analyze by using a Java
thread dump.
> We feel confident that we don't have stale connections. Connections on
> the httpd side and connections on the tomcat side tally up. lsof on the
> tomcat box is also exactly the same as the netstat result.
> Commands used:
> [root@LONSTCT01AGX ~]# lsof|grep -i 8009 |grep -i established |grep -vi
> localhost -c
> 75
> [root@LONSTCT01AGX ~]# netstat -ant|grep 8009 |awk '{print $5}'|awk -F:
> '{print $4}'|sort|uniq -c
> 8
> 1 *
> 75 10.100.11.225
> 8 127.0.0.1
>
>
> Regards
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache getting in "sending reply" state when connecting to tomcat
Posted by Ge...@gta-travel.com.
Hi
This part is still sligtly confusing to me:
We have 575 potential connections that we can except on the httpd
server, according to my understanding mod_jk will load balance these
connections to the tomcat servers. Thus typically 48 connections per
tomcat. This does seem obviously wrong ...
We are seeing avarage amount of connections of about 245 per tomcat
server on the apache box when we reach a state were all servers are in
"W" state. That is about 3000 connections in total over the 12 tomcats.
This does not tally up with the default maximum of 200. Any ideas where
this additional connections comes from??
We feel confident that we don't have stale connections. Connections on
the httpd side and connections on the tomcat side tally up. lsof on the
tomcat box is also exactly the same as the netstat result.
Commands used:
[root@LONSTCT01AGX ~]# lsof|grep -i 8009 |grep -i established |grep -vi
localhost -c
75
[root@LONSTCT01AGX ~]# netstat -ant|grep 8009 |awk '{print $5}'|awk -F:
'{print $4}'|sort|uniq -c
8
1 *
75 10.100.11.225
8 127.0.0.1
Regards
> -----Original Message-----
> From: Rainer Jung [mailto:rainer.jung@kippdata.de]
> Sent: 29 August 2007 21:29
> To: Tomcat Users List
> Subject: Re: apache getting in "sending reply" state when
> connecting to tomcat
>
> Hi Gerhardus,
>
> you allow 575 parallel requests with Apache. If something
> gets slow and you get more and more "W" states, this
> parallelism will really get used.
> Each parallel requests needs a connection to your Tomcat
> (unless it gets served directly by Apache), and each
> connection to Tomcat needs a thread to handle the request.
>
> You didn't configure the number of threads for Tomcats AJP connector.
> The default is something like 200 threads. So you allow much
> more incoming parallelism on Apache, than you configured on
> you backend. Once you get above the 200 parallel requests,
> you should see messages like "all threads are busy" in your
> Tomcat log files.
>
> But: This doesn't really explain, why you get into a
> situation, where that many requests need to be run in
> parallel. If your requests start to queue up (more "W" than
> normally), you should do Java Thread Dumps for Tomcat
> (sending kill -QUIT) which will go to catalina.out. Those
> will tell you/us/your webapp developers, which parts of the
> code Tomcat or more likely your webapp is getting slow in.
>
> Java thread dumps only give a snapshot information, so it is
> good to do a couple (3-5) of them a few seconds apart from each other.
>
> Regards,
>
> Rainer
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: apache getting in "sending reply" state when connecting to tomcat
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Gerhardus,
you allow 575 parallel requests with Apache. If something gets slow and
you get more and more "W" states, this parallelism will really get used.
Each parallel requests needs a connection to your Tomcat (unless it gets
served directly by Apache), and each connection to Tomcat needs a thread
to handle the request.
You didn't configure the number of threads for Tomcats AJP connector.
The default is something like 200 threads. So you allow much more
incoming parallelism on Apache, than you configured on you backend. Once
you get above the 200 parallel requests, you should see messages like
"all threads are busy" in your Tomcat log files.
But: This doesn't really explain, why you get into a situation, where
that many requests need to be run in parallel. If your requests start to
queue up (more "W" than normally), you should do Java Thread Dumps for
Tomcat (sending kill -QUIT) which will go to catalina.out. Those will
tell you/us/your webapp developers, which parts of the code Tomcat or
more likely your webapp is getting slow in.
Java thread dumps only give a snapshot information, so it is good to do
a couple (3-5) of them a few seconds apart from each other.
Regards,
Rainer
Gerhardus.Geldenhuis@gta-travel.com wrote:
> Hi
> I'm kind of between a rock and a hard place.
>
> We have a problem in our production system that occurs quite regularly.
> Apache's connections all get into a Sending Reply ( W ) state and which
> makes the application unresponsive.
> We have an apache 2.0.52 fronting 12 tomcat 5.5 all on CentOS 4.5 using
> mod_jk with, dare is say it, default settings. :-(
> We have a stripped down apache installed on each Tomcat blade and what
> is interesting is that when we reach this stage of all connections in
> "W" state we can access the application using the local apache on the
> tomcat blade using port 8080 but not access it on port 8009 using the
> local apache on the tomcat blade. This to me points to a connector
> problem.
>
> I believe that the problem is related to our mod_jk settings or lack
> there off and also the version we are using.
>
> What makes matters a bit difficult for me is that we are unable to
> recreate the problems we are seeing in production on our test systems
> which makes it very difficult to push out changes to production.
> Management is quite strict in allowing production changes, which is
> understandable because downtime is expensive.
>
> We are using
> mod_jk-1.2.22-2.0.52-linux-x86_64.so
> httpd-2.0.52-28.ent prefork
> Tomcat 5.5
>
> Questions:
> ~~~~~~~~~~
> * Do you agree that it is mod_jk settings?
> * What more information do I need or should look at to determine
> problems.( The developers regularly scrutinize thread dumps we make)
> * mod_jk docs says: mod_jk-1.2.25-httpd-2.0.59.so is for Apache 2.0.x
> and works with Apache 2.0.59 and later,
> will using httpd-2.0.52-28.ent be a problem?
>
>
> Settings
> ~~~~~~
>
> httpd.conf
> ~~~~~~~~~~
> <IfModule prefork.c>
> StartServers 8
> MinSpareServers 8
> MaxSpareServers 300
> ServerLimit 575
> MaxClients 575
> MaxRequestsPerChild 4000
>
> workers.properties
> ~~~~~~~~~~~~~~~~~~
> # Worker list
> worker.list=xml-gta,jkstatus
>
> # Worker definitions
> worker.xml-gta.type=lb
> worker.xml-gta.method=Busyness
> worker.xml-gta.balanced_workers=
> lonstct01agx,lonstct01bgx,lonstct01cgx,,lonstct01dgx,lonstct01egx,lonstc
> t01fgx,lonstct01ggx,lonstct01hgx,lonstct01igx,lonstct01jgx,lonstct01kgx,
> lonstct01lgx
> worker.jkstatus.type=status
>
> # Balance workers
> worker.lonstct01agx.port=8009
> worker.lonstct01agx.host=xx.xx.xx.xx
> worker.lonstct01agx.type=ajp13
> worker.lonstct01agx.lbfactor=1
>
> ...
>
> worker.lonstct01lgx.port=8009
> worker.lonstct01lgx.host=xx.xx.xx.xx
> worker.lonstct01lgx.type=ajp13
> worker.lonstct01lgx.lbfactor=1
>
> server.xml
> ~~~~~~~~~~
> <Connector port="8009"
> enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />
>
>
> Suggested Changes I want to make (but still need approval for)
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Upgrade to mod_jk .25
>
> Change workers.properties to:
>
> workers.properties
> ~~~~~~~~~~~~~~~~~~
> # Worker list
> worker.list=xml-oct-gta,jkstatus
>
> # Worker definitions
> worker.xml-oct-gta.type=lb
> worker.xml-oct-gta.method=Busyness
> worker.xml-oct-gta.balance_workers=longtct02c,longtct02d
> worker.xml-oct-gta.lock=Pessimistic
> worker.xml-oct-gta.max_reply_timeouts=10
>
> worker.jkstatus.type=status
>
>
> # Worker Template
> worker.reference.port=8009
> worker.reference.type=ajp13
> worker.reference.lbfactor=1
> worker.reference.socket_timeout=60
> worker.reference.socket_keepalive=true
> worker.reference.connect_timeout=500
> worker.reference.prepost_timeout=500
> worker.reference.reply_timeout=32000
> worker.reference.recovery_options=27
> # 16 8 2 1
> worker.reference.retries=12 #
>
> # Balance workers
> worker.longtct02c.reference=worker.reference
> worker.longtct02c.host=xx.xx.xx.xx
> ... (there are 10 other servers not listed here for space saving
> purposes)
> worker.longtct02d.reference=worker.reference
> worker.longtct02d.host=xx.xx.xx.xx
>
> Regards
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache getting in "sending reply" state when connecting to tomcat
Posted by Ge...@gta-travel.com.
Charles, when I say apache I mean httpd service.
There is 3 different servers mentioned.
* The httpd server that fronts the tomcat servers
* The tomcat servers that serves the application
* The httpd server, scaled down on the same physical machine as the
tomcat server which we use for basic monitoring and testing.
I hope that clears up any confusion.
Regards
> -----Original Message-----
> From: Caldarale, Charles R [mailto:Chuck.Caldarale@unisys.com]
> Sent: 29 August 2007 14:18
> To: Tomcat Users List
> Subject: RE: apache getting in "sending reply" state when
> connecting to tomcat
>
> > From: Gerhardus.Geldenhuis@gta-travel.com
> > [mailto:Gerhardus.Geldenhuis@gta-travel.com]
> > Subject: RE: apache getting in "sending reply" state when
> connecting
> > to tomcat
> >
> > I forgot to add, that our solution at the moment is to
> restart apache
> > which 9 out of 10 times solves the problems.
>
> Please clarify your use of the term "apache", which is a
> software organization with numerous products, among them
> httpd and Tomcat. In some instances, you seem to be
> referring to httpd, but in others you clearly mean Tomcat
> (e.g., "using the local apache on the tomcat blade using port 8080").
>
> When you say you "restart apache", which component are you
> really referring to?
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE
> PROPRIETARY MATERIAL and is thus for use only by the intended
> recipient. If you received this in error, please contact the
> sender and delete the e-mail and its attachments from all computers.
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org To
> unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache getting in "sending reply" state when connecting to tomcat
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Gerhardus.Geldenhuis@gta-travel.com
> [mailto:Gerhardus.Geldenhuis@gta-travel.com]
> Subject: RE: apache getting in "sending reply" state when
> connecting to tomcat
>
> I forgot to add, that our solution at the moment is to restart apache
> which 9 out of 10 times solves the problems.
Please clarify your use of the term "apache", which is a software
organization with numerous products, among them httpd and Tomcat. In
some instances, you seem to be referring to httpd, but in others you
clearly mean Tomcat (e.g., "using the local apache on the tomcat blade
using port 8080").
When you say you "restart apache", which component are you really
referring to?
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: apache getting in "sending reply" state when connecting to tomcat
Posted by Ge...@gta-travel.com.
Hi
I forgot to add, that our solution at the moment is to restart apache
which 9 out of 10 times solves the problems. We see connections reach
almost 250 (248,249) per tomcat server on the apache box. After restart
this drops down
We also only see the problem when we are experiencing heavy load and it
happens very quickly when it does.
Regards
> -----Original Message-----
> From: Gerhardus Geldenhuis (GTA-LON)
> Sent: 29 August 2007 11:47
> To: users@tomcat.apache.org
> Subject: apache getting in "sending reply" state when
> connecting to tomcat
>
> Hi
> I'm kind of between a rock and a hard place.
>
> We have a problem in our production system that occurs quite
> regularly.
> Apache's connections all get into a Sending Reply ( W ) state
> and which makes the application unresponsive.
> We have an apache 2.0.52 fronting 12 tomcat 5.5 all on CentOS
> 4.5 using mod_jk with, dare is say it, default settings. :-(
> We have a stripped down apache installed on each Tomcat blade
> and what is interesting is that when we reach this stage of
> all connections in "W" state we can access the application
> using the local apache on the tomcat blade using port 8080
> but not access it on port 8009 using the local apache on the
> tomcat blade. This to me points to a connector problem.
>
> I believe that the problem is related to our mod_jk settings
> or lack there off and also the version we are using.
>
> What makes matters a bit difficult for me is that we are
> unable to recreate the problems we are seeing in production
> on our test systems which makes it very difficult to push out
> changes to production.
> Management is quite strict in allowing production changes,
> which is understandable because downtime is expensive.
>
> We are using
> mod_jk-1.2.22-2.0.52-linux-x86_64.so
> httpd-2.0.52-28.ent prefork
> Tomcat 5.5
>
> Questions:
> ~~~~~~~~~~
> * Do you agree that it is mod_jk settings?
> * What more information do I need or should look at to
> determine problems.( The developers regularly scrutinize
> thread dumps we make)
> * mod_jk docs says: mod_jk-1.2.25-httpd-2.0.59.so is for
> Apache 2.0.x and works with Apache 2.0.59 and later, will
> using httpd-2.0.52-28.ent be a problem?
>
>
> Settings
> ~~~~~~
>
> httpd.conf
> ~~~~~~~~~~
> <IfModule prefork.c>
> StartServers 8
> MinSpareServers 8
> MaxSpareServers 300
> ServerLimit 575
> MaxClients 575
> MaxRequestsPerChild 4000
>
> workers.properties
> ~~~~~~~~~~~~~~~~~~
> # Worker list
> worker.list=xml-gta,jkstatus
>
> # Worker definitions
> worker.xml-gta.type=lb
> worker.xml-gta.method=Busyness
> worker.xml-gta.balanced_workers=
> lonstct01agx,lonstct01bgx,lonstct01cgx,,lonstct01dgx,lonstct01
> egx,lonstc
> t01fgx,lonstct01ggx,lonstct01hgx,lonstct01igx,lonstct01jgx,lon
> stct01kgx,
> lonstct01lgx
> worker.jkstatus.type=status
>
> # Balance workers
> worker.lonstct01agx.port=8009
> worker.lonstct01agx.host=xx.xx.xx.xx
> worker.lonstct01agx.type=ajp13
> worker.lonstct01agx.lbfactor=1
>
> ...
>
> worker.lonstct01lgx.port=8009
> worker.lonstct01lgx.host=xx.xx.xx.xx
> worker.lonstct01lgx.type=ajp13
> worker.lonstct01lgx.lbfactor=1
>
> server.xml
> ~~~~~~~~~~
> <Connector port="8009"
> enableLookups="false" redirectPort="8443" protocol="AJP/1.3" />
>
>
> Suggested Changes I want to make (but still need approval
> for) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> Upgrade to mod_jk .25
>
> Change workers.properties to:
>
> workers.properties
> ~~~~~~~~~~~~~~~~~~
> # Worker list
> worker.list=xml-oct-gta,jkstatus
>
> # Worker definitions
> worker.xml-oct-gta.type=lb
> worker.xml-oct-gta.method=Busyness
> worker.xml-oct-gta.balance_workers=longtct02c,longtct02d
> worker.xml-oct-gta.lock=Pessimistic
> worker.xml-oct-gta.max_reply_timeouts=10
>
> worker.jkstatus.type=status
>
>
> # Worker Template
> worker.reference.port=8009
> worker.reference.type=ajp13
> worker.reference.lbfactor=1
> worker.reference.socket_timeout=60
> worker.reference.socket_keepalive=true
> worker.reference.connect_timeout=500
> worker.reference.prepost_timeout=500
> worker.reference.reply_timeout=32000
> worker.reference.recovery_options=27
> # 16 8 2 1
> worker.reference.retries=12 #
>
> # Balance workers
> worker.longtct02c.reference=worker.reference
> worker.longtct02c.host=xx.xx.xx.xx
> ... (there are 10 other servers not listed here for space saving
> purposes)
> worker.longtct02d.reference=worker.reference
> worker.longtct02d.host=xx.xx.xx.xx
>
> Regards
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit
> http://www.messagelabs.com/email
> ______________________________________________________________________
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org To
> unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email
______________________________________________________________________
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org