You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Kurt Overberg <ku...@hotdogrecords.com> on 2006/04/05 02:29:27 UTC
Apache/1.3.33 - mod_jk/1.2.15 - tomcat 5.0.28 loses connection?
Gang,
I have a somewhat heavily loaded tomcat application that uses a
fairly standard Apache 1.3.33 / mod_jk 1.2.15 / tomcat 5.0.28 set up. I
am doing load balancing in the mod_jk, going to 2 RedHat 2.6.9 boxes
that run the tomcats. The load is split evenly across the two tomcat
boxes. The site processes anywhere between 400,000 and 700,000 JSP
pages per day. I've included the configuration files at the bottom of
this message.
Usually, the site runs great. No problems whatsoever (well, my code
occasionally crashes, but that's to be expected). But once in awhile,
my mod_jk seems to suddenly start to "lose" connections to my tomcats.
mod_jk.log sees this:
[Tue Apr 04 18:23:35 2006] [info] ajp_process_callback::jk_ajp_common.c
(1384): Connection aborted or network problems
[Tue Apr 04 18:23:35 2006] [info] ajp_service::jk_ajp_common.c (1731):
Receiving from tomcat failed, because of client error without recovery
in send loop 0
[Tue Apr 04 18:23:35 2006] [info] service::jk_lb_worker.c (711):
unrecoverable error 400, request failed. Client failed in the middle of
request, we can't recover to another instance.
[Tue Apr 04 18:23:35 2006] [info] jk_handler::mod_jk.c (1841): Aborting
connection for worker=loadbalancer
[Tue Apr 04 18:23:35 2006] [info] ajp_process_callback::jk_ajp_common.c
(1384): Connection aborted or network problems
[Tue Apr 04 18:23:35 2006] [info] ajp_service::jk_ajp_common.c (1731):
Receiving from tomcat failed, because of client error without recovery
in send loop 0
[Tue Apr 04 18:23:35 2006] [info] service::jk_lb_worker.c (711):
unrecoverable error 400, request failed. Client failed in the middle of
request, we can't recover to another instance.
[Tue Apr 04 18:23:35 2006] [info] jk_handler::mod_jk.c (1841): Aborting
connection for worker=loadbalancer
[Tue Apr 04 18:23:35 2006] [info] ajp_process_callback::jk_ajp_common.c
(1384): Connection aborted or network problems
[Tue Apr 04 18:23:35 2006] [info] ajp_service::jk_ajp_common.c (1731):
Receiving from tomcat failed, because of client error without recovery
in send loop 0
[Tue Apr 04 18:23:35 2006] [info] service::jk_lb_worker.c (711):
unrecoverable error 400, request failed. Client failed in the middle of
request, we can't recover to another instance.
[Tue Apr 04 18:23:35 2006] [info] jk_handler::mod_jk.c (1841): Aborting
connection for worker=loadbalancer
...when this happens, my Apache scoreboard fills up with threads in the
"W" state. All new requests seem to get stuck in this state. Quickly,
the apache fills up and new requests are locked out. If I look at my
tomcat logs, I see that processing has slowed and stopped. One
interesting thing is that if I kill and restart my apache, the tomcat
logs go crazy with output, as if killing the apache somehow "unsticks"
them.
Is something running out of sockets? If I had to guess, I'd say that it
feels like mod_jk can't receive the data back from the tomcats, but
thats just a hunch. I've seen the apache mod_jk config page which seems
to have a bunch of different parameters for dealing with "stuck"
tomcats, but I'm unsure of which one to use, because I don't really know
whats happening or what the problem is. I've found other posts on the
web that list similar problems, but haven't seen any "Oh this is the
problem and this is the simple solution". I haven't yet dug in with
netstat to see what's going on network-wise, I'm hoping there some
magical bullet configuration parameter that will help. C'moooonnnnn
magic bullet!!
Thanks for any insight you can provide!
/kurt
Appropriate configs:
My Apache is set to 1024 simultaneous connections and the JkMounts are
configured properly. I have my maximum number of file descriptors
(sockets) set to 65535 on all machines.
Tomcat AJP connector configs on machines tc1 and tc2
<Connector port="8089" protocol="AJP/1.3" maxThreads="400"
maxProcessors="0" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" redirectPort="8593"/>
workers.properties config
worker.list=tc1-w1,tc2-w1,status,loadbalancer
# tomcat 1 on host tc1
worker.tc1-w1.host=192.168.1.254
worker.tc1-w1.port=8089
worker.tc1-w1.type=ajp13
worker.tc1-w1.lbfactor=8
# tomcat 1 on host tc2
worker.tc2-w1.host=192.168.1.247
worker.tc2-w1.port=8089
worker.tc2-w1.type=ajp13
worker.tc2-w1.lbfactor=8
# status and loadbalancer workers
worker.status.type=status
worker.loadbalancer.type=lb
worker.loadbalancer.balanced_workers=tc1-w1,tc2-w1
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache/1.3.33 - mod_jk/1.2.15 - tomcat 5.0.28 loses connection?
Posted by Kurt Overberg <ku...@hotdogrecords.com>.
Mladen,
Thank you for this excellent information, it sounds like exactly what I
needed- some context for those parameters.
One last question with regards to setting connectionTimeout="xx" in my
server.xml. I'm using tomcat5.0.28- judging by the docs at
http://tomcat.apache.org/tomcat-5.0-doc/config/ajp.html
...that parameter doesn't exist (it does exist, though, in the 5.5
docs). Am I just SOL on that parameter because I'm using 5.0.28?
Many thanks again...
/kurt
Mladen Turk wrote:
> Kurt Overberg wrote:
>> It seems that the number of connections between the apache and the
>> tomcats just start escalating for some reason. mod_jk.log shows a ton
>> (>50) of these errors:
>>
>
> Hard to tell without httpd.conf and server.xml.
> Two things you should know:
> 1. MaxClients shuld be <= maxThreads
> 2. If you have MaxRequestsPerChild != 0 you must
> set connectionTimeout in server.xml
>
> If you have connectionTimeout in server.xml add connect_timeout=60000
> and prepost_timeout=60000 to each worker (except loadbalancer).
>
> Also if you have a firewall between Apache Httpd and Tomcat use
> connect_timeout and prepost_timeout because the firewall can
> cut the inactive connections. Also add connectionTimeout that
> will reflect the firewall cut time.
>
> Anyhow AJP13 protocol is persistent, so if any part can close
> the socket (simple restart) the other one will not know about
> that, the socket will be half-closed and the connection count
> will rise until one part breaks down (usually Tomcat).
> To deal with that issues add
> connectionTimeout="600000" (10 minutes) to server.xml
> and ...
> worker.xxx.connect_timeout=60000 (1 minute)
> worker.xxx.prepost_timeout=60000 (1 minute)
>
> The later will force so called PING/PONG packets to be
> send between web server and Tomcat. If the response packet
> is not received within the specified time the socket will
> be closed and the new one will be created.
>
> Regards,
> Mladen.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache/1.3.33 - mod_jk/1.2.15 - tomcat 5.0.28 loses connection?
Posted by Mladen Turk <ml...@jboss.com>.
Kurt Overberg wrote:
> It seems that the number of connections
> between the apache and the tomcats just start escalating for some
> reason. mod_jk.log shows a ton (>50) of these errors:
>
Hard to tell without httpd.conf and server.xml.
Two things you should know:
1. MaxClients shuld be <= maxThreads
2. If you have MaxRequestsPerChild != 0 you must
set connectionTimeout in server.xml
If you have connectionTimeout in server.xml add connect_timeout=60000
and prepost_timeout=60000 to each worker (except loadbalancer).
Also if you have a firewall between Apache Httpd and Tomcat use
connect_timeout and prepost_timeout because the firewall can
cut the inactive connections. Also add connectionTimeout that
will reflect the firewall cut time.
Anyhow AJP13 protocol is persistent, so if any part can close
the socket (simple restart) the other one will not know about
that, the socket will be half-closed and the connection count
will rise until one part breaks down (usually Tomcat).
To deal with that issues add
connectionTimeout="600000" (10 minutes) to server.xml
and ...
worker.xxx.connect_timeout=60000 (1 minute)
worker.xxx.prepost_timeout=60000 (1 minute)
The later will force so called PING/PONG packets to be
send between web server and Tomcat. If the response packet
is not received within the specified time the socket will
be closed and the new one will be created.
Regards,
Mladen.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache/1.3.33 - mod_jk/1.2.15 - tomcat 5.0.28 loses connection?
Posted by Kurt Overberg <ku...@hotdogrecords.com>.
Mladen, thanks for the response. I've been digging into the problem
with netstat a bit. Here is a log of the date, # of connections to
tomcat1, number of connections to tomcat2 (by netstat -a | grep
webserverIP | wc -l). At 10:58 I had to restart the apache because it
was in the "Stuck" state. It seems that the number of connections
between the apache and the tomcats just start escalating for some
reason. mod_jk.log shows a ton (>50) of these errors:
mod_jk.log
==========
[Wed Apr 05 10:57:35 2006] [error]
ajp_connection_tcp_get_message::jk_ajp_common.c (961): Can't receive the
response message from tomcat, network problems or tomcat is down
(192.168.1.254:8089), err=-104
Wed Apr 05 10:57:35 2006] [error] ajp_get_reply::jk_ajp_common.c (1503):
Tomcat is down or refused connection. No response has been sent to the
client (yet)
[Wed Apr 05 10:57:35 2006] [error] ajp_service::jk_ajp_common.c (1758):
Error connecting to tomcat. Tomcat is probably not started or is
listening on the wrong port. worker=tc1-w1 failed
[Wed Apr 05 10:57:35 2006] [info] service::jk_lb_worker.c (696):
service failed, worker tc1-w1 is in error state
It seems like some socket is not getting closed or something, and
resources are building up? BTW- jkstatus shows that my MaxBusy threads
as 10 and 8 respectively. Any ideas why it would be creating so many
sockets between apache and tomcat? Also- I was able to connect to
tomcat1 via its web port while things were wedged. Also- before a
previous crash/wedge, I noticed that the number of connections to tc1
was was dropping down to 10 or 12. Very confusing.
Thanks again for any insight...
/kurt
04/05/2006 10:52:35 AM 73 72
04/05/2006 10:52:45 AM 74 74
04/05/2006 10:52:56 AM 72 74
04/05/2006 10:53:06 AM 65 72
04/05/2006 10:53:17 AM 63 71
04/05/2006 10:53:27 AM 61 72
04/05/2006 10:53:37 AM 61 69
04/05/2006 10:53:48 AM 63 69
04/05/2006 10:53:58 AM 62 69
04/05/2006 10:54:09 AM 64 70
04/05/2006 10:54:19 AM 67 67
04/05/2006 10:54:30 AM 69 67
04/05/2006 10:54:41 AM 70 68
04/05/2006 10:54:52 AM 71 71
04/05/2006 10:55:02 AM 74 69
04/05/2006 10:55:12 AM 77 68
04/05/2006 10:55:23 AM 77 67
04/05/2006 10:55:33 AM 86 65
04/05/2006 10:55:44 AM 97 67
04/05/2006 10:55:54 AM 98 69
04/05/2006 10:56:04 AM 99 75
04/05/2006 10:56:15 AM 104 78
04/05/2006 10:56:25 AM 109 83
04/05/2006 10:56:36 AM 116 85
04/05/2006 10:56:46 AM 120 86
04/05/2006 10:56:56 AM 121 86
04/05/2006 10:57:07 AM 125 84
04/05/2006 10:57:17 AM 128 82
04/05/2006 10:57:28 AM 134 84
04/05/2006 10:57:39 AM 130 142
04/05/2006 10:57:49 AM 130 145
04/05/2006 10:58:00 AM 130 147
04/05/2006 10:58:11 AM 130 144
04/05/2006 10:58:21 AM 130 140
04/05/2006 10:58:32 AM 11 136
04/05/2006 10:58:43 AM 10 30
Mladen Turk wrote:
> Kurt Overberg wrote:
>>
>>
>> [Tue Apr 04 18:23:35 2006] [info]
>> ajp_process_callback::jk_ajp_common.c (1384): Connection aborted or
>> network problems
>> [Tue Apr 04 18:23:35 2006] [info] ajp_service::jk_ajp_common.c
>> (1731): Receiving from tomcat failed, because of client error without
>> recovery in send loop 0
>> [Tue Apr 04 18:23:35 2006] [info] service::jk_lb_worker.c (711):
>> unrecoverable error 400, request failed. Client failed in the middle
>> of request, we can't recover to another instance.
>> [Tue Apr 04 18:23:35 2006] [info] jk_handler::mod_jk.c (1841):
>> Aborting connection for worker=loadbalancer
>
> This means exactly what it says. The client has closed it's part
> of connection in the middle of the request, thus Tomcat is responding
> with 400 (Bad Request).
>
> If you wish to get rid of those messaged rise the JkLogLevel to error.
>
> Regards,
> Mladen.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Apache/1.3.33 - mod_jk/1.2.15 - tomcat 5.0.28 loses connection?
Posted by Mladen Turk <ml...@jboss.com>.
Kurt Overberg wrote:
>
>
> [Tue Apr 04 18:23:35 2006] [info] ajp_process_callback::jk_ajp_common.c
> (1384): Connection aborted or network problems
> [Tue Apr 04 18:23:35 2006] [info] ajp_service::jk_ajp_common.c (1731):
> Receiving from tomcat failed, because of client error without recovery
> in send loop 0
> [Tue Apr 04 18:23:35 2006] [info] service::jk_lb_worker.c (711):
> unrecoverable error 400, request failed. Client failed in the middle of
> request, we can't recover to another instance.
> [Tue Apr 04 18:23:35 2006] [info] jk_handler::mod_jk.c (1841): Aborting
> connection for worker=loadbalancer
This means exactly what it says. The client has closed it's part
of connection in the middle of the request, thus Tomcat is responding
with 400 (Bad Request).
If you wish to get rid of those messaged rise the JkLogLevel to error.
Regards,
Mladen.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org