You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Ian Long <ia...@opterus.com> on 2009/03/05 18:56:56 UTC
Tomcat and Apache with mod_jk
Hi,
I have dedicated RHEL5 boxes (dedicate quad core, 4GB RAM) setup with
both apache httpd and tomcat running on the same box, with a hardware
load balancer selecting the node a user is routed to. I have mod_jk
setup, but am seeing some behaviour I'm not sure is correct. This is
on httpd 2.2.3, tomcat 6.0.16, and mod_lk version of 1.2.26. The load
balancer pings the machines every 2 seconds as part of a health check.
Below are the relevant bits of my configuration:
mod_jk.conf
==========
LoadModule jk_module modules/mod_jk.so
JkLogFile /var/log/httpd/mod_jk.log
JkLogLevel info
JkWorkersFile /etc/httpd/conf.d/workers.properties
JkShmFile /var/log/httpd/jk.shm
JkShmSize 800
JkOptions +ForwardURICompatUnparsed
JkMount /* ajp13
JkUnMount /cacti/* ajp13
JkUnMount /manager/* ajp13
workers.properties
==============
worker.list=ajp13
worker.ajp13.port=8009
worker.ajp13.host=localhost
worker.ajp13.maintain=60
worker.ajp13.type=ajp13
# worker.ajp13.connection_pool_minsize=0
worker.ajp13.connection_pool_timeout=600
worker.ajp13.connect_timeout=10000
worker.ajp13.prepost_timeout=10000
worker.ajp13.max_packet_size=16384
httpd.conf
========
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 800
MaxClients 800
MaxRequestsPerChild 0
</IfModule>
server.xml
========
<Connector port="8009" connectionTimeout="600000"
minSpareThreads="5" packetSize="16384" address="127.0.0.1"
enableLookups="false" disableUploadTimeout="true"
maxSpareThreads="75" maxThreads="800" protocol="AJP/1.3" />
On one machine, after 5 or 6 hours, when I run netstat to look at
connections between http and tomcat, I have 250 or so:
tcp 0 0 ::ffff:127.0.0.1:8009 ::ffff:
127.0.0.1:60408 ESTABLISHED
whereas the other machine only has 40 or so, and they are equally load
balanced. I tried setting worker.ajp13.connection_pool_minsize=0 but
this has no effect on what I'm seeing.
Is this the behaviour I should expect? The load on the machine is not
that heavy, at the busiest period there would be 3 request per second
going to the box.
I wouldn't expect this number to keep growing, especially under light
load.
I'm also seeing a few errors like the following in mod_jk.log:
[Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 10:34:28.527 2009] [26222:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 10:34:28.527 2009] [26222:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 10:34:28.528 2009] [26222:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 10:36:01.071 2009] [25855:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 10:36:01.072 2009] [25855:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 10:36:01.072 2009] [25855:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 11:53:42.298 2009] [27523:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 11:53:42.298 2009] [27523:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 11:53:42.298 2009] [27523:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 12:08:07.551 2009] [28365:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 12:08:07.551 2009] [28365:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 12:08:07.551 2009] [28365:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 12:08:17.432 2009] [28428:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 12:08:17.432 2009] [28428:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 12:08:17.433 2009] [28428:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 12:12:19.656 2009] [28363:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 12:12:19.656 2009] [28363:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 12:12:19.657 2009] [28363:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
Can anyone see anything wrong with the configuration?
Cheers,
Ian Long
Chief Software Architect, Opterus Inc.
Email: ian.long@opterus.com
Phone: 416-840-8495 x666
Mobile: 416-817-9206
Web: http://www.opterus.com
Re: Tomcat and Apache with mod_jk
Posted by buzzterrier <te...@gmail.com>.
BTW, I finally got this working. The final key was using the latest version
of tcnative binary. Note to others, make sure your code sets the
cache-control header to "no-transform, max-age=0".
buzzterrier wrote:
>
> Actually this may not be just a case of the end user clicking away. There
> is an Internet Explorer issue with downloading files, where IE deletes the
> cached file before the user can load it. Your tomcat logs would probably
> show an error like: ClientAbortException: java.io.IOException. I am
> suffering with this at present, and the suggestion of setting the
> cache-control headers to max-age=0 does not work.
>
>
>
-----
Buzzterrier
http://buzzterrier.blogspot.com/ View my blog: Ordinary Average Developer...
--
View this message in context: http://www.nabble.com/Tomcat-and-Apache-with-mod_jk-tp22356983p22361683.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat and Apache with mod_jk
Posted by buzzterrier <te...@gmail.com>.
Actually this may not be just a case of the end user clicking away. There is
an Internet Explorer issue with downloading files, where IE deletes the
cached file before the user can load it. Your tomcat logs would probably
show an error like: ClientAbortException: java.io.IOException. I am
suffering with this at present, and the suggestion of setting the
cache-control headers to max-age=0 does not work.
-----
Buzzterrier
http://buzzterrier.blogspot.com/ View my blog: Ordinary Average Developer...
--
View this message in context: http://www.nabble.com/Tomcat-and-Apache-with-mod_jk-tp22356983p22360196.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat and Apache with mod_jk
Posted by "Anthony J. Biacco" <ab...@formatdynamics.com>.
You might want to have them not hit / but hit a small static file that
apache will always have, maybe even keep it in cache. No sense in them
monitoring / which you've shown goes through to your tomcat. Seems a
little wasteful. Unless of course, you want their monitoring to hit
tomcat.
-Tony
---------------------------
Manager, IT Operations
Format Dynamics, Inc.
303-573-1800x27
abiacco@formatdynamics.com
http://www.formatdynamics.com
-----Original Message-----
From: Ian Long [mailto:ian.long@opterus.com]
Sent: Thursday, March 05, 2009 4:44 PM
To: Tomcat Users List
Subject: Re: Tomcat and Apache with mod_jk
Here is some sample output from the system-status:
Srv PID Acc M CPU SS Req Conn Child
Slot Client VHost Request
0-0 2096 0/50/50 _ 0.01 23 2 0.0 0.09
0.09 X web1.opterus.com GET /
lbcheck.faces HTTP/1.0
1-0 2097 0/12/12 W 0.01 513 0 0.0 0.05
0.05 X web1.opterus.com GET /
HTTP/1.1
2-0 2098 0/24/24 W 0.00 450 0 0.0 0.01
0.01 X web1.opterus.com GET /
login.faces HTTP/1.1
3-0 2099 0/39/39 W 0.01 207 0 0.0 0.06
0.06 X web1.opterus.com GET /
HTTP/1.1
4-0 2100 0/11/11 W 0.00 518 0 0.0 0.01
0.01 X web1.opterus.com GET /
HTTP/1.1
5-0 2101 0/31/31 W 0.00 271 0 0.0 0.06
0.06 X web1.opterus.com GET /
HTTP/1.0
The 'Get /' is from my data center provider, it's a piece of
monitoring software, but it looks like it doesn't wait for a response,
because the processes
are stuck in writing, and apache doesn't seem to realize the
connection is gone, as you can see some of the SS times are over 10
minutes, which is longer than
all my timeout settings...
Is there any way to make apache realize the connections are gone? I
will talk to the provider about what they see on their end....
Cheers,
Ian Long
Chief Software Architect, Opterus Inc.
Email: ian.long@opterus.com
Phone: 416-840-8495 x666
Mobile: 416-817-9206
Web: http://www.opterus.com
On 5-Mar-09, at 4:03 PM, Rainer Jung wrote:
> On 05.03.2009 21:04, Ian Long wrote:
>> I haven't actually - I will turn it on and restart the server
>> tonight.
>>
>> Another thing that I find strange is the apache (using prefork MPM)
>> seems to keep increasing the number of child processes. For example,
>> when I run:
>>
>> service httpd status
>> httpd (pid 31093 31055 31048 31047 31046 31045 31044 31043 31042
>> 31036
>> 30922 30919 30860 30853 30769 30753 30733 30728 30690 30684 30653
>> 30652
>> 30637 30555 30440 30439 30438 30421 30383 30348 30312 30310 30309
>> 30271
>> 30177 30158 30157 30086 30063 30057 29906 29905 29904 29796 29795
>> 29794
>> 29793 29791 29790 29701 29639 29638 29633 29600 29547 29530 29529
>> 29491
>> 29489 29480 29443 29442 29441 29440 29397 29396 29394 29358 29357
>> 29351
>> 29304 29270 29191 29190 29184 29138 29137 29116 29115 29051 29012
>> 29007
>> 29002 28966 28959 28957 28911 28904 28903 28897 28893 28848 28809
>> 28807
>> 28796 28790 28782 28773 28734 28719 28645 28544 28543 28542 28475
>> 28474
>> 28473 28471 28470 28428 28403 28365 28363 28325 28319 28318 28316
>> 28315
>> 28314 28271 28269 28228 28184 28183 28138 28132 28125 28063 28022
>> 28016
>> 28015 28014 28013 28011 27968 27966 27965 27964 27963 27882 27836
>> 27815
>> 27777 27776 27689 27688 27687 27672 27671 27617 27615 27610 27568
>> 27567
>> 27560 27559 27523 27522 27521 27477 27476 27475 27474 27473 27467
>> 27466
>> 27465 27464 27457 27456 27455 27454 26439 26222 26045 26003 25855
>> 25854
>> 25853 25852 25851 25850 25849 25848 25846) is running...
>>
>> the number of processes is always increasing....
>
> That's definitely something to worry about. Your configuration lets
> us expect only 20 idle processes. So either the configuration is
> something different, or those processes are not idle.
>
> What does the httpd server-status tell you? Are those processes
> working on something?
>
> They are most likely the reason for the many established
> connections, so it will be interesting to find out, what they do.
>
> You can also use gstack or gdb to look at the stack of some of them
> and get an idea, what they do. In case they are inside mod_jk and
> wait for an answer from the backend, do a "kill -QUIT" on the Tomcat
> and look in catalina.out. It will contain a list of all Tomcat
> threads and for each thread the full method stack, so you can see,
> whether they are running requests, and what it is they are working
> on or waiting for.
>
> Regards,
>
> Rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat and Apache with mod_jk
Posted by Ian Long <ia...@opterus.com>.
The 64.225.156.250 is the IP address that apache httpd and tomcat are
running on. To me it looks as if there is a 'default' host that those
'GET /' requests are being handled by. I assume
that VHost is the virtual host within tomcat that actually handled the
request, based on the <Host> definition in server.xml.
To me, VHost should only ever show the name of the configured <Host>
elements.
I will try the thread dump on the weekend when the site is less busy.
Cheers,
Ian Long
Chief Software Architect, Opterus Inc.
Email: ian.long@opterus.com
Phone: 416-840-8495 x666
Mobile: 416-817-9206
Web: http://www.opterus.com
On 6-Mar-09, at 3:58 PM, Rainer Jung wrote:
> On 06.03.2009 21:02, Ian Long wrote:
>> I have fixed the problem by stopping the https ping so that these
>> stuck
>> processes don't happen, but am digging into the problem a bit
>> further.
>>
>> I think you are right that it's apache waiting for tomcat to send
>> data
>> back for the request, because if I look at the tomcat manager
>> application, I have entries like the following:
>>
>> Stage Time B Sent B Recv Client VHost Request
>> S 28515259 ms 0 KB 0 KB 216.25.10.81 64.225.156.250 GET / HTTP/1.1
>> S 16878407 ms 0 KB 0 KB 76.64.59.226 ops-center.opterus.net GET
>> /login.faces HTTP/1.1
>>
>> The VHost lists the ip address 64.225.156.250 instead of
>> ops-center.opterus.net as it does for 'proper' access.
>>
>> The relevant bit of tomcat config:
>>
>> <Engine name="Catalina" defaultHost="ops-center.opterus.net">
>> <Host appBase="webapps" name="ops-center.opterus.net"
>> unpackWARs="true"
>> autoDeploy="false" deployOnStartup="false" >
>> <Context path="" allowLinking="true" docBase="ops-center-application"
>> debug="0" reloadable="false" >
>> <Manager className="org.apache.catalina.session.StandardManager"
>> pathname="" />
>> </Context>
>> <Context path="/manager" docBase="manager" debug="0"
>> privileged="true"
>> reloadable="false" />
>> <Logger className="org.apache.catalina.logger.FileLogger"
>> prefix="ops-center.opterus.net." suffix=".txt" timestamp="true"/>
>> </Host>
>>
>> <Host appBase="webapps" name="webservice.ops-center.opterus.net"
>> unpackWARs="true" autoDeploy="false" deployOnStartup="false" >
>> <Context path="" allowLinking="true" docBase="ops-center-webservice"
>> debug="0" reloadable="false" >
>> <Manager className="org.apache.catalina.session.StandardManager"
>> pathname="" />
>> </Context>
>> <Logger className="org.apache.catalina.logger.FileLogger"
>> prefix="webservice.ops-center.opterus.net." suffix=".txt"
>> timestamp="true"/>
>> </Host>
>>
>> My JkMount is straigtforward:
>>
>> JkMount /* ajp13
>> jKMount / ajp13
>>
>> Since the defaultHost is ops-center.opterus.net, I would assume they
>> should all go to that host, and not /.
>>
>> From all that I have read this looks like it is setup correctly, but
>> why is VHost 64.225.156.250 (I'm not even sure what that is)
>> receiving
>> the 'GET /' request?
>
> You can check what this IP resolves to, and also what your VHost's
> name resolves to with nslookup:
>
> nslookup 64.225.156.250
> nslookup ops-center.opterus.net
>
> I didn't check, what the manager actually displays in the VHost
> column, but it's also possible, that it shows the string from a host
> header, that might have been part of the request, like
>
> GET / HTTP/1.1
> Host: 64.225.156.250
> Connection: Keep-Alive
>
> You can send such a request to any host, not just 64.225.156.250.
> But it is only speculation, that the manager VHost column might show
> the name from the host header.
>
> You can log a host header in your Tomcat access log (also in the
> httpd access log) by extending the log format with "%{Host}i".
>
> Anyways: the strange IP doesn't really explain, why those requests
> get stuck. And even though you now stopped those, it sounds bad,
> that you can fill up your app that easily with a lot of hanging
> requests. If you can reproduce I definitely recommend using a thread
> dump (kill -QUIT against Tomcat) to check what those threads are
> doing (results in catalina.out).
>
> Regards,
>
> Rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat and Apache with mod_jk
Posted by Rainer Jung <ra...@kippdata.de>.
On 06.03.2009 21:02, Ian Long wrote:
> I have fixed the problem by stopping the https ping so that these stuck
> processes don't happen, but am digging into the problem a bit further.
>
> I think you are right that it's apache waiting for tomcat to send data
> back for the request, because if I look at the tomcat manager
> application, I have entries like the following:
>
> Stage Time B Sent B Recv Client VHost Request
> S 28515259 ms 0 KB 0 KB 216.25.10.81 64.225.156.250 GET / HTTP/1.1
> S 16878407 ms 0 KB 0 KB 76.64.59.226 ops-center.opterus.net GET
> /login.faces HTTP/1.1
>
> The VHost lists the ip address 64.225.156.250 instead of
> ops-center.opterus.net as it does for 'proper' access.
>
> The relevant bit of tomcat config:
>
> <Engine name="Catalina" defaultHost="ops-center.opterus.net">
> <Host appBase="webapps" name="ops-center.opterus.net" unpackWARs="true"
> autoDeploy="false" deployOnStartup="false" >
> <Context path="" allowLinking="true" docBase="ops-center-application"
> debug="0" reloadable="false" >
> <Manager className="org.apache.catalina.session.StandardManager"
> pathname="" />
> </Context>
> <Context path="/manager" docBase="manager" debug="0" privileged="true"
> reloadable="false" />
> <Logger className="org.apache.catalina.logger.FileLogger"
> prefix="ops-center.opterus.net." suffix=".txt" timestamp="true"/>
> </Host>
>
> <Host appBase="webapps" name="webservice.ops-center.opterus.net"
> unpackWARs="true" autoDeploy="false" deployOnStartup="false" >
> <Context path="" allowLinking="true" docBase="ops-center-webservice"
> debug="0" reloadable="false" >
> <Manager className="org.apache.catalina.session.StandardManager"
> pathname="" />
> </Context>
> <Logger className="org.apache.catalina.logger.FileLogger"
> prefix="webservice.ops-center.opterus.net." suffix=".txt"
> timestamp="true"/>
> </Host>
>
> My JkMount is straigtforward:
>
> JkMount /* ajp13
> jKMount / ajp13
>
> Since the defaultHost is ops-center.opterus.net, I would assume they
> should all go to that host, and not /.
>
> From all that I have read this looks like it is setup correctly, but
> why is VHost 64.225.156.250 (I'm not even sure what that is) receiving
> the 'GET /' request?
You can check what this IP resolves to, and also what your VHost's name
resolves to with nslookup:
nslookup 64.225.156.250
nslookup ops-center.opterus.net
I didn't check, what the manager actually displays in the VHost column,
but it's also possible, that it shows the string from a host header,
that might have been part of the request, like
GET / HTTP/1.1
Host: 64.225.156.250
Connection: Keep-Alive
You can send such a request to any host, not just 64.225.156.250. But it
is only speculation, that the manager VHost column might show the name
from the host header.
You can log a host header in your Tomcat access log (also in the httpd
access log) by extending the log format with "%{Host}i".
Anyways: the strange IP doesn't really explain, why those requests get
stuck. And even though you now stopped those, it sounds bad, that you
can fill up your app that easily with a lot of hanging requests. If you
can reproduce I definitely recommend using a thread dump (kill -QUIT
against Tomcat) to check what those threads are doing (results in
catalina.out).
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat and Apache with mod_jk
Posted by Ian Long <ia...@opterus.com>.
I have fixed the problem by stopping the https ping so that these
stuck processes don't happen, but am digging into the problem a bit
further.
I think you are right that it's apache waiting for tomcat to send data
back for the request, because if I look at the tomcat manager
application, I have entries like the following:
Stage Time B Sent B Recv Client VHost Request
S 28515259 ms 0 KB 0 KB 216.25.10.81 64.225.156.250 GET / HTTP/1.1
S 16878407 ms 0 KB 0 KB 76.64.59.226 ops-center.opterus.net GET /
login.faces HTTP/1.1
The VHost lists the ip address 64.225.156.250 instead of ops-
center.opterus.net as it does for 'proper' access.
The relevant bit of tomcat config:
<Engine name="Catalina" defaultHost="ops-center.opterus.net">
<Host appBase="webapps" name="ops-center.opterus.net"
unpackWARs="true" autoDeploy="false" deployOnStartup="false" >
<Context path="" allowLinking="true" docBase="ops-center-application"
debug="0" reloadable="false" >
<Manager className="org.apache.catalina.session.StandardManager"
pathname="" />
</Context>
<Context path="/manager" docBase="manager" debug="0" privileged="true"
reloadable="false" />
<Logger className="org.apache.catalina.logger.FileLogger" prefix="ops-
center.opterus.net." suffix=".txt" timestamp="true"/>
</Host>
<Host appBase="webapps" name="webservice.ops-center.opterus.net"
unpackWARs="true" autoDeploy="false" deployOnStartup="false" >
<Context path="" allowLinking="true" docBase="ops-center-webservice"
debug="0" reloadable="false" >
<Manager className="org.apache.catalina.session.StandardManager"
pathname="" />
</Context>
<Logger className="org.apache.catalina.logger.FileLogger"
prefix="webservice.ops-center.opterus.net." suffix=".txt"
timestamp="true"/>
</Host>
My JkMount is straigtforward:
JkMount /* ajp13
jKMount / ajp13
Since the defaultHost is ops-center.opterus.net, I would assume they
should all go to that host, and not /.
From all that I have read this looks like it is setup correctly, but
why is VHost 64.225.156.250 (I'm not even sure what that is) receiving
the 'GET /' request?
Thanks,
Ian Long
Chief Software Architect, Opterus Inc.
Email: ian.long@opterus.com
Phone: 416-840-8495 x666
Mobile: 416-817-9206
Web: http://www.opterus.com
On 5-Mar-09, at 7:05 PM, Rainer Jung wrote:
> On 06.03.2009 00:44, Ian Long wrote:
>> Here is some sample output from the system-status:
>>
>> Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
>> 0-0 2096 0/50/50 _ 0.01 23 2 0.0 0.09 0.09 X web1.opterus.com GET
>> /lbcheck.faces HTTP/1.0
>> 1-0 2097 0/12/12 W 0.01 513 0 0.0 0.05 0.05 X web1.opterus.com GET /
>> HTTP/1.1
>> 2-0 2098 0/24/24 W 0.00 450 0 0.0 0.01 0.01 X web1.opterus.com GET
>> /login.faces HTTP/1.1
>> 3-0 2099 0/39/39 W 0.01 207 0 0.0 0.06 0.06 X web1.opterus.com GET /
>> HTTP/1.1
>> 4-0 2100 0/11/11 W 0.00 518 0 0.0 0.01 0.01 X web1.opterus.com GET /
>> HTTP/1.1
>> 5-0 2101 0/31/31 W 0.00 271 0 0.0 0.06 0.06 X web1.opterus.com GET /
>> HTTP/1.0
>>
>> The 'Get /' is from my data center provider, it's a piece of
>> monitoring
>> software, but it looks like it doesn't wait for a response, because
>> the
>> processes
>> are stuck in writing, and apache doesn't seem to realize the
>> connection
>> is gone, as you can see some of the SS times are over 10 minutes,
>> which
>> is longer than
>> all my timeout settings...
>>
>> Is there any way to make apache realize the connections are gone? I
>> will
>> talk to the provider about what they see on their end....
>
> "W" is not necessarily writing, it could also be waiting for Tomcat
> to produce an answer. If I remember it correctly, you also send "/"
> to Tomcat.
>
> Thos probe-but-not-wait requests could well be responsible for the
> client abort log messages.
>
> Usually once the web server tries to write back the first response
> packet, and the client has already closed or reset the connection,
> the web server will note this immediately.
>
> So somehow I suspect, that those requests are still waiting for the
> result form the backend.
>
> Is it just those few? What are the other 100-200 httpd processes
> doing?
>
> Regards,
>
> Rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat and Apache with mod_jk
Posted by Ian Long <ia...@opterus.com>.
I just took a subset of the list, the other few hundred processes were
also stuck in the same state. From the tomcat logs it looks like the
response has been sent back. I have seen cases where a firewall
doesn't recognize a connection has been closed.
On 5-Mar-09, at 7:05 PM, Rainer Jung wrote:
> On 06.03.2009 00:44, Ian Long wrote:
>> Here is some sample output from the system-status:
>>
>> Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
>> 0-0 2096 0/50/50 _ 0.01 23 2 0.0 0.09 0.09 X web1.opterus.com GET
>> /lbcheck.faces HTTP/1.0
>> 1-0 2097 0/12/12 W 0.01 513 0 0.0 0.05 0.05 X web1.opterus.com GET /
>> HTTP/1.1
>> 2-0 2098 0/24/24 W 0.00 450 0 0.0 0.01 0.01 X web1.opterus.com GET
>> /login.faces HTTP/1.1
>> 3-0 2099 0/39/39 W 0.01 207 0 0.0 0.06 0.06 X web1.opterus.com GET /
>> HTTP/1.1
>> 4-0 2100 0/11/11 W 0.00 518 0 0.0 0.01 0.01 X web1.opterus.com GET /
>> HTTP/1.1
>> 5-0 2101 0/31/31 W 0.00 271 0 0.0 0.06 0.06 X web1.opterus.com GET /
>> HTTP/1.0
>>
>> The 'Get /' is from my data center provider, it's a piece of
>> monitoring
>> software, but it looks like it doesn't wait for a response, because
>> the
>> processes
>> are stuck in writing, and apache doesn't seem to realize the
>> connection
>> is gone, as you can see some of the SS times are over 10 minutes,
>> which
>> is longer than
>> all my timeout settings...
>>
>> Is there any way to make apache realize the connections are gone? I
>> will
>> talk to the provider about what they see on their end....
>
> "W" is not necessarily writing, it could also be waiting for Tomcat
> to produce an answer. If I remember it correctly, you also send "/"
> to Tomcat.
>
> Thos probe-but-not-wait requests could well be responsible for the
> client abort log messages.
>
> Usually once the web server tries to write back the first response
> packet, and the client has already closed or reset the connection,
> the web server will note this immediately.
>
> So somehow I suspect, that those requests are still waiting for the
> result form the backend.
>
> Is it just those few? What are the other 100-200 httpd processes
> doing?
>
> Regards,
>
> Rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat and Apache with mod_jk
Posted by Rainer Jung <ra...@kippdata.de>.
On 06.03.2009 00:44, Ian Long wrote:
> Here is some sample output from the system-status:
>
> Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
> 0-0 2096 0/50/50 _ 0.01 23 2 0.0 0.09 0.09 X web1.opterus.com GET
> /lbcheck.faces HTTP/1.0
> 1-0 2097 0/12/12 W 0.01 513 0 0.0 0.05 0.05 X web1.opterus.com GET /
> HTTP/1.1
> 2-0 2098 0/24/24 W 0.00 450 0 0.0 0.01 0.01 X web1.opterus.com GET
> /login.faces HTTP/1.1
> 3-0 2099 0/39/39 W 0.01 207 0 0.0 0.06 0.06 X web1.opterus.com GET /
> HTTP/1.1
> 4-0 2100 0/11/11 W 0.00 518 0 0.0 0.01 0.01 X web1.opterus.com GET /
> HTTP/1.1
> 5-0 2101 0/31/31 W 0.00 271 0 0.0 0.06 0.06 X web1.opterus.com GET /
> HTTP/1.0
>
> The 'Get /' is from my data center provider, it's a piece of monitoring
> software, but it looks like it doesn't wait for a response, because the
> processes
> are stuck in writing, and apache doesn't seem to realize the connection
> is gone, as you can see some of the SS times are over 10 minutes, which
> is longer than
> all my timeout settings...
>
> Is there any way to make apache realize the connections are gone? I will
> talk to the provider about what they see on their end....
"W" is not necessarily writing, it could also be waiting for Tomcat to
produce an answer. If I remember it correctly, you also send "/" to Tomcat.
Thos probe-but-not-wait requests could well be responsible for the
client abort log messages.
Usually once the web server tries to write back the first response
packet, and the client has already closed or reset the connection, the
web server will note this immediately.
So somehow I suspect, that those requests are still waiting for the
result form the backend.
Is it just those few? What are the other 100-200 httpd processes doing?
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat and Apache with mod_jk
Posted by Ian Long <ia...@opterus.com>.
Here is some sample output from the system-status:
Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 2096 0/50/50 _ 0.01 23 2 0.0 0.09 0.09 X web1.opterus.com GET /
lbcheck.faces HTTP/1.0
1-0 2097 0/12/12 W 0.01 513 0 0.0 0.05 0.05 X web1.opterus.com GET /
HTTP/1.1
2-0 2098 0/24/24 W 0.00 450 0 0.0 0.01 0.01 X web1.opterus.com GET /
login.faces HTTP/1.1
3-0 2099 0/39/39 W 0.01 207 0 0.0 0.06 0.06 X web1.opterus.com GET /
HTTP/1.1
4-0 2100 0/11/11 W 0.00 518 0 0.0 0.01 0.01 X web1.opterus.com GET /
HTTP/1.1
5-0 2101 0/31/31 W 0.00 271 0 0.0 0.06 0.06 X web1.opterus.com GET /
HTTP/1.0
The 'Get /' is from my data center provider, it's a piece of
monitoring software, but it looks like it doesn't wait for a response,
because the processes
are stuck in writing, and apache doesn't seem to realize the
connection is gone, as you can see some of the SS times are over 10
minutes, which is longer than
all my timeout settings...
Is there any way to make apache realize the connections are gone? I
will talk to the provider about what they see on their end....
Cheers,
Ian Long
Chief Software Architect, Opterus Inc.
Email: ian.long@opterus.com
Phone: 416-840-8495 x666
Mobile: 416-817-9206
Web: http://www.opterus.com
On 5-Mar-09, at 4:03 PM, Rainer Jung wrote:
> On 05.03.2009 21:04, Ian Long wrote:
>> I haven't actually - I will turn it on and restart the server
>> tonight.
>>
>> Another thing that I find strange is the apache (using prefork MPM)
>> seems to keep increasing the number of child processes. For example,
>> when I run:
>>
>> service httpd status
>> httpd (pid 31093 31055 31048 31047 31046 31045 31044 31043 31042
>> 31036
>> 30922 30919 30860 30853 30769 30753 30733 30728 30690 30684 30653
>> 30652
>> 30637 30555 30440 30439 30438 30421 30383 30348 30312 30310 30309
>> 30271
>> 30177 30158 30157 30086 30063 30057 29906 29905 29904 29796 29795
>> 29794
>> 29793 29791 29790 29701 29639 29638 29633 29600 29547 29530 29529
>> 29491
>> 29489 29480 29443 29442 29441 29440 29397 29396 29394 29358 29357
>> 29351
>> 29304 29270 29191 29190 29184 29138 29137 29116 29115 29051 29012
>> 29007
>> 29002 28966 28959 28957 28911 28904 28903 28897 28893 28848 28809
>> 28807
>> 28796 28790 28782 28773 28734 28719 28645 28544 28543 28542 28475
>> 28474
>> 28473 28471 28470 28428 28403 28365 28363 28325 28319 28318 28316
>> 28315
>> 28314 28271 28269 28228 28184 28183 28138 28132 28125 28063 28022
>> 28016
>> 28015 28014 28013 28011 27968 27966 27965 27964 27963 27882 27836
>> 27815
>> 27777 27776 27689 27688 27687 27672 27671 27617 27615 27610 27568
>> 27567
>> 27560 27559 27523 27522 27521 27477 27476 27475 27474 27473 27467
>> 27466
>> 27465 27464 27457 27456 27455 27454 26439 26222 26045 26003 25855
>> 25854
>> 25853 25852 25851 25850 25849 25848 25846) is running...
>>
>> the number of processes is always increasing....
>
> That's definitely something to worry about. Your configuration lets
> us expect only 20 idle processes. So either the configuration is
> something different, or those processes are not idle.
>
> What does the httpd server-status tell you? Are those processes
> working on something?
>
> They are most likely the reason for the many established
> connections, so it will be interesting to find out, what they do.
>
> You can also use gstack or gdb to look at the stack of some of them
> and get an idea, what they do. In case they are inside mod_jk and
> wait for an answer from the backend, do a "kill -QUIT" on the Tomcat
> and look in catalina.out. It will contain a list of all Tomcat
> threads and for each thread the full method stack, so you can see,
> whether they are running requests, and what it is they are working
> on or waiting for.
>
> Regards,
>
> Rainer
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat and Apache with mod_jk
Posted by Rainer Jung <ra...@kippdata.de>.
On 05.03.2009 21:04, Ian Long wrote:
> I haven't actually - I will turn it on and restart the server tonight.
>
> Another thing that I find strange is the apache (using prefork MPM)
> seems to keep increasing the number of child processes. For example,
> when I run:
>
> service httpd status
> httpd (pid 31093 31055 31048 31047 31046 31045 31044 31043 31042 31036
> 30922 30919 30860 30853 30769 30753 30733 30728 30690 30684 30653 30652
> 30637 30555 30440 30439 30438 30421 30383 30348 30312 30310 30309 30271
> 30177 30158 30157 30086 30063 30057 29906 29905 29904 29796 29795 29794
> 29793 29791 29790 29701 29639 29638 29633 29600 29547 29530 29529 29491
> 29489 29480 29443 29442 29441 29440 29397 29396 29394 29358 29357 29351
> 29304 29270 29191 29190 29184 29138 29137 29116 29115 29051 29012 29007
> 29002 28966 28959 28957 28911 28904 28903 28897 28893 28848 28809 28807
> 28796 28790 28782 28773 28734 28719 28645 28544 28543 28542 28475 28474
> 28473 28471 28470 28428 28403 28365 28363 28325 28319 28318 28316 28315
> 28314 28271 28269 28228 28184 28183 28138 28132 28125 28063 28022 28016
> 28015 28014 28013 28011 27968 27966 27965 27964 27963 27882 27836 27815
> 27777 27776 27689 27688 27687 27672 27671 27617 27615 27610 27568 27567
> 27560 27559 27523 27522 27521 27477 27476 27475 27474 27473 27467 27466
> 27465 27464 27457 27456 27455 27454 26439 26222 26045 26003 25855 25854
> 25853 25852 25851 25850 25849 25848 25846) is running...
>
> the number of processes is always increasing....
That's definitely something to worry about. Your configuration lets us
expect only 20 idle processes. So either the configuration is something
different, or those processes are not idle.
What does the httpd server-status tell you? Are those processes working
on something?
They are most likely the reason for the many established connections, so
it will be interesting to find out, what they do.
You can also use gstack or gdb to look at the stack of some of them and
get an idea, what they do. In case they are inside mod_jk and wait for
an answer from the backend, do a "kill -QUIT" on the Tomcat and look in
catalina.out. It will contain a list of all Tomcat threads and for each
thread the full method stack, so you can see, whether they are running
requests, and what it is they are working on or waiting for.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat and Apache with mod_jk
Posted by Ian Long <ia...@opterus.com>.
I haven't actually - I will turn it on and restart the server tonight.
Another thing that I find strange is the apache (using prefork MPM)
seems to keep increasing the number of child processes. For example,
when I run:
service httpd status
httpd (pid 31093 31055 31048 31047 31046 31045 31044 31043 31042 31036
30922 30919 30860 30853 30769 30753 30733 30728 30690 30684 30653
30652 30637 30555 30440 30439 30438 30421 30383 30348 30312 30310
30309 30271 30177 30158 30157 30086 30063 30057 29906 29905 29904
29796 29795 29794 29793 29791 29790 29701 29639 29638 29633 29600
29547 29530 29529 29491 29489 29480 29443 29442 29441 29440 29397
29396 29394 29358 29357 29351 29304 29270 29191 29190 29184 29138
29137 29116 29115 29051 29012 29007 29002 28966 28959 28957 28911
28904 28903 28897 28893 28848 28809 28807 28796 28790 28782 28773
28734 28719 28645 28544 28543 28542 28475 28474 28473 28471 28470
28428 28403 28365 28363 28325 28319 28318 28316 28315 28314 28271
28269 28228 28184 28183 28138 28132 28125 28063 28022 28016 28015
28014 28013 28011 27968 27966 27965 27964 27963 27882 27836 27815
27777 27776 27689 27688 27687 27672 27671 27617 27615 27610 27568
27567 27560 27559 27523 27522 27521 27477 27476 27475 27474 27473
27467 27466 27465 27464 27457 27456 27455 27454 26439 26222 26045
26003 25855 25854 25853 25852 25851 25850 25849 25848 25846) is
running...
the number of processes is always increasing....
Thanks,
Ian Long
Chief Software Architect, Opterus Inc.
Email: ian.long@opterus.com
Phone: 416-840-8495 x666
Mobile: 416-817-9206
Web: http://www.opterus.com
On 5-Mar-09, at 1:32 PM, Anthony J. Biacco wrote:
> Have you counted the actual # of requests for both apaches via
> something
> like server-status, or looked in mod_jk.log, that they are both
> receiving an equal # of successful requests?
> The "Writing to client aborted or client network problems" are nothing
> to really worry about. They usually mean the browser/client has closed
> the connection (clicked Stop or clicked Back) before the request was
> done being processed/sent.
>
> -Tony
> ---------------------------
> Manager, IT Operations
> Format Dynamics, Inc.
> 303-573-1800x27
> abiacco@formatdynamics.com
> http://www.formatdynamics.com
>
>
> -----Original Message-----
> From: Ian Long [mailto:ian.long@opterus.com]
> Sent: Thursday, March 05, 2009 10:57 AM
> To: users@tomcat.apache.org
> Subject: Tomcat and Apache with mod_jk
>
> Hi,
>
> I have dedicated RHEL5 boxes (dedicate quad core, 4GB RAM) setup with
> both apache httpd and tomcat running on the same box, with a hardware
> load balancer selecting the node a user is routed to. I have mod_jk
> setup, but am seeing some behaviour I'm not sure is correct. This is
> on httpd 2.2.3, tomcat 6.0.16, and mod_lk version of 1.2.26. The load
> balancer pings the machines every 2 seconds as part of a health check.
>
>
> Below are the relevant bits of my configuration:
>
> mod_jk.conf
> ==========
> LoadModule jk_module modules/mod_jk.so
> JkLogFile /var/log/httpd/mod_jk.log
> JkLogLevel info
> JkWorkersFile /etc/httpd/conf.d/workers.properties
> JkShmFile /var/log/httpd/jk.shm
> JkShmSize 800
> JkOptions +ForwardURICompatUnparsed
> JkMount /* ajp13
> JkUnMount /cacti/* ajp13
> JkUnMount /manager/* ajp13
>
> workers.properties
> ==============
> worker.list=ajp13
> worker.ajp13.port=8009
> worker.ajp13.host=localhost
> worker.ajp13.maintain=60
> worker.ajp13.type=ajp13
> # worker.ajp13.connection_pool_minsize=0
> worker.ajp13.connection_pool_timeout=600
> worker.ajp13.connect_timeout=10000
> worker.ajp13.prepost_timeout=10000
> worker.ajp13.max_packet_size=16384
>
> httpd.conf
> ========
> <IfModule prefork.c>
> StartServers 8
> MinSpareServers 5
> MaxSpareServers 20
> ServerLimit 800
> MaxClients 800
> MaxRequestsPerChild 0
> </IfModule>
>
> server.xml
> ========
> <Connector port="8009" connectionTimeout="600000"
> minSpareThreads="5" packetSize="16384" address="127.0.0.1"
> enableLookups="false" disableUploadTimeout="true"
> maxSpareThreads="75" maxThreads="800" protocol="AJP/1.3" />
>
> On one machine, after 5 or 6 hours, when I run netstat to look at
> connections between http and tomcat, I have 250 or so:
>
> tcp 0 0 ::ffff:127.0.0.1:8009 ::ffff:
> 127.0.0.1:60408 ESTABLISHED
>
> whereas the other machine only has 40 or so, and they are equally load
> balanced. I tried setting worker.ajp13.connection_pool_minsize=0 but
> this has no effect on what I'm seeing.
>
> Is this the behaviour I should expect? The load on the machine is not
> that heavy, at the busiest period there would be 3 request per second
> going to the box.
>
> I wouldn't expect this number to keep growing, especially under light
> load.
>
> I'm also seeing a few errors like the following in mod_jk.log:
>
> [Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
> ajp_process_callback::jk_ajp_common.c (1603): Writing to client
> aborted or client network problems
> [Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
> ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
> failed (unrecoverable), because of client write error (attempt=1)
> [Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
> jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
> [Thu Mar 05 10:34:28.527 2009] [26222:3086382864] [info]
> ajp_process_callback::jk_ajp_common.c (1603): Writing to client
> aborted or client network problems
> [Thu Mar 05 10:34:28.527 2009] [26222:3086382864] [info]
> ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
> failed (unrecoverable), because of client write error (attempt=1)
> [Thu Mar 05 10:34:28.528 2009] [26222:3086382864] [info]
> jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
> [Thu Mar 05 10:36:01.071 2009] [25855:3086382864] [info]
> ajp_process_callback::jk_ajp_common.c (1603): Writing to client
> aborted or client network problems
> [Thu Mar 05 10:36:01.072 2009] [25855:3086382864] [info]
> ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
> failed (unrecoverable), because of client write error (attempt=1)
> [Thu Mar 05 10:36:01.072 2009] [25855:3086382864] [info]
> jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
> [Thu Mar 05 11:53:42.298 2009] [27523:3086382864] [info]
> ajp_process_callback::jk_ajp_common.c (1603): Writing to client
> aborted or client network problems
> [Thu Mar 05 11:53:42.298 2009] [27523:3086382864] [info]
> ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
> failed (unrecoverable), because of client write error (attempt=1)
> [Thu Mar 05 11:53:42.298 2009] [27523:3086382864] [info]
> jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
> [Thu Mar 05 12:08:07.551 2009] [28365:3086382864] [info]
> ajp_process_callback::jk_ajp_common.c (1603): Writing to client
> aborted or client network problems
> [Thu Mar 05 12:08:07.551 2009] [28365:3086382864] [info]
> ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
> failed (unrecoverable), because of client write error (attempt=1)
> [Thu Mar 05 12:08:07.551 2009] [28365:3086382864] [info]
> jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
> [Thu Mar 05 12:08:17.432 2009] [28428:3086382864] [info]
> ajp_process_callback::jk_ajp_common.c (1603): Writing to client
> aborted or client network problems
> [Thu Mar 05 12:08:17.432 2009] [28428:3086382864] [info]
> ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
> failed (unrecoverable), because of client write error (attempt=1)
> [Thu Mar 05 12:08:17.433 2009] [28428:3086382864] [info]
> jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
> [Thu Mar 05 12:12:19.656 2009] [28363:3086382864] [info]
> ajp_process_callback::jk_ajp_common.c (1603): Writing to client
> aborted or client network problems
> [Thu Mar 05 12:12:19.656 2009] [28363:3086382864] [info]
> ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
> failed (unrecoverable), because of client write error (attempt=1)
> [Thu Mar 05 12:12:19.657 2009] [28363:3086382864] [info]
> jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
>
> Can anyone see anything wrong with the configuration?
>
> Cheers,
> Ian Long
> Chief Software Architect, Opterus Inc.
>
> Email: ian.long@opterus.com
> Phone: 416-840-8495 x666
> Mobile: 416-817-9206
> Web: http://www.opterus.com
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat and Apache with mod_jk
Posted by "Anthony J. Biacco" <ab...@formatdynamics.com>.
Have you counted the actual # of requests for both apaches via something
like server-status, or looked in mod_jk.log, that they are both
receiving an equal # of successful requests?
The "Writing to client aborted or client network problems" are nothing
to really worry about. They usually mean the browser/client has closed
the connection (clicked Stop or clicked Back) before the request was
done being processed/sent.
-Tony
---------------------------
Manager, IT Operations
Format Dynamics, Inc.
303-573-1800x27
abiacco@formatdynamics.com
http://www.formatdynamics.com
-----Original Message-----
From: Ian Long [mailto:ian.long@opterus.com]
Sent: Thursday, March 05, 2009 10:57 AM
To: users@tomcat.apache.org
Subject: Tomcat and Apache with mod_jk
Hi,
I have dedicated RHEL5 boxes (dedicate quad core, 4GB RAM) setup with
both apache httpd and tomcat running on the same box, with a hardware
load balancer selecting the node a user is routed to. I have mod_jk
setup, but am seeing some behaviour I'm not sure is correct. This is
on httpd 2.2.3, tomcat 6.0.16, and mod_lk version of 1.2.26. The load
balancer pings the machines every 2 seconds as part of a health check.
Below are the relevant bits of my configuration:
mod_jk.conf
==========
LoadModule jk_module modules/mod_jk.so
JkLogFile /var/log/httpd/mod_jk.log
JkLogLevel info
JkWorkersFile /etc/httpd/conf.d/workers.properties
JkShmFile /var/log/httpd/jk.shm
JkShmSize 800
JkOptions +ForwardURICompatUnparsed
JkMount /* ajp13
JkUnMount /cacti/* ajp13
JkUnMount /manager/* ajp13
workers.properties
==============
worker.list=ajp13
worker.ajp13.port=8009
worker.ajp13.host=localhost
worker.ajp13.maintain=60
worker.ajp13.type=ajp13
# worker.ajp13.connection_pool_minsize=0
worker.ajp13.connection_pool_timeout=600
worker.ajp13.connect_timeout=10000
worker.ajp13.prepost_timeout=10000
worker.ajp13.max_packet_size=16384
httpd.conf
========
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 800
MaxClients 800
MaxRequestsPerChild 0
</IfModule>
server.xml
========
<Connector port="8009" connectionTimeout="600000"
minSpareThreads="5" packetSize="16384" address="127.0.0.1"
enableLookups="false" disableUploadTimeout="true"
maxSpareThreads="75" maxThreads="800" protocol="AJP/1.3" />
On one machine, after 5 or 6 hours, when I run netstat to look at
connections between http and tomcat, I have 250 or so:
tcp 0 0 ::ffff:127.0.0.1:8009 ::ffff:
127.0.0.1:60408 ESTABLISHED
whereas the other machine only has 40 or so, and they are equally load
balanced. I tried setting worker.ajp13.connection_pool_minsize=0 but
this has no effect on what I'm seeing.
Is this the behaviour I should expect? The load on the machine is not
that heavy, at the busiest period there would be 3 request per second
going to the box.
I wouldn't expect this number to keep growing, especially under light
load.
I'm also seeing a few errors like the following in mod_jk.log:
[Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 10:34:28.527 2009] [26222:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 10:34:28.527 2009] [26222:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 10:34:28.528 2009] [26222:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 10:36:01.071 2009] [25855:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 10:36:01.072 2009] [25855:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 10:36:01.072 2009] [25855:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 11:53:42.298 2009] [27523:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 11:53:42.298 2009] [27523:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 11:53:42.298 2009] [27523:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 12:08:07.551 2009] [28365:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 12:08:07.551 2009] [28365:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 12:08:07.551 2009] [28365:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 12:08:17.432 2009] [28428:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 12:08:17.432 2009] [28428:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 12:08:17.433 2009] [28428:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
[Thu Mar 05 12:12:19.656 2009] [28363:3086382864] [info]
ajp_process_callback::jk_ajp_common.c (1603): Writing to client
aborted or client network problems
[Thu Mar 05 12:12:19.656 2009] [28363:3086382864] [info]
ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
failed (unrecoverable), because of client write error (attempt=1)
[Thu Mar 05 12:12:19.657 2009] [28363:3086382864] [info]
jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
Can anyone see anything wrong with the configuration?
Cheers,
Ian Long
Chief Software Architect, Opterus Inc.
Email: ian.long@opterus.com
Phone: 416-840-8495 x666
Mobile: 416-817-9206
Web: http://www.opterus.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat and Apache with mod_jk
Posted by Ian Long <ia...@opterus.com>.
Ok thanks. Based on monitoring, I don't think it's the server taking
a long time to respond, it's probably the users doing as you suggested
- I just wanted to make sure the log file entries weren't serious. I
want to make sure the rising established connection count isn't
something to worry about. I will watch it to make sure it doesn't
keep growing until it reaches the tomcat thread limit, since the
server really isn't that busy.
I would assume that even with a request every second, only a small #
of established connections should be required to service the load.
Basically, the application is unused after 9PM, so I would expect the
# of established connections to be cleaned up as they are no longer
required.
Ian Long
Chief Software Architect, Opterus Inc.
On 5-Mar-09, at 1:38 PM, André Warnier wrote:
> Ian Long wrote:
> [...]
>
> Hi Ian. I don't know about the load balancing part, but the
> following errors :
>> I'm also seeing a few errors like the following in mod_jk.log:
>> [Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
>> ajp_process_callback::jk_ajp_common.c (1603): Writing to client
>> aborted or client network problems
>> [Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
>> ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to
>> tomcat failed (unrecoverable), because of client write error
>> (attempt=1)
>> [Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
>> jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
>> [Thu Mar 05 10:34:28.527 2009] [26222:3086382864] [info]
>> ajp_process_callback::jk_ajp_common.c (1603): Writing to client
>> aborted or client network problems
> etc...
>
> indicate, more often than not, simply that the connection with the
> client was dropped (by the client or something else - but not the
> server), before the server could send the response to the client's
> request.
> For instance, this happens when the user hits the cancel button in
> the browser, or when the user gets impatient and starts clicking all
> over before he gets the server response to his request.
> If you are absolutely sure that neither of the above is the case
> (e.g. you issue a request yourself, and patiently wait, and note
> this message in the logs when it can only be due to your request,
> then it /could/ indicate some network issue.
> One such issue which I have experienced is some piece of software or
> hardware between your browser and the server, dropping the
> connection unilaterally, because for instance the connection was
> idle more than xxx seconds.
> But from the part of the server, it indicates with 100% certainty,
> that the server was trying to send a response to the client, but no
> longer could because the connection was no longer there.
> In the absence of a cancel click or a click on another link, a
> browser client will only "give up" after a request and close the
> connection, after several minutes. That's when you get the kind of
> message "This server does not respond, is maybe down, etc..".
>
> HTH
> (if only inasmuch as the above messages do not lead you into the
> wrong path for the problem you are experiencing. Of course, a user
> clicking on cancel or another link, could be the result of your
> server really taking an inordinate amount of time to respond,
> causing the user to get impatient...)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat and Apache with mod_jk
Posted by André Warnier <aw...@ice-sa.com>.
Ian Long wrote:
[...]
Hi Ian. I don't know about the load balancing part, but the following
errors :
>
> I'm also seeing a few errors like the following in mod_jk.log:
>
> [Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
> ajp_process_callback::jk_ajp_common.c (1603): Writing to client aborted
> or client network problems
> [Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
> ajp_service::jk_ajp_common.c (2186): (ajp13) sending request to tomcat
> failed (unrecoverable), because of client write error (attempt=1)
> [Thu Mar 05 10:34:08.878 2009] [25849:3086382864] [info]
> jk_handler::mod_jk.c (2357): Aborting connection for worker=ajp13
> [Thu Mar 05 10:34:28.527 2009] [26222:3086382864] [info]
> ajp_process_callback::jk_ajp_common.c (1603): Writing to client aborted
> or client network problems
etc...
indicate, more often than not, simply that the connection with the
client was dropped (by the client or something else - but not the
server), before the server could send the response to the client's request.
For instance, this happens when the user hits the cancel button in the
browser, or when the user gets impatient and starts clicking all over
before he gets the server response to his request.
If you are absolutely sure that neither of the above is the case (e.g.
you issue a request yourself, and patiently wait, and note this message
in the logs when it can only be due to your request, then it /could/
indicate some network issue.
One such issue which I have experienced is some piece of software or
hardware between your browser and the server, dropping the connection
unilaterally, because for instance the connection was idle more than xxx
seconds.
But from the part of the server, it indicates with 100% certainty, that
the server was trying to send a response to the client, but no longer
could because the connection was no longer there.
In the absence of a cancel click or a click on another link, a browser
client will only "give up" after a request and close the connection,
after several minutes. That's when you get the kind of message "This
server does not respond, is maybe down, etc..".
HTH
(if only inasmuch as the above messages do not lead you into the wrong
path for the problem you are experiencing. Of course, a user clicking
on cancel or another link, could be the result of your server really
taking an inordinate amount of time to respond, causing the user to get
impatient...)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org