You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Roy McMorran <mc...@mdibl.org> on 2008/12/04 20:39:57 UTC
mod_jk -> ajp hangs indefinitely on some requests
Hello all,
I'm seeing frequent cases where requests sent via mod_jk to Tomcat will
hang indefinitely. The request will never return to Apache and the
client will eventually time out. There are no error messages.
In my test case I run wget against various URLs until the hang occurs
(usually within 2-3 minutes). Here are the last few entries from my
mod_jk.log file (JkLogLevel = trace):
[Thu Dec 04 13:28:08.509 2008] [27224:1] [debug]
ajp_connection_tcp_send_message::jk_ajp_common.c (911): 00c0 FF 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
[Thu Dec 04 13:28:08.510 2008] [27224:1] [trace]
ajp_connection_tcp_send_message::jk_ajp_common.c (928): exit
[Thu Dec 04 13:28:08.510 2008] [27224:1] [debug]
ajp_send_request::jk_ajp_common.c (1325): (ajp13) request body to send 0
- request body to resend 0
[Thu Dec 04 13:28:08.511 2008] [27224:1] [trace]
ajp_send_request::jk_ajp_common.c (1422): exit
[Thu Dec 04 13:28:08.511 2008] [27224:1] [trace]
ajp_get_reply::jk_ajp_common.c (1613): enter
[Thu Dec 04 13:28:08.511 2008] [27224:1] [trace]
ajp_connection_tcp_get_message::jk_ajp_common.c (953): enter
Beyond this point there is no activity. There are never any error
messages generated, not in mod_jk.log, catalina.out, nor in the Apache logs.
There is a thread dump of Tomcat (after the hang begins) at:
http://www.mdibl.org/~mcmorran/tomcat_thread_dump.txt
This is a pretty simple setup, both Apache & Tomcat are running on the
same host, so there is no firewall issue. No load balancing, just one
worker defined.
workers.properties file:
worker.list=ajp13
worker.ajp13.type=ajp13
worker.ajp13.host=localhost
worker.ajp13.port=8009
worker.ajp13.socket_timeout=120
(have tried with and without the socket_timeout parameter - makes no
difference)
My server.xml file is at:
http://www.mdibl.org/~mcmorran/server.xml.txt
(I've tried various settings (or none) for connectionTimeout inside the
AJP connector - makes no difference)
mod_jk related parts of my httpd.conf file are at:
http://www.mdibl.org/~mcmorran/httpd.conf.txt
Version info:
Server: Solaris 10
Apache: 2.2.6
mod_jk: 1.2.25
Tomcat: 6.0.14
libtcnative: 1.10
Java: 1.5.0_14
I'm running Tomcat under jsvc. The JVM gets 2G of memory (there are 8G
total on the system) - I don't think there's any memory issue. I'm in
test mode so I'm the only user; there's virtually no CPU load on the box.
I've reached a dead end here -I'm hoping someone on the list may have
some ideas. Thanks for any help you may be able to provide.
Cheers,
Roy
--
Roy McMorran
Systems Administrator
MDI Biological Laboratory
mcmorran@mdibl.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk -> ajp hangs indefinitely on some requests
Posted by Rainer Jung <ra...@kippdata.de>.
Mark Thomas schrieb:
> Roy McMorran wrote:
>> Roy McMorran wrote:
>>> Despite my misgivings, 1.1.5 seems to have helped. It's run my test
>>> script for about 30 minutes now without problems (previously would
>>> have hung by minute 3).
>>> However!
>>> My dev system (which was already running 1.1.15) still exhibits the
>>> problem. It also has newer versions of Tomcat (6.0.18), Apache
>>> (2.2.10) and mod_jk (1.2.27).
>>>
>> Correction. The combination of updating libtcnative to 1.1.15 _AND_
>> setting "JkOptions +DisableReuse" prevents both systems from hanging.
>>
>> I'm concerned about the performance implications of using this directive
>> based on what I read at
>> http://tomcat.apache.org/connectors-doc/reference/apache.html
>
> Don't be. With httpd and Tomcat on the same box (or even the same LAN) the
> overhead of establishing the TCP connection on every request is minimal.
> Whilst theoretically there would be a performance issue in practice you
> won't notice it and it is much more stable. I have heard some reports of
> people seeing better overall performance. The usual YMMV warning applies.
> The only way to know for sure is to test it.
>
>> and I'm unclear as to why it's necessary. Can anyone educate me on
>> this?
>
> Various causes, usually comes down to one end dropping the connection
> without the other one realising. HTTP keep-alive usually plays a part in
> this. Can also be related to connection time outs and when each end checks
> to see if the time out is expired.
>
> In short, disabling re-use removes all of these issues.
>
>> Do I still have a problem with my configuration?
>
> You could probably tune the time out and keep alive settings to reduce the
> issue but just disabling reuse is usually the better option.
Easier vs. better: If you setup sensible timeouts and use CPing/CPong as
described on
http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
There should be no such problems, even without DisableReuse.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk -> ajp hangs indefinitely on some requests
Posted by Mark Thomas <ma...@apache.org>.
Roy McMorran wrote:
> Roy McMorran wrote:
>> Despite my misgivings, 1.1.5 seems to have helped. It's run my test
>> script for about 30 minutes now without problems (previously would
>> have hung by minute 3).
>> However!
>> My dev system (which was already running 1.1.15) still exhibits the
>> problem. It also has newer versions of Tomcat (6.0.18), Apache
>> (2.2.10) and mod_jk (1.2.27).
>>
> Correction. The combination of updating libtcnative to 1.1.15 _AND_
> setting "JkOptions +DisableReuse" prevents both systems from hanging.
>
> I'm concerned about the performance implications of using this directive
> based on what I read at
> http://tomcat.apache.org/connectors-doc/reference/apache.html
Don't be. With httpd and Tomcat on the same box (or even the same LAN) the
overhead of establishing the TCP connection on every request is minimal.
Whilst theoretically there would be a performance issue in practice you
won't notice it and it is much more stable. I have heard some reports of
people seeing better overall performance. The usual YMMV warning applies.
The only way to know for sure is to test it.
> and I'm unclear as to why it's necessary. Can anyone educate me on
> this?
Various causes, usually comes down to one end dropping the connection
without the other one realising. HTTP keep-alive usually plays a part in
this. Can also be related to connection time outs and when each end checks
to see if the time out is expired.
In short, disabling re-use removes all of these issues.
> Do I still have a problem with my configuration?
You could probably tune the time out and keep alive settings to reduce the
issue but just disabling reuse is usually the better option.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk -> ajp hangs indefinitely on some requests
Posted by Roy McMorran <mc...@mdibl.org>.
Roy McMorran wrote:
> Despite my misgivings, 1.1.5 seems to have helped. It's run my test
> script for about 30 minutes now without problems (previously would
> have hung by minute 3).
> However!
> My dev system (which was already running 1.1.15) still exhibits the
> problem. It also has newer versions of Tomcat (6.0.18), Apache
> (2.2.10) and mod_jk (1.2.27).
>
Correction. The combination of updating libtcnative to 1.1.15 _AND_
setting "JkOptions +DisableReuse" prevents both systems from hanging.
I'm concerned about the performance implications of using this directive
based on what I read at
http://tomcat.apache.org/connectors-doc/reference/apache.html
and I'm unclear as to why it's necessary. Can anyone educate me on
this? Do I still have a problem with my configuration?
Thanks again,
-r
--
Roy McMorran
Systems Administrator
MDI Biological Laboratory
mcmorran@mdibl.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk -> ajp hangs indefinitely on some requests
Posted by Roy McMorran <mc...@mdibl.org>.
Roy McMorran wrote:
> Mark Thomas wrote:
>> Roy McMorran wrote:
>>>>> libtcnative: 1.10
>>
>> Sorry, should have seen this sooner. Try disabling it or using a
>> newer version.
>
> I'll give it a try but I'm not hopeful. You see I have two systems
> exhibiting this hang behavior. The other one (dev) has the latest
> libtcnative (1.1.15). And I have a third (the production box) also
> using 1.1.10 that is fine (it doesn't hang). I admit, it's strange.
Despite my misgivings, 1.1.5 seems to have helped. It's run my test
script for about 30 minutes now without problems (previously would have
hung by minute 3).
However!
My dev system (which was already running 1.1.15) still exhibits the
problem. It also has newer versions of Tomcat (6.0.18), Apache (2.2.10)
and mod_jk (1.2.27).
Looks like I'm not done yet.
Thanks,
-r
--
Roy McMorran
Systems Administrator
MDI Biological Laboratory
mcmorran@mdibl.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk -> ajp hangs indefinitely on some requests
Posted by André Warnier <aw...@ice-sa.com>.
Doctor Khumalo wrote:
> How can i be removed from this awful blog?
>
Just scroll down the page, Doctor, there's a link to do just that on
every one of them.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: mod_jk -> ajp hangs indefinitely on some requests
Posted by Doctor Khumalo <do...@hotmail.com>.
How can i be removed from this awful blog?
> Date: Thu, 4 Dec 2008 15:50:30 -0500
> From: mcmorran@mdibl.org
> To: users@tomcat.apache.org
> Subject: Re: mod_jk -> ajp hangs indefinitely on some requests
>
> Mark Thomas wrote:
> > Roy McMorran wrote:
> >
> >>>> libtcnative: 1.10
> >>>>
> >
> > Sorry, should have seen this sooner. Try disabling it or using a newer version.
> >
>
> I'll give it a try but I'm not hopeful. You see I have two systems
> exhibiting this hang behavior. The other one (dev) has the latest
> libtcnative (1.1.15). And I have a third (the production box) also
> using 1.1.10 that is fine (it doesn't hang). I admit, it's strange.
>
> Thanks
> -r
>
> --
>
> Roy McMorran
> Systems Administrator
> MDI Biological Laboratory
> mcmorran@mdibl.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
_________________________________________________________________
Send e-mail anywhere. No map, no compass.
http://windowslive.com/Explore/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008
Re: mod_jk -> ajp hangs indefinitely on some requests
Posted by Roy McMorran <mc...@mdibl.org>.
Mark Thomas wrote:
> Roy McMorran wrote:
>
>>>> libtcnative: 1.10
>>>>
>
> Sorry, should have seen this sooner. Try disabling it or using a newer version.
>
I'll give it a try but I'm not hopeful. You see I have two systems
exhibiting this hang behavior. The other one (dev) has the latest
libtcnative (1.1.15). And I have a third (the production box) also
using 1.1.10 that is fine (it doesn't hang). I admit, it's strange.
Thanks
-r
--
Roy McMorran
Systems Administrator
MDI Biological Laboratory
mcmorran@mdibl.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk -> ajp hangs indefinitely on some requests
Posted by Mark Thomas <ma...@apache.org>.
Roy McMorran wrote:
>>> libtcnative: 1.10
Sorry, should have seen this sooner. Try disabling it or using a newer version.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk -> ajp hangs indefinitely on some requests
Posted by Roy McMorran <mc...@mdibl.org>.
Mark Thomas wrote:
> Roy McMorran wrote:
>
>> Hello all,
>>
>> I'm seeing frequent cases where requests sent via mod_jk to Tomcat will
>> hang indefinitely. The request will never return to Apache and the
>> client will eventually time out. There are no error messages.
>>
>> In my test case I run wget against various URLs until the hang occurs
>> (usually within 2-3 minutes). Here are the last few entries from my
>> mod_jk.log file (JkLogLevel = trace):
>>
>> [Thu Dec 04 13:28:08.509 2008] [27224:1] [debug]
>> ajp_connection_tcp_send_message::jk_ajp_common.c (911): 00c0 FF 00 00
>> 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
>> [Thu Dec 04 13:28:08.510 2008] [27224:1] [trace]
>> ajp_connection_tcp_send_message::jk_ajp_common.c (928): exit
>> [Thu Dec 04 13:28:08.510 2008] [27224:1] [debug]
>> ajp_send_request::jk_ajp_common.c (1325): (ajp13) request body to send 0
>> - request body to resend 0
>> [Thu Dec 04 13:28:08.511 2008] [27224:1] [trace]
>> ajp_send_request::jk_ajp_common.c (1422): exit
>> [Thu Dec 04 13:28:08.511 2008] [27224:1] [trace]
>> ajp_get_reply::jk_ajp_common.c (1613): enter
>> [Thu Dec 04 13:28:08.511 2008] [27224:1] [trace]
>> ajp_connection_tcp_get_message::jk_ajp_common.c (953): enter
>>
>> Beyond this point there is no activity. There are never any error
>> messages generated, not in mod_jk.log, catalina.out, nor in the Apache
>> logs.
>>
>> There is a thread dump of Tomcat (after the hang begins) at:
>> http://www.mdibl.org/~mcmorran/tomcat_thread_dump.txt
>>
>> This is a pretty simple setup, both Apache & Tomcat are running on the
>> same host, so there is no firewall issue. No load balancing, just one
>> worker defined.
>>
>> workers.properties file:
>> worker.list=ajp13
>> worker.ajp13.type=ajp13
>> worker.ajp13.host=localhost
>> worker.ajp13.port=8009
>> worker.ajp13.socket_timeout=120
>>
>> (have tried with and without the socket_timeout parameter - makes no
>> difference)
>>
>> My server.xml file is at:
>> http://www.mdibl.org/~mcmorran/server.xml.txt
>>
>> (I've tried various settings (or none) for connectionTimeout inside the
>> AJP connector - makes no difference)
>>
>> mod_jk related parts of my httpd.conf file are at:
>> http://www.mdibl.org/~mcmorran/httpd.conf.txt
>>
>> Version info:
>> Server: Solaris 10
>> Apache: 2.2.6
>> mod_jk: 1.2.25
>> Tomcat: 6.0.14
>> libtcnative: 1.10
>> Java: 1.5.0_14
>>
>> I'm running Tomcat under jsvc. The JVM gets 2G of memory (there are 8G
>> total on the system) - I don't think there's any memory issue. I'm in
>> test mode so I'm the only user; there's virtually no CPU load on the box.
>>
>> I've reached a dead end here -I'm hoping someone on the list may have
>> some ideas. Thanks for any help you may be able to provide.
>>
>
> Have you tried:
> JkOptions +DisableReuse
> ?
>
> Mark
>
>
Hi Mark, thanks for the reply.
I hadn't tried DisableReuse - the documentation at
http://tomcat.apache.org/connectors-doc/reference/apache.html led me to
believe that 1) it would likely be very detrimental to performance and
2) it probably doesn't apply my situation (no firewall, no network
involved at all).
Nevertheless, I gave it a go just now. Unfortunately it made no difference.
Thanks,
-r
--
Roy McMorran
Systems Administrator
MDI Biological Laboratory
mcmorran@mdibl.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: mod_jk -> ajp hangs indefinitely on some requests
Posted by Mark Thomas <ma...@apache.org>.
Roy McMorran wrote:
> Hello all,
>
> I'm seeing frequent cases where requests sent via mod_jk to Tomcat will
> hang indefinitely. The request will never return to Apache and the
> client will eventually time out. There are no error messages.
>
> In my test case I run wget against various URLs until the hang occurs
> (usually within 2-3 minutes). Here are the last few entries from my
> mod_jk.log file (JkLogLevel = trace):
>
> [Thu Dec 04 13:28:08.509 2008] [27224:1] [debug]
> ajp_connection_tcp_send_message::jk_ajp_common.c (911): 00c0 FF 00 00
> 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
> [Thu Dec 04 13:28:08.510 2008] [27224:1] [trace]
> ajp_connection_tcp_send_message::jk_ajp_common.c (928): exit
> [Thu Dec 04 13:28:08.510 2008] [27224:1] [debug]
> ajp_send_request::jk_ajp_common.c (1325): (ajp13) request body to send 0
> - request body to resend 0
> [Thu Dec 04 13:28:08.511 2008] [27224:1] [trace]
> ajp_send_request::jk_ajp_common.c (1422): exit
> [Thu Dec 04 13:28:08.511 2008] [27224:1] [trace]
> ajp_get_reply::jk_ajp_common.c (1613): enter
> [Thu Dec 04 13:28:08.511 2008] [27224:1] [trace]
> ajp_connection_tcp_get_message::jk_ajp_common.c (953): enter
>
> Beyond this point there is no activity. There are never any error
> messages generated, not in mod_jk.log, catalina.out, nor in the Apache
> logs.
>
> There is a thread dump of Tomcat (after the hang begins) at:
> http://www.mdibl.org/~mcmorran/tomcat_thread_dump.txt
>
> This is a pretty simple setup, both Apache & Tomcat are running on the
> same host, so there is no firewall issue. No load balancing, just one
> worker defined.
>
> workers.properties file:
> worker.list=ajp13
> worker.ajp13.type=ajp13
> worker.ajp13.host=localhost
> worker.ajp13.port=8009
> worker.ajp13.socket_timeout=120
>
> (have tried with and without the socket_timeout parameter - makes no
> difference)
>
> My server.xml file is at:
> http://www.mdibl.org/~mcmorran/server.xml.txt
>
> (I've tried various settings (or none) for connectionTimeout inside the
> AJP connector - makes no difference)
>
> mod_jk related parts of my httpd.conf file are at:
> http://www.mdibl.org/~mcmorran/httpd.conf.txt
>
> Version info:
> Server: Solaris 10
> Apache: 2.2.6
> mod_jk: 1.2.25
> Tomcat: 6.0.14
> libtcnative: 1.10
> Java: 1.5.0_14
>
> I'm running Tomcat under jsvc. The JVM gets 2G of memory (there are 8G
> total on the system) - I don't think there's any memory issue. I'm in
> test mode so I'm the only user; there's virtually no CPU load on the box.
>
> I've reached a dead end here -I'm hoping someone on the list may have
> some ideas. Thanks for any help you may be able to provide.
Have you tried:
JkOptions +DisableReuse
?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org