You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jorge Medina <jm...@e-dialog.com> on 2009/02/02 20:01:46 UTC
Problem with mod_jk and Tomcat Native Connectors on Solaris
Hi
I am having problems with the mod_jk module 1.2.26 and Tomcat Native
connectors running in Solaris 10.
The problem occurs in both processors x86 (64-bit) and sparc (64-bit).
(The problem does not occur on RedHat EL5 64-bit).
On the mod_jk workers.properties file I am specifying a value for
reply_timeout:
worker.localworker.reply_timeout=10000 (10 seconds)
I am anticipating that some requests would need more time, so I
decided to run my test suite just increasing the value to 30000.
Oddly, with this value and running the same set of tests I get
sporadic errors. The mod_jk log shows the following message when this
occurs:
[error] ajp_get_reply::jk_ajp_common.c (1730) : (localworker) Timeout
with waiting reply from tomcat. Tomcat is down, stopped or nertwork
problems (errno=145)
I am using the following components:
Apache Web Server 2.2.11
APR 1.3.3
APR-util 1.3.4
openssl-0.9.8
mod_jk 1.2.26
Tomcat 6.0.16
Tomcat Native Connectors 1.1.12 (using same APR version
as Apache)
jsvc
Sun Java JDK 1.6
I also found that the problem only occurs if the Tomcat Native
Connectors is enabled in Tomcat.
I can't figure why increasing the reply timeout would actually
produce the errors it is supposed to prevent!!
Any help is appreciated. I would like to benefit from the Tomcat
Native Connectors, but I can't figure what I am configuring wrong.
-----------------------------------------------
Below is my workers.properties ...
-----------------------------------------------
worker.list= wlb, jkwatch, jkmanage
# Properties for worker: localworker
worker.localworker.type=ajp13
worker.localworker.host=localhost
worker.localworker.port=8009
worker.localworker.lbfactor=1
worker.localworker.connection_pool_timeout=600
worker.localworker.socket_keepalive=True
worker.localworker.socket_timeout=60
################### this parameter causes trouble if increased to 30000
worker.localworker.reply_timeout=10000
#################################
# Defining a load balancer (with a single worker, the local worker)
worker.wlb.type=lb
worker.wlb.balance_workers=localworker
worker.wlb.max_reply_timeouts=3
-----------------------------------------------
... and here is the segment of server.xml
-----------------------------------------------
<Connector port="8009"
protocol="AJP/1.3"
connectionTimeout="600000"
enableLookups="false"
bufferSize="32768"
maxThreads="3000"/>
- Jorge
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with mod_jk and Tomcat Native Connectors on Solaris
Posted by Scott Severtson <sc...@severtson.us>.
Another similar but not identical case:
Solaris 10 on x86 (occurs on both 32/64 bit hardware)
Tomcat 6.0.18
Tomcat Native 1.1.16
APR 1.3.3
Apache 2.2.11
Using mod_proxy_ajp and AjpAprProtocol
All tools compiled with GCC 3.4.3 (default shipped with Solaris 10).
mod_proxy_ajp times out waiting for tomcat to respond with:
The timeout specified has expired: ajp_ilink_receive() can't receive header
This occurs intermittently while loading a JSP consisting of:
<%=System.currentTimeMillis()%>
I've proxied the AJP port, and can see the request being issued from
mod_proxy_ajp, but no response is generated. I've added an AccessLogValve,
but the requests that timeout are not logged.
Switching from AjpAprProtocol to AjpProtocol works around the problem, but
we really could benefit from send_file support.
jordiang wrote:
>
> In Solaris, I think the Tomcat Native library just loses some requests.
>
I concur; likely polling is somehow broken on Solaris. I think we can rule
out mod_jk/mod_proxy_ajp, as both are experiencing similar issues.
I am available to assist anyone with investigation/debugging - please let me
know how I can help.
Thanks,
--Scott Severtson
--
View this message in context: http://www.nabble.com/Problem-with-mod_jk-and-Tomcat-Native-Connectors-on-Solaris-tp21796027p21937252.html
Sent from the Tomcat - User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Problem with mod_jk and Tomcat Native Connectors on Solaris
Posted by Jorge Medina <jm...@e-dialog.com>.
Hallo Rainer!
Danke für dein Hilfe.
I will follow your recommendation on upgrading, and I will need to learn how to configure a more verbose log and interpret them.
I found out the following:
- My client code (a java Axis2 client) had a timeout (I found it was 30 sec, I was under the assumption the default was 60s). When the client timeoust I would get a message "Read timed out"
- The above timeout was too short, the client would timeout before Apache (mod_jk) had a chance to retry the request to Tomcat.
I played with both timeouts, making sure the client timeout was always larger than the mod_jk reply_timeout for at least a factor of 2.
This works pefectly in RedHat, but not in Solaris 10 (I tried sparc and x86, both compiled in 64-bit)
In Solaris, I think the Tomcat Native library just loses some requests. No matter how I configured the timeouts and retries, I always get some sporadic "504 Gateway timeout". Disabling the Tomcat Native library and keeping everything in mod_jk the same works fine. So, for now I just disabled the Tomcat Native Library.
Thanks again for your response.
-Jorge
-----Original Message-----
From: Rainer Jung [mailto:rainer.jung@kippdata.de]
Sent: Friday, February 06, 2009 11:38 AM
To: Tomcat Users List
Subject: Re: Problem with mod_jk and Tomcat Native Connectors on Solaris
On 02.02.2009 20:01, Jorge Medina wrote:
> I am having problems with the mod_jk module 1.2.26 and Tomcat
> Native connectors running in Solaris 10.
> The problem occurs in both processors x86 (64-bit) and sparc (64-bit).
> (The problem does not occur on RedHat EL5 64-bit).
>
> On the mod_jk workers.properties file I am specifying a value for
> reply_timeout:
>
> worker.localworker.reply_timeout=10000 (10 seconds)
>
> I am anticipating that some requests would need more time, so I
> decided to run my test suite just increasing the value to 30000.
> Oddly, with this value and running the same set of tests I get
> sporadic errors. The mod_jk log shows the following message when this
> occurs:
>
> [error] ajp_get_reply::jk_ajp_common.c (1730) : (localworker)
> Timeout with waiting reply from tomcat. Tomcat is down, stopped or
> nertwork problems (errno=145)
>
>
> I am using the following components:
> Apache Web Server 2.2.11
> APR 1.3.3
> APR-util 1.3.4
> openssl-0.9.8
> mod_jk 1.2.26
Before debugging a problem you might consider switching to 1.2.27, because once everything works, you'll most likely not do the upgrade ;)
> Tomcat 6.0.16
> Tomcat Native Connectors 1.1.12 (using same APR
> version as Apache)
Please update to 1.1.16. There have been a couple of fixes in tcnative.
> jsvc
> Sun Java JDK 1.6
>
> I also found that the problem only occurs if the Tomcat Native
> Connectors is enabled in Tomcat.
>
> I can't figure why increasing the reply timeout would actually
> produce the errors it is supposed to prevent!!
I don't really understand: why do you think the timeout produces the problem? Setting the timeout means, that any reply taking longer than the timeout will be interrupted inside Apache and logged. So setting no timeout will not produce any log statements (by default the timeout is off), but I expect that you will still have long running requests.
To find out about the response times and the requests with long response times, add "%D" to your Apache LogFormat (response times in
microseconds) and activate an access log in tomcat too, with a pattern that also contains "%D" 8response times in milliseconds). That way you can control during your tests, what tomcat resp. Apache think the response time actually was.
After verifying that Tomcat actually needs to long for some requests, you need to start to analse why (locking in application, waiting for backend or database, etc.). You can use Java thread dumps for this (kill -QUIT).
> Any help is appreciated. I would like to benefit from the Tomcat
> Native Connectors, but I can't figure what I am configuring wrong.
Try 1.1.16.
> -----------------------------------------------
> Below is my workers.properties ...
> -----------------------------------------------
>
> worker.list= wlb, jkwatch, jkmanage
> # Properties for worker: localworker
> worker.localworker.type=ajp13
> worker.localworker.host=localhost
> worker.localworker.port=8009
> worker.localworker.lbfactor=1
> worker.localworker.connection_pool_timeout=600
> worker.localworker.socket_keepalive=True
> worker.localworker.socket_timeout=60
> ################### this parameter causes trouble if increased to
> 30000 worker.localworker.reply_timeout=10000
> #################################
>
> # Defining a load balancer (with a single worker, the local worker)
> worker.wlb.type=lb worker.wlb.balance_workers=localworker
> worker.wlb.max_reply_timeouts=3
I would add cping/cpong. See
http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
> -----------------------------------------------
> ... and here is the segment of server.xml
> -----------------------------------------------
>
> <Connector port="8009"
> protocol="AJP/1.3"
> connectionTimeout="600000"
> enableLookups="false"
> bufferSize="32768"
> maxThreads="3000"/>
>
>
> - Jorge
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Problem with mod_jk and Tomcat Native Connectors on Solaris
Posted by Rainer Jung <ra...@kippdata.de>.
On 02.02.2009 20:01, Jorge Medina wrote:
> I am having problems with the mod_jk module 1.2.26 and Tomcat Native
> connectors running in Solaris 10.
> The problem occurs in both processors x86 (64-bit) and sparc (64-bit).
> (The problem does not occur on RedHat EL5 64-bit).
>
> On the mod_jk workers.properties file I am specifying a value for
> reply_timeout:
>
> worker.localworker.reply_timeout=10000 (10 seconds)
>
> I am anticipating that some requests would need more time, so I
> decided to run my test suite just increasing the value to 30000.
> Oddly, with this value and running the same set of tests I get
> sporadic errors. The mod_jk log shows the following message when this
> occurs:
>
> [error] ajp_get_reply::jk_ajp_common.c (1730) : (localworker) Timeout
> with waiting reply from tomcat. Tomcat is down, stopped or nertwork
> problems (errno=145)
>
>
> I am using the following components:
> Apache Web Server 2.2.11
> APR 1.3.3
> APR-util 1.3.4
> openssl-0.9.8
> mod_jk 1.2.26
Before debugging a problem you might consider switching to 1.2.27,
because once everything works, you'll most likely not do the upgrade ;)
> Tomcat 6.0.16
> Tomcat Native Connectors 1.1.12 (using same APR version
> as Apache)
Please update to 1.1.16. There have been a couple of fixes in tcnative.
> jsvc
> Sun Java JDK 1.6
>
> I also found that the problem only occurs if the Tomcat Native
> Connectors is enabled in Tomcat.
>
> I can't figure why increasing the reply timeout would actually
> produce the errors it is supposed to prevent!!
I don't really understand: why do you think the timeout produces the
problem? Setting the timeout means, that any reply taking longer than
the timeout will be interrupted inside Apache and logged. So setting no
timeout will not produce any log statements (by default the timeout is
off), but I expect that you will still have long running requests.
To find out about the response times and the requests with long response
times, add "%D" to your Apache LogFormat (response times in
microseconds) and activate an access log in tomcat too, with a pattern
that also contains "%D" 8response times in milliseconds). That way you
can control during your tests, what tomcat resp. Apache think the
response time actually was.
After verifying that Tomcat actually needs to long for some requests,
you need to start to analse why (locking in application, waiting for
backend or database, etc.). You can use Java thread dumps for this (kill
-QUIT).
> Any help is appreciated. I would like to benefit from the Tomcat
> Native Connectors, but I can't figure what I am configuring wrong.
Try 1.1.16.
> -----------------------------------------------
> Below is my workers.properties ...
> -----------------------------------------------
>
> worker.list= wlb, jkwatch, jkmanage
> # Properties for worker: localworker
> worker.localworker.type=ajp13
> worker.localworker.host=localhost
> worker.localworker.port=8009
> worker.localworker.lbfactor=1
> worker.localworker.connection_pool_timeout=600
> worker.localworker.socket_keepalive=True
> worker.localworker.socket_timeout=60
> ################### this parameter causes trouble if increased to 30000
> worker.localworker.reply_timeout=10000
> #################################
>
> # Defining a load balancer (with a single worker, the local worker)
> worker.wlb.type=lb
> worker.wlb.balance_workers=localworker
> worker.wlb.max_reply_timeouts=3
I would add cping/cpong. See
http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
> -----------------------------------------------
> ... and here is the segment of server.xml
> -----------------------------------------------
>
> <Connector port="8009"
> protocol="AJP/1.3"
> connectionTimeout="600000"
> enableLookups="false"
> bufferSize="32768"
> maxThreads="3000"/>
>
>
> - Jorge
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org