You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Kari Scott <Ka...@cdw.com> on 2011/12/06 21:19:23 UTC
AJP connection timeout setting/Tomcat 6 vs. 7 questions
We are running Tomcat 6. 0.32 with jdk1.6.0_26 on Solaris 10, mod_ajp 1.3 and Apache 2.2.21 on all but one production server which is the same except for it's running Tomcat 7.0.21.
I have some questions regarding connection timeout settings. Occasionally, when the site is busier we see jumps in the number of connections to 8009 and then that number stays high for about 30 minutes before settling back down into our average range. A thread dump shows that these connections correspond to these socket threads:
"TP-Processor222" daemon prio=3 tid=0x00c76400 nid=0x5669 runnable [0x8cf7f000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0xcb2a0eb0> (a java.io.BufferedInputStream)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
at java.lang.Thread.run(Thread.java:662)
The problem isn't so much that they stick around, but when these first start increasing, there is a noticeable hit in performance and evidence that threads are waiting for resources. Oddly, the one trial Tomcat 7 server with the same connector, load and code never experiences this problem. We currently don't have a connectionTimeout specified for our connector so my plan is to try the following:
<Connector port="8009" protocol="AJP/1.3" connectionTimeout="20000" redirectPort="8443" />
Here are my questions:
*Do I also need to set the connection_pool_timeout in the worker? Or is that the one I should be changing instead of connectionTimeout?
*Is there a different time out setting I should be looking at?
*Is there an easy explanation as to why Tomcat 7 never experiences this issue? I'm just wondering (o.k. hoping) that there is some magic Tomcat 7 default setting some place that we can add to our Tomcat 6 environments that can help us out until we've upgraded everything.
Thank you,
Kari
_________________________________
Kari Scott
Senior Programmer
kari.scott@cdw.com<ma...@cdw.com>
CDW
5520 Research Park Drive
Madison, WI 53711
Office: 608 298 1223
Fax: 608 288 3007
Re: AJP connection timeout setting/Tomcat 6 vs. 7 questions
Posted by André Warnier <aw...@ice-sa.com>.
Kari Scott wrote:
>
>
>
> On Dec 6, 2011, at 2:25 PM, André Warnier wrote:
>
>> Kari Scott wrote:
>>> We are running Tomcat 6. 0.32 with jdk1.6.0_26 on Solaris 10, mod_ajp 1.3 and Apache 2.2.21 on all but one production server which is the same except for it's running Tomcat 7.0.21.
>>> I have some questions regarding connection timeout settings. Occasionally, when the site is busier we see jumps in the number of connections to 8009 and then that number stays high for about 30 minutes before settling back down into our average range. A thread dump shows that these connections correspond to these socket threads:
>>> "TP-Processor222" daemon prio=3 tid=0x00c76400 nid=0x5669 runnable [0x8cf7f000]
>>> java.lang.Thread.State: RUNNABLE
>>> at java.net.SocketInputStream.socketRead0(Native Method)
>>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>>> - locked <0xcb2a0eb0> (a java.io.BufferedInputStream)
>>> at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
>>> at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
>>> at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
>>> at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
>>> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
>>> at java.lang.Thread.run(Thread.java:662)
>>> The problem isn't so much that they stick around, but when these first start increasing, there is a noticeable hit in performance and evidence that threads are waiting for resources. Oddly, the one trial Tomcat 7 server with the same connector, load and code never experiences this problem. We currently don't have a connectionTimeout specified for our connector so my plan is to try the following:
>>> <Connector port="8009" protocol="AJP/1.3" connectionTimeout="20000" redirectPort="8443" />
>>> Here are my questions:
>>> *Do I also need to set the connection_pool_timeout in the worker? Or is that the one I should be changing instead of connectionTimeout?
>>> *Is there a different time out setting I should be looking at?
>>> *Is there an easy explanation as to why Tomcat 7 never experiences this issue? I'm just wondering (o.k. hoping) that there is some magic Tomcat 7 default setting some place that we can add to our Tomcat 6 environments that can help us out until we've upgraded everything.
>> Just a question, to add to your excellent summary above : in your front-end server configuration, what are the settings related to keep-alive ?
>>
>
>
> All the servers have the following Apache settings:
>
> KeepAlive On
> MaxKeepAliveRequests 200
> KeepAliveTimeout 15
>
So, what happens if you set
KeepAliveTimeout 3
?
>
>
>> And maybe, can you provide an example of the server.xml (comments and sensitive info removed) for both a server which experiences the issue, and for the 7.0 server which doesn't ? (paste them inside the message, the list strips most attachments).
>>
>
>
> I sure can. I also removed some of the entries that were exactly the same so it's easier to see the differences:
>
> *********************************************************************************
> Tomcat 7 server.xml:
>
> <Server port="8005" shutdown="SHUTDOWN">
> <Service name="Catalina">
> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
> <Engine name="Catalina" defaultHost="localhost">
>
> <Host name="localhost" appBase="webapps"
> unpackWARs="false" autoDeploy="false">
>
> <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
> prefix="localhost_access_log." suffix=".txt"
> pattern="%h %l %u %t "%r" %s %b" resolveHosts="false"/>
>
> </Host>
> </Engine>
> </Service>
> </Server>
>
>
> Tomcat 6 server.xml:
>
> <Server port="8005" shutdown="SHUTDOWN">
> <Service name="Catalina">
> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
> <Engine name="Catalina" defaultHost="localhost">
>
> <Valve className="com.jamonapi.http.JAMonTomcatValve"/>
>
> <Host name="localhost" appBase="webapps"
> unpackWARs="false" autoDeploy="false"
> xmlValidation="false" xmlNamespaceAware="false">
> </Host>
> </Engine>
> </Service>
> </Server>
>
> *********************************************************************************
>
> So the big difference is the presence of the JaMON Valve we're using on Tomcat 6 and but accidentally forgot to put on Tomcat 7. Maybe this was a fortuitous mistake. I'll try removing it from one of our Tomcat 6 servers to see if that's the culprit. We don't need that access logging valve enabled on Tomcat 7 either, so this was a really good exercise to go through. Thanks!
>
>
>
> -kari
>
>
>
>
> _________________________________
> Kari Scott
> Senior Programmer
> kari.scott@cdw.com
>
> CDW
> 5520 Research Park Drive
> Madison, WI 53711
> Office: 608 298 1223
> Fax: 608 288 3007
>
>
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP connection timeout setting/Tomcat 6 vs. 7 questions
Posted by Kari Scott <Ka...@cdw.com>.
On Dec 6, 2011, at 2:25 PM, André Warnier wrote:
> Kari Scott wrote:
>> We are running Tomcat 6. 0.32 with jdk1.6.0_26 on Solaris 10, mod_ajp 1.3 and Apache 2.2.21 on all but one production server which is the same except for it's running Tomcat 7.0.21.
>> I have some questions regarding connection timeout settings. Occasionally, when the site is busier we see jumps in the number of connections to 8009 and then that number stays high for about 30 minutes before settling back down into our average range. A thread dump shows that these connections correspond to these socket threads:
>> "TP-Processor222" daemon prio=3 tid=0x00c76400 nid=0x5669 runnable [0x8cf7f000]
>> java.lang.Thread.State: RUNNABLE
>> at java.net.SocketInputStream.socketRead0(Native Method)
>> at java.net.SocketInputStream.read(SocketInputStream.java:129)
>> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
>> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
>> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
>> - locked <0xcb2a0eb0> (a java.io.BufferedInputStream)
>> at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
>> at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
>> at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
>> at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
>> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
>> at java.lang.Thread.run(Thread.java:662)
>> The problem isn't so much that they stick around, but when these first start increasing, there is a noticeable hit in performance and evidence that threads are waiting for resources. Oddly, the one trial Tomcat 7 server with the same connector, load and code never experiences this problem. We currently don't have a connectionTimeout specified for our connector so my plan is to try the following:
>> <Connector port="8009" protocol="AJP/1.3" connectionTimeout="20000" redirectPort="8443" />
>> Here are my questions:
>> *Do I also need to set the connection_pool_timeout in the worker? Or is that the one I should be changing instead of connectionTimeout?
>> *Is there a different time out setting I should be looking at?
>> *Is there an easy explanation as to why Tomcat 7 never experiences this issue? I'm just wondering (o.k. hoping) that there is some magic Tomcat 7 default setting some place that we can add to our Tomcat 6 environments that can help us out until we've upgraded everything.
> Just a question, to add to your excellent summary above : in your front-end server configuration, what are the settings related to keep-alive ?
>
All the servers have the following Apache settings:
KeepAlive On
MaxKeepAliveRequests 200
KeepAliveTimeout 15
> And maybe, can you provide an example of the server.xml (comments and sensitive info removed) for both a server which experiences the issue, and for the 7.0 server which doesn't ? (paste them inside the message, the list strips most attachments).
>
I sure can. I also removed some of the entries that were exactly the same so it's easier to see the differences:
*********************************************************************************
Tomcat 7 server.xml:
<Server port="8005" shutdown="SHUTDOWN">
<Service name="Catalina">
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost">
<Host name="localhost" appBase="webapps"
unpackWARs="false" autoDeploy="false">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="localhost_access_log." suffix=".txt"
pattern="%h %l %u %t "%r" %s %b" resolveHosts="false"/>
</Host>
</Engine>
</Service>
</Server>
Tomcat 6 server.xml:
<Server port="8005" shutdown="SHUTDOWN">
<Service name="Catalina">
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
<Engine name="Catalina" defaultHost="localhost">
<Valve className="com.jamonapi.http.JAMonTomcatValve"/>
<Host name="localhost" appBase="webapps"
unpackWARs="false" autoDeploy="false"
xmlValidation="false" xmlNamespaceAware="false">
</Host>
</Engine>
</Service>
</Server>
*********************************************************************************
So the big difference is the presence of the JaMON Valve we're using on Tomcat 6 and but accidentally forgot to put on Tomcat 7. Maybe this was a fortuitous mistake. I'll try removing it from one of our Tomcat 6 servers to see if that's the culprit. We don't need that access logging valve enabled on Tomcat 7 either, so this was a really good exercise to go through. Thanks!
-kari
_________________________________
Kari Scott
Senior Programmer
kari.scott@cdw.com
CDW
5520 Research Park Drive
Madison, WI 53711
Office: 608 298 1223
Fax: 608 288 3007
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AJP connection timeout setting/Tomcat 6 vs. 7 questions
Posted by André Warnier <aw...@ice-sa.com>.
Kari Scott wrote:
>
> We are running Tomcat 6. 0.32 with jdk1.6.0_26 on Solaris 10, mod_ajp 1.3 and Apache 2.2.21 on all but one production server which is the same except for it's running Tomcat 7.0.21.
>
> I have some questions regarding connection timeout settings. Occasionally, when the site is busier we see jumps in the number of connections to 8009 and then that number stays high for about 30 minutes before settling back down into our average range. A thread dump shows that these connections correspond to these socket threads:
>
>
> "TP-Processor222" daemon prio=3 tid=0x00c76400 nid=0x5669 runnable [0x8cf7f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
> - locked <0xcb2a0eb0> (a java.io.BufferedInputStream)
> at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
> at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
> at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
> at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
> at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
> at java.lang.Thread.run(Thread.java:662)
>
>
>
> The problem isn't so much that they stick around, but when these first start increasing, there is a noticeable hit in performance and evidence that threads are waiting for resources. Oddly, the one trial Tomcat 7 server with the same connector, load and code never experiences this problem. We currently don't have a connectionTimeout specified for our connector so my plan is to try the following:
>
> <Connector port="8009" protocol="AJP/1.3" connectionTimeout="20000" redirectPort="8443" />
>
>
> Here are my questions:
>
> *Do I also need to set the connection_pool_timeout in the worker? Or is that the one I should be changing instead of connectionTimeout?
>
> *Is there a different time out setting I should be looking at?
>
> *Is there an easy explanation as to why Tomcat 7 never experiences this issue? I'm just wondering (o.k. hoping) that there is some magic Tomcat 7 default setting some place that we can add to our Tomcat 6 environments that can help us out until we've upgraded everything.
>
>
Just a question, to add to your excellent summary above : in your front-end server
configuration, what are the settings related to keep-alive ?
And maybe, can you provide an example of the server.xml (comments and sensitive info
removed) for both a server which experiences the issue, and for the 7.0 server which
doesn't ? (paste them inside the message, the list strips most attachments).
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org