You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Andrew Lee Rubinger <al...@9mmedia.com> on 2007/07/26 20:54:13 UTC
Tomcat/AJP Client Abort Socket Errors - Leads to File Descriptor
Leak
Hi.
I am running Apache HTTPD 2.2.4 with ModJK 1.2.21, both compiled from
source on CentOS 4.4. I have a workers.properties file configured to
send requests to Tomcat 5.5 on a separate machine within the internal
network.
When receiving a ClientAbortException from Tomcat (shown in
"tomcat_broken_pipe_error.txt"), it seems that the socket connection
between ModJK and Tomcat is not closed out, and on the Application
Server running Tomcat I'm observing a Life Descriptor leak (as shown in
"lsof.txt").
I've tried setting "recovery_options" to 4 as suggested by
http://tomcat.apache.org/connectors-doc/reference/workers.html, but this
has not corrected the problem.
Have attached all relevant log excerpts and configurations in hopes that
someone has come across this and might have some insight.
Thanks in advance for your help. Will be happy to provide any other
information I may have excluded.
S,
ALR
Re: Tomcat/AJP Client Abort Socket Errors - Leads to File Descriptor
Leak
Posted by Andrew Lee Rubinger <al...@9mmedia.com>.
Thank you so much for all your help. Will post back if a conclusion is
determined, or if it's ruled out that AJP/Tomcat is not the cause.
S,
ALR
Rainer Jung wrote:
> Hi Andrew,
>
> this snapshot looks normal. There are 11 connections used and both sides
> have the same idea about their state.
>
> Web server to Tomcat connections are managed by mod_jk via pools. The
> pool can grow, and if you configure a connection pool idle timeout, it
> will also shrink. Each web server process has its own pool. For Apache
> the default maximum size is equal to the number of request processing
> threads per process.
>
> The prefork Apache MPM uses a single thread per process, so you will end
> up with a single connection per web server process (at least if the
> process already forwarded at most one request to tomcat). The total
> maximum number of connections will then be equal to the maximum number
> of Apache processes configured.
>
> The worker MPM has a configurable number of threads per process, by
> default 25. The total maximum number of connections will be the maximum
> number of processes configured multiplied with the number of threads per
> process.
>
> The default configuration for prefork is 150 processes = connections,
> for worker it is 6 processes with 25 threads each, again = 150 connections.
>
> If your load goes up, the number of processes will grow and more and
> more connections get established.
>
> If you want those to get closed again during idle times, you find hints
> about that in the timeouts document.
>
> Important: you need to set the thread count in the connector element for
> the AJP connector to match the expected maximum of connections, that
> should be handled by Tomcat. Each incoming connection needs a dedicated
> thread, unless you are using the native APR connector.
>
> Keep fd leak hunting. Many admins forget about this tradition.
>
> Regards,
>
> Rainer
>
> Andrew Lee Rubinger wrote:
>> Hi.
>>
>> Rainer Jung wrote:
>>> Do you have special evidence, that the broken piepe errors correlate
>>> to the increase in file descriptors? Or are you only assuming this,
>>> because you can't find any other errors in the logs?
>>
>> Assuming. Should have made this more clear.
>>
>>> Can you see the connections for the file descriptors on the Apache
>>> machine and on the Tomcat machine (using netstat)? Are they in the
>>> same state? Which state?
>>
>> Attached as file. :)
>>
>>> At the moment I see no correlation between the client abort and a
>>> lost backend connection. There is another possible issue, if the file
>>> descriptor leak happens during tiume of low load.
>>
>> FD Leak seems to occur at higher rate during high load.
>>
>>> Since we just had a post about this on the list, I'm simply citing my
>>> last answer from another mail. Even if you think this does not apply
>>> to you, you should read the document the text refers to:
>>> If your analysis is correct, it's a known problem, that firewalls
>>> often drop idle connections, so mod_jk and Tomcat should both be
>>> configured to shut down idle connections. On the Tomcat side you
>>> should use the attribute connectionTimeout in the configuration of
>>> your AJP Connector in server.xml. The units are in milliseconds, so
>>> 10 Minutes would be 600000. On the mod_jk side you should use
>>> connection_pool_timeout, units are seconds, so 10 minutes are 600.
>>
>> Thanks so much; will absolutely look into this.
>>
>>> We have a new docs page about these things, which will go live next
>>> week with our 1.2.24 release. There is a preview for the page under
>>>
>>> http://people.apache.org/~rjung/mod_jk-dev/docs/generic_howto/timeouts.html
>>>
>>>
>>> The next release 1.2.24 will include some improvements for this
>>> situation, but you would still need the timeouts on both ends.
>>
>> Thanks so much for your help. Again, not 100% sure this is a ModJK
>> issue (especially after your comments), but those are my only errors
>> in the logs and it seemed a natural conclusion.
>>
>> S,
>> ALR
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat/AJP Client Abort Socket Errors - Leads to File Descriptor
Leak
Posted by Rainer Jung <ra...@kippdata.de>.
Hi Andrew,
this snapshot looks normal. There are 11 connections used and both sides
have the same idea about their state.
Web server to Tomcat connections are managed by mod_jk via pools. The
pool can grow, and if you configure a connection pool idle timeout, it
will also shrink. Each web server process has its own pool. For Apache
the default maximum size is equal to the number of request processing
threads per process.
The prefork Apache MPM uses a single thread per process, so you will end
up with a single connection per web server process (at least if the
process already forwarded at most one request to tomcat). The total
maximum number of connections will then be equal to the maximum number
of Apache processes configured.
The worker MPM has a configurable number of threads per process, by
default 25. The total maximum number of connections will be the maximum
number of processes configured multiplied with the number of threads per
process.
The default configuration for prefork is 150 processes = connections,
for worker it is 6 processes with 25 threads each, again = 150 connections.
If your load goes up, the number of processes will grow and more and
more connections get established.
If you want those to get closed again during idle times, you find hints
about that in the timeouts document.
Important: you need to set the thread count in the connector element for
the AJP connector to match the expected maximum of connections, that
should be handled by Tomcat. Each incoming connection needs a dedicated
thread, unless you are using the native APR connector.
Keep fd leak hunting. Many admins forget about this tradition.
Regards,
Rainer
Andrew Lee Rubinger wrote:
> Hi.
>
> Rainer Jung wrote:
>> Do you have special evidence, that the broken piepe errors correlate
>> to the increase in file descriptors? Or are you only assuming this,
>> because you can't find any other errors in the logs?
>
> Assuming. Should have made this more clear.
>
>> Can you see the connections for the file descriptors on the Apache
>> machine and on the Tomcat machine (using netstat)? Are they in the
>> same state? Which state?
>
> Attached as file. :)
>
>> At the moment I see no correlation between the client abort and a lost
>> backend connection. There is another possible issue, if the file
>> descriptor leak happens during tiume of low load.
>
> FD Leak seems to occur at higher rate during high load.
>
>> Since we just had a post about this on the list, I'm simply citing my
>> last answer from another mail. Even if you think this does not apply
>> to you, you should read the document the text refers to:
>> If your analysis is correct, it's a known problem, that firewalls
>> often drop idle connections, so mod_jk and Tomcat should both be
>> configured to shut down idle connections. On the Tomcat side you
>> should use the attribute connectionTimeout in the configuration of
>> your AJP Connector in server.xml. The units are in milliseconds, so 10
>> Minutes would be 600000. On the mod_jk side you should use
>> connection_pool_timeout, units are seconds, so 10 minutes are 600.
>
> Thanks so much; will absolutely look into this.
>
>> We have a new docs page about these things, which will go live next
>> week with our 1.2.24 release. There is a preview for the page under
>>
>> http://people.apache.org/~rjung/mod_jk-dev/docs/generic_howto/timeouts.html
>>
>>
>> The next release 1.2.24 will include some improvements for this
>> situation, but you would still need the timeouts on both ends.
>
> Thanks so much for your help. Again, not 100% sure this is a ModJK
> issue (especially after your comments), but those are my only errors in
> the logs and it seemed a natural conclusion.
>
> S,
> ALR
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat/AJP Client Abort Socket Errors - Leads to File Descriptor
Leak
Posted by Andrew Lee Rubinger <al...@9mmedia.com>.
Hi.
Rainer Jung wrote:
> Do you have special evidence, that the broken piepe errors correlate to
> the increase in file descriptors? Or are you only assuming this, because
> you can't find any other errors in the logs?
Assuming. Should have made this more clear.
> Can you see the connections for the file descriptors on the Apache
> machine and on the Tomcat machine (using netstat)? Are they in the same
> state? Which state?
Attached as file. :)
> At the moment I see no correlation between the client abort and a lost
> backend connection. There is another possible issue, if the file
> descriptor leak happens during tiume of low load.
FD Leak seems to occur at higher rate during high load.
> Since we just had a
> post about this on the list, I'm simply citing my last answer from
> another mail. Even if you think this does not apply to you, you should
> read the document the text refers to:
> If your analysis is correct, it's a known problem, that firewalls often
> drop idle connections, so mod_jk and Tomcat should both be configured to
> shut down idle connections. On the Tomcat side you should use the
> attribute connectionTimeout in the configuration of your AJP Connector
> in server.xml. The units are in milliseconds, so 10 Minutes would be
> 600000. On the mod_jk side you should use connection_pool_timeout, units
> are seconds, so 10 minutes are 600.
Thanks so much; will absolutely look into this.
> We have a new docs page about these things, which will go live next week
> with our 1.2.24 release. There is a preview for the page under
>
> http://people.apache.org/~rjung/mod_jk-dev/docs/generic_howto/timeouts.html
>
> The next release 1.2.24 will include some improvements for this
> situation, but you would still need the timeouts on both ends.
Thanks so much for your help. Again, not 100% sure this is a ModJK
issue (especially after your comments), but those are my only errors in
the logs and it seemed a natural conclusion.
S,
ALR
Re: Tomcat/AJP Client Abort Socket Errors - Leads to File Descriptor
Leak
Posted by Rainer Jung <ra...@kippdata.de>.
Andrew Lee Rubinger wrote:
> Hi.
>
> I am running Apache HTTPD 2.2.4 with ModJK 1.2.21, both compiled from
> source on CentOS 4.4. I have a workers.properties file configured to
> send requests to Tomcat 5.5 on a separate machine within the internal
> network.
>
> When receiving a ClientAbortException from Tomcat (shown in
> "tomcat_broken_pipe_error.txt"), it seems that the socket connection
> between ModJK and Tomcat is not closed out, and on the Application
> Server running Tomcat I'm observing a Life Descriptor leak (as shown in
> "lsof.txt").
>
> I've tried setting "recovery_options" to 4 as suggested by
> http://tomcat.apache.org/connectors-doc/reference/workers.html, but this
> has not corrected the problem.
>
> Have attached all relevant log excerpts and configurations in hopes that
> someone has come across this and might have some insight.
>
> Thanks in advance for your help. Will be happy to provide any other
> information I may have excluded.
Do you have special evidence, that the broken piepe errors correlate to
the increase in file descriptors? Or are you only assuming this, because
you can't find any other errors in the logs?
Can you see the connections for the file descriptors on the Apache
machine and on the Tomcat machine (using netstat)? Are they in the same
state? Which state?
At the moment I see no correlation between the client abort and a lost
backend connection. There is another possible issue, if the file
descriptor leak happens during tiume of low load. Since we just had a
post about this on the list, I'm simply citing my last answer from
another mail. Even if you think this does not apply to you, you should
read the document the text refers to:
...
If your analysis is correct, it's a known problem, that firewalls often
drop idle connections, so mod_jk and Tomcat should both be configured to
shut down idle connections. On the Tomcat side you should use the
attribute connectionTimeout in the configuration of your AJP Connector
in server.xml. The units are in milliseconds, so 10 Minutes would be
600000. On the mod_jk side you should use connection_pool_timeout, units
are seconds, so 10 minutes are 600.
We have a new docs page about these things, which will go live next week
with our 1.2.24 release. There is a preview for the page under
http://people.apache.org/~rjung/mod_jk-dev/docs/generic_howto/timeouts.html
The next release 1.2.24 will include some improvements for this
situation, but you would still need the timeouts on both ends.
...
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org