You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Lucian I <lu...@gmail.com> on 2013/03/15 21:03:03 UTC
Unix domain socket leak using NIO connector
Hi,
We are seeing a leak of Unix domain sockets in Tomcat 7.0.26 configured to
run with Http11NioProtocol connector.
This is running in Ubuntu Linux 12.04.1 LTS with OpenJDK 6u24.
The leak is about 15 Unix domain sockets per hour (based on our typical
request rate), and is reaching the 4096 file limit we configured in 10 days.
These Unix domain sockets tied to the Tomcat process can be seen by issuing
lsof command (e.g. lsof | grep tomcat7 | grep socket).
The leak is absent when Tomcat runs using the BIO connector, and this would
indicate a problem in the NIO connector implementation.
I've seen the same situation indicated by someone else in the past, but
with no response:
http://mail-archives.apache.org/mod_mbox/tomcat-users/201201.mbox/%3CCAJkSUv-DDKTCQ-pD7W=QOVmPH1dXeXOvcr+3mCgu05cqpT7Zjg@mail.gmail.com%3E
Thanks, Luca
Re: Unix domain socket leak using NIO connector
Posted by Lucian I <lu...@gmail.com>.
Hi Christopher,
My bad for not seeing the response to the similar question posted last year.
Using lsof:
$ lsof | grep tomcat7 | grep socket | head -2
java 1341 tomcat7 74u unix 0xffff88007a26a3c0 0t0
16985 socket
java 1341 tomcat7 98u unix 0xffff88007a26a3c0 0t0
16985 socket
......
The count of these unix domain sockets keeps increasing and they
belong to the same Tomcat process with PID 1341.
I've talked to someone that is able to run a "load" test again and
he's going to use Tomcat 7.0.37 now and see if it makes a difference.
I'll know that next Tuesday and let you know.
I know there is no JDK API to create a Unix domain socket, but,
apparently, the JVM may create them when the application is using NIO
API, and that was a surprise for me as well.
Something that caught my eye was posted here:
http://stackoverflow.com/questions/7038688/java-nio-causes-file-descriptor-leak
That guy implemented his own server using Java NIO and ran into the
same leak by cancelling the selection key in one thread and closing
the socket channel in another thread.
I don't know whether this is the case in Tomcat, but it's something to
consider if someone familiar with the implementation is willing to do
a bit of code inspection.
Thanks for looking at it,
Lucian
On Fri, Mar 15, 2013 at 4:31 PM, Christopher Schultz
<ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Lucian,
>
> On 3/15/13 4:03 PM, Lucian I wrote:
>> We are seeing a leak of Unix domain sockets in Tomcat 7.0.26
>> configured to run with Http11NioProtocol connector. This is running
>> in Ubuntu Linux 12.04.1 LTS with OpenJDK 6u24.
>>
>> The leak is about 15 Unix domain sockets per hour (based on our
>> typical request rate), and is reaching the 4096 file limit we
>> configured in 10 days. These Unix domain sockets tied to the Tomcat
>> process can be seen by issuing lsof command (e.g. lsof | grep
>> tomcat7 | grep socket).
>
> Can we get a sample of that output?
>
> I'm fairly sure that Java does not provide access to UNIX domain
> sockets, so this would have to be at the JVM implementation layer.
> Obviously, Tomcat could be mismanaging some resource which ultimately
> leaks these things at the native level, but it's not like we can grep
> the Tomcat code for "new UNIXDomainSocket" or anything like that.
>
>> The leak is absent when Tomcat runs using the BIO connector, and
>> this would indicate a problem in the NIO connector implementation.
>>
>> I've seen the same situation indicated by someone else in the past,
>> but with no response:
>> http://mail-archives.apache.org/mod_mbox/tomcat-users/201201.mbox/%3CCAJkSUv-DDKTCQ-pD7W=QOVmPH1dXeXOvcr+3mCgu05cqpT7Zjg@mail.gmail.com%3E
>
> There
>>
> certainly was a response:
> http://markmail.org/message/wvurlmltc4mxtrcp
>
> If you can provide more information, I'm sure it will help to diagnose
> any problem. Are you able to upgrade to the current version of Tomcat 7?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iEYEAREIAAYFAlFDhS8ACgkQ9CaO5/Lv0PAr8gCgvAAXZkMQetf2UpsxlPfei9BC
> YesAnjeYavpJsW5cN4sOuj/nowE7ODs/
> =5nyS
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Unix domain socket leak using NIO connector
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Lucian,
On 3/15/13 4:03 PM, Lucian I wrote:
> We are seeing a leak of Unix domain sockets in Tomcat 7.0.26
> configured to run with Http11NioProtocol connector. This is running
> in Ubuntu Linux 12.04.1 LTS with OpenJDK 6u24.
>
> The leak is about 15 Unix domain sockets per hour (based on our
> typical request rate), and is reaching the 4096 file limit we
> configured in 10 days. These Unix domain sockets tied to the Tomcat
> process can be seen by issuing lsof command (e.g. lsof | grep
> tomcat7 | grep socket).
Can we get a sample of that output?
I'm fairly sure that Java does not provide access to UNIX domain
sockets, so this would have to be at the JVM implementation layer.
Obviously, Tomcat could be mismanaging some resource which ultimately
leaks these things at the native level, but it's not like we can grep
the Tomcat code for "new UNIXDomainSocket" or anything like that.
> The leak is absent when Tomcat runs using the BIO connector, and
> this would indicate a problem in the NIO connector implementation.
>
> I've seen the same situation indicated by someone else in the past,
> but with no response:
> http://mail-archives.apache.org/mod_mbox/tomcat-users/201201.mbox/%3CCAJkSUv-DDKTCQ-pD7W=QOVmPH1dXeXOvcr+3mCgu05cqpT7Zjg@mail.gmail.com%3E
There
>
certainly was a response:
http://markmail.org/message/wvurlmltc4mxtrcp
If you can provide more information, I'm sure it will help to diagnose
any problem. Are you able to upgrade to the current version of Tomcat 7?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEAREIAAYFAlFDhS8ACgkQ9CaO5/Lv0PAr8gCgvAAXZkMQetf2UpsxlPfei9BC
YesAnjeYavpJsW5cN4sOuj/nowE7ODs/
=5nyS
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org