You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Adam Holmberg <ad...@gmail.com> on 2010/10/15 21:34:21 UTC

StoragePort Socket Leak 0.6.5

Greetings.

I'm operating a several two-node clusters (version 0.6.5) on VMs in our
development and test environments.

After about a week of operation under similar conditions, one of them
started throwing this:

WARN [main] 2010-10-12 08:08:31,245 CustomTThreadPoolServer.java (line 104)
Transport error occurred during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException:
Too many open files
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
        at
org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
        at
org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:98)
        at
org.apache.cassandra.thrift.CassandraDaemon.start(CassandraDaemon.java:186)
        at
org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:227)
Caused by: java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        at java.net.ServerSocket.implAccept(ServerSocket.java:453)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at
org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
        ... 5 more

I found that the offending node had hundreds of sockets (on the StoragePort,
between the two nodes) in CLOSE_WAIT state, which was causing new
connections to bump into the fd limit. It seems similar to what is
originally described (but never resolved) several months ago in this thread:


http://www.mail-archive.com/user@cassandra.apache.org/msg01381.html

Has anyone else encountered this problem? I am curious about what might
trigger this in one cluster and not on the others (which operate in the same
environment, and are configured similarly).

Any insight would be appreciated.

Thanks,
Adam

Re: StoragePort Socket Leak 0.6.5

Posted by Jonathan Ellis <jb...@gmail.com>.
sounds like https://issues.apache.org/jira/browse/CASSANDRA-1528 which
was fixed in 0.6.6.

On Fri, Oct 15, 2010 at 2:34 PM, Adam Holmberg
<ad...@gmail.com> wrote:
> Greetings.
>
> I'm operating a several two-node clusters (version 0.6.5) on VMs in our
> development and test environments.
>
> After about a week of operation under similar conditions, one of them
> started throwing this:
>
> WARN [main] 2010-10-12 08:08:31,245 CustomTThreadPoolServer.java (line 104)
> Transport error occurred during acceptance of message.
> org.apache.thrift.transport.TTransportException: java.net.SocketException:
> Too many open files
>         at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
>         at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
>         at
> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>         at
> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:98)
>         at
> org.apache.cassandra.thrift.CassandraDaemon.start(CassandraDaemon.java:186)
>         at
> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:227)
> Caused by: java.net.SocketException: Too many open files
>         at java.net.PlainSocketImpl.socketAccept(Native Method)
>         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>         at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>         at java.net.ServerSocket.accept(ServerSocket.java:421)
>         at
> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
>         ... 5 more
>
> I found that the offending node had hundreds of sockets (on the StoragePort,
> between the two nodes) in CLOSE_WAIT state, which was causing new
> connections to bump into the fd limit. It seems similar to what is
> originally described (but never resolved) several months ago in this thread:
>
> http://www.mail-archive.com/user@cassandra.apache.org/msg01381.html
>
> Has anyone else encountered this problem? I am curious about what might
> trigger this in one cluster and not on the others (which operate in the same
> environment, and are configured similarly).
>
> Any insight would be appreciated.
>
> Thanks,
> Adam
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com