You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Marcelo Elias Del Valle <ma...@s1mbi0se.com.br> on 2014/08/08 21:35:55 UTC
too many open files
Hi,
I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having "too
many open files" exceptions when I try to perform a large number of
operations in my 10 node cluster.
I saw the documentation
http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
and I have set everything to the recommended settings, but I keep getting
the errors.
In the documentation it says: "Another, much less likely possibility, is a
file descriptor leak in Cassandra. Run lsof -n | grep java to check that
the number of file descriptors opened by Java is reasonable and reports the
error if the number is greater than a few thousand."
I guess it's not the case, or else a lot of people would be complaining
about it, but I am not sure what I could do to solve the problem.
Any hint about how to solve it?
My client is written in python and uses Cassandra Python Driver. Here are
the exceptions I am having in the client:
[s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
seconds: Timed out connecting to 200.200.200.144
[s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
seconds: Timed out connecting to 200.200.200.77
And here is the exception I am having in the server:
WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
BatchStatement.java (line 223) Batch of prepared statements for
[identification.entity_lookup, identification.entity] is of size 25216,
exceeding specified threshold of 5120 by 20096.
ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
at
org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at
org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Here is the amount of open files before and after I restart Cassandra:
root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
936580
root@h:/etc/security/limits.d# sudo service cassandra restart
[ ok ] Restarting Cassandra: cassandra.
root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
80295
Best regards,
Marcelo Valle.
Re: too many open files
Posted by Marcelo Elias Del Valle <ma...@s1mbi0se.com.br>.
I just solved the issue, it was Cassandra process which was opening too
many fds, indeed, but the problem was the amount of client connections
being opened. It was opening more connection than needed in the client'
side.
Thanks for the help.
[]s
2014-08-08 17:17 GMT-03:00 Kevin Burton <bu...@spinn3r.com>:
> You may want to look at the the actual filenames. You might have an app
> leaving them open. Also, remember, sockets use FDs so they are in the list
> too.
>
>
> On Fri, Aug 8, 2014 at 1:13 PM, Marcelo Elias Del Valle <
> marcelo@s1mbi0se.com.br> wrote:
>
>> I am using datastax community, the packaged version for Debian. I am also
>> using last version of opscenter and datastax-agent
>>
>> However, I just listed open files here:
>>
>> sudo lsof -n | grep java | wc -l
>> 1096599
>>
>> It seems it has exceed. Should I just increase? Or is it possible to be a
>> memory leak?
>>
>> Best regards,
>> Marcelo.
>>
>>
>>
>> 2014-08-08 17:06 GMT-03:00 Shane Hansen <sh...@gmail.com>:
>>
>> Are you using apache or Datastax cassandra?
>>>
>>> The datastax distribution ups the file handle limit to 100000. That
>>> number's hard to exceed.
>>>
>>>
>>>
>>> On Fri, Aug 8, 2014 at 1:35 PM, Marcelo Elias Del Valle <
>>> marcelo@s1mbi0se.com.br> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having
>>>> "too many open files" exceptions when I try to perform a large number of
>>>> operations in my 10 node cluster.
>>>>
>>>> I saw the documentation
>>>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
>>>> and I have set everything to the recommended settings, but I keep getting
>>>> the errors.
>>>>
>>>> In the documentation it says: "Another, much less likely possibility,
>>>> is a file descriptor leak in Cassandra. Run lsof -n | grep java to
>>>> check that the number of file descriptors opened by Java is reasonable and
>>>> reports the error if the number is greater than a few thousand."
>>>>
>>>> I guess it's not the case, or else a lot of people would be complaining
>>>> about it, but I am not sure what I could do to solve the problem.
>>>>
>>>> Any hint about how to solve it?
>>>>
>>>> My client is written in python and uses Cassandra Python Driver. Here
>>>> are the exceptions I am having in the client:
>>>> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
>>>> seconds: Timed out connecting to 200.200.200.144
>>>> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
>>>> seconds: Timed out connecting to 200.200.200.77
>>>>
>>>>
>>>> And here is the exception I am having in the server:
>>>>
>>>> WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
>>>> BatchStatement.java (line 223) Batch of prepared statements for
>>>> [identification.entity_lookup, identification.entity] is of size 25216,
>>>> exceeding specified threshold of 5120 by 20096.
>>>> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
>>>> ErrorMessage.java (line 222) Unexpected exception during request
>>>> java.io.IOException: Connection reset by peer
>>>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
>>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> Here is the amount of open files before and after I restart Cassandra:
>>>>
>>>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>>>> 936580
>>>> root@h:/etc/security/limits.d# sudo service cassandra restart
>>>> [ ok ] Restarting Cassandra: cassandra.
>>>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>>>> 80295
>>>>
>>>>
>>>> Best regards,
>>>> Marcelo Valle.
>>>>
>>>
>>>
>>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
>
Re: too many open files
Posted by Kevin Burton <bu...@spinn3r.com>.
You may want to look at the the actual filenames. You might have an app
leaving them open. Also, remember, sockets use FDs so they are in the list
too.
On Fri, Aug 8, 2014 at 1:13 PM, Marcelo Elias Del Valle <
marcelo@s1mbi0se.com.br> wrote:
> I am using datastax community, the packaged version for Debian. I am also
> using last version of opscenter and datastax-agent
>
> However, I just listed open files here:
>
> sudo lsof -n | grep java | wc -l
> 1096599
>
> It seems it has exceed. Should I just increase? Or is it possible to be a
> memory leak?
>
> Best regards,
> Marcelo.
>
>
>
> 2014-08-08 17:06 GMT-03:00 Shane Hansen <sh...@gmail.com>:
>
> Are you using apache or Datastax cassandra?
>>
>> The datastax distribution ups the file handle limit to 100000. That
>> number's hard to exceed.
>>
>>
>>
>> On Fri, Aug 8, 2014 at 1:35 PM, Marcelo Elias Del Valle <
>> marcelo@s1mbi0se.com.br> wrote:
>>
>>> Hi,
>>>
>>> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having
>>> "too many open files" exceptions when I try to perform a large number of
>>> operations in my 10 node cluster.
>>>
>>> I saw the documentation
>>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
>>> and I have set everything to the recommended settings, but I keep getting
>>> the errors.
>>>
>>> In the documentation it says: "Another, much less likely possibility,
>>> is a file descriptor leak in Cassandra. Run lsof -n | grep java to
>>> check that the number of file descriptors opened by Java is reasonable and
>>> reports the error if the number is greater than a few thousand."
>>>
>>> I guess it's not the case, or else a lot of people would be complaining
>>> about it, but I am not sure what I could do to solve the problem.
>>>
>>> Any hint about how to solve it?
>>>
>>> My client is written in python and uses Cassandra Python Driver. Here
>>> are the exceptions I am having in the client:
>>> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
>>> seconds: Timed out connecting to 200.200.200.144
>>> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
>>> seconds: Timed out connecting to 200.200.200.77
>>>
>>>
>>> And here is the exception I am having in the server:
>>>
>>> WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
>>> BatchStatement.java (line 223) Batch of prepared statements for
>>> [identification.entity_lookup, identification.entity] is of size 25216,
>>> exceeding specified threshold of 5120 by 20096.
>>> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
>>> ErrorMessage.java (line 222) Unexpected exception during request
>>> java.io.IOException: Connection reset by peer
>>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Here is the amount of open files before and after I restart Cassandra:
>>>
>>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>>> 936580
>>> root@h:/etc/security/limits.d# sudo service cassandra restart
>>> [ ok ] Restarting Cassandra: cassandra.
>>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>>> 80295
>>>
>>>
>>> Best regards,
>>> Marcelo Valle.
>>>
>>
>>
>
--
Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Re: too many open files
Posted by Marcelo Elias Del Valle <ma...@s1mbi0se.com.br>.
I am using datastax community, the packaged version for Debian. I am also
using last version of opscenter and datastax-agent
However, I just listed open files here:
sudo lsof -n | grep java | wc -l
1096599
It seems it has exceed. Should I just increase? Or is it possible to be a
memory leak?
Best regards,
Marcelo.
2014-08-08 17:06 GMT-03:00 Shane Hansen <sh...@gmail.com>:
> Are you using apache or Datastax cassandra?
>
> The datastax distribution ups the file handle limit to 100000. That
> number's hard to exceed.
>
>
>
> On Fri, Aug 8, 2014 at 1:35 PM, Marcelo Elias Del Valle <
> marcelo@s1mbi0se.com.br> wrote:
>
>> Hi,
>>
>> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having "too
>> many open files" exceptions when I try to perform a large number of
>> operations in my 10 node cluster.
>>
>> I saw the documentation
>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
>> and I have set everything to the recommended settings, but I keep getting
>> the errors.
>>
>> In the documentation it says: "Another, much less likely possibility, is
>> a file descriptor leak in Cassandra. Run lsof -n | grep java to check
>> that the number of file descriptors opened by Java is reasonable and
>> reports the error if the number is greater than a few thousand."
>>
>> I guess it's not the case, or else a lot of people would be complaining
>> about it, but I am not sure what I could do to solve the problem.
>>
>> Any hint about how to solve it?
>>
>> My client is written in python and uses Cassandra Python Driver. Here are
>> the exceptions I am having in the client:
>> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
>> seconds: Timed out connecting to 200.200.200.144
>> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
>> seconds: Timed out connecting to 200.200.200.77
>>
>>
>> And here is the exception I am having in the server:
>>
>> WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
>> BatchStatement.java (line 223) Batch of prepared statements for
>> [identification.entity_lookup, identification.entity] is of size 25216,
>> exceeding specified threshold of 5120 by 20096.
>> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
>> ErrorMessage.java (line 222) Unexpected exception during request
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Here is the amount of open files before and after I restart Cassandra:
>>
>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>> 936580
>> root@h:/etc/security/limits.d# sudo service cassandra restart
>> [ ok ] Restarting Cassandra: cassandra.
>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>> 80295
>>
>>
>> Best regards,
>> Marcelo Valle.
>>
>
>
Re: too many open files
Posted by Shane Hansen <sh...@gmail.com>.
Are you using apache or Datastax cassandra?
The datastax distribution ups the file handle limit to 100000. That
number's hard to exceed.
On Fri, Aug 8, 2014 at 1:35 PM, Marcelo Elias Del Valle <
marcelo@s1mbi0se.com.br> wrote:
> Hi,
>
> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having "too
> many open files" exceptions when I try to perform a large number of
> operations in my 10 node cluster.
>
> I saw the documentation
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
> and I have set everything to the recommended settings, but I keep getting
> the errors.
>
> In the documentation it says: "Another, much less likely possibility, is
> a file descriptor leak in Cassandra. Run lsof -n | grep java to check
> that the number of file descriptors opened by Java is reasonable and
> reports the error if the number is greater than a few thousand."
>
> I guess it's not the case, or else a lot of people would be complaining
> about it, but I am not sure what I could do to solve the problem.
>
> Any hint about how to solve it?
>
> My client is written in python and uses Cassandra Python Driver. Here are
> the exceptions I am having in the client:
> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
> seconds: Timed out connecting to 200.200.200.144
> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
> seconds: Timed out connecting to 200.200.200.77
>
>
> And here is the exception I am having in the server:
>
> WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
> BatchStatement.java (line 223) Batch of prepared statements for
> [identification.entity_lookup, identification.entity] is of size 25216,
> exceeding specified threshold of 5120 by 20096.
> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
> ErrorMessage.java (line 222) Unexpected exception during request
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> Here is the amount of open files before and after I restart Cassandra:
>
> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
> 936580
> root@h:/etc/security/limits.d# sudo service cassandra restart
> [ ok ] Restarting Cassandra: cassandra.
> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
> 80295
>
>
> Best regards,
> Marcelo Valle.
>
Re: too many open files
Posted by Andrew <re...@gmail.com>.
Yes, that was the problem—I actually knew better, but had briefly overlooked this that when I was doing some refactoring. I am not the OP (although he himself realized his mistake).
if you follow the thread, I was explaining that the Datastax Java driver allowed me to basically open a significantly large number of connections until the Cassandra server ran out of connections. Tyler was asking for a repro case and requesting that I file a possible bug, if this was something that SHOULD have been caught on the client side (via the max connections client configuration).
Andrew
On August 9, 2014 at 2:17:57 PM, Jonathan Haddad (jon@jonhaddad.com) wrote:
It really doesn't need to be this complicated. You only need 1
session per application. It's thread safe and manages the connection
pool for you.
http://www.datastax.com/drivers/java/2.0/com/datastax/driver/core/Session.html
On Sat, Aug 9, 2014 at 1:29 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> Another idea to detect this is when the number of open sessions exceeds the
> number of threads.
>
> On Aug 9, 2014 10:59 AM, "Andrew" <re...@gmail.com> wrote:
>>
>> I just had a generator that (in the incorrect way) had a cluster as a
>> member variable, and would call .connect() repeatedly. I _thought_,
>> incorrectly, that the Session was thread unsafe, and so I should request a
>> separate Session each time—obviously wrong in hind sight.
>>
>> There was no special logic; I had a restriction of about 128 connections
>> per host, but the connections were in the 100s of thousands, like the OP
>> mentioned. Again, I’ll see about reproducing it on Monday, but just wanted
>> the repro steps (overall) to live somewhere in case I can’t. :)
>>
>> Andrew
>>
>> On August 8, 2014 at 4:08:50 PM, Tyler Hobbs (tyler@datastax.com) wrote:
>>
>>
>> On Fri, Aug 8, 2014 at 5:52 PM, Redmumba <re...@gmail.com> wrote:
>>>
>>> Just to chime in, I also ran into this issue when I was migrating to the
>>> Datastax client. Instead of reusing the session, I was opening a new session
>>> each time. For some reason, even though I was still closing the session on
>>> the client side, I was getting the same error.
>>
>>
>> Which driver? If you can still reproduce this, would you mind opening a
>> ticket? (https://datastax-oss.atlassian.net/secure/BrowseProjects.jspa#all)
>>
>>
>> --
>> Tyler Hobbs
>> DataStax
--
Jon Haddad
http://www.rustyrazorblade.com
skype: rustyrazorblade
Re: too many open files
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
It really doesn't need to be this complicated. You only need 1
session per application. It's thread safe and manages the connection
pool for you.
http://www.datastax.com/drivers/java/2.0/com/datastax/driver/core/Session.html
On Sat, Aug 9, 2014 at 1:29 PM, Kevin Burton <bu...@spinn3r.com> wrote:
> Another idea to detect this is when the number of open sessions exceeds the
> number of threads.
>
> On Aug 9, 2014 10:59 AM, "Andrew" <re...@gmail.com> wrote:
>>
>> I just had a generator that (in the incorrect way) had a cluster as a
>> member variable, and would call .connect() repeatedly. I _thought_,
>> incorrectly, that the Session was thread unsafe, and so I should request a
>> separate Session each time—obviously wrong in hind sight.
>>
>> There was no special logic; I had a restriction of about 128 connections
>> per host, but the connections were in the 100s of thousands, like the OP
>> mentioned. Again, I’ll see about reproducing it on Monday, but just wanted
>> the repro steps (overall) to live somewhere in case I can’t. :)
>>
>> Andrew
>>
>> On August 8, 2014 at 4:08:50 PM, Tyler Hobbs (tyler@datastax.com) wrote:
>>
>>
>> On Fri, Aug 8, 2014 at 5:52 PM, Redmumba <re...@gmail.com> wrote:
>>>
>>> Just to chime in, I also ran into this issue when I was migrating to the
>>> Datastax client. Instead of reusing the session, I was opening a new session
>>> each time. For some reason, even though I was still closing the session on
>>> the client side, I was getting the same error.
>>
>>
>> Which driver? If you can still reproduce this, would you mind opening a
>> ticket? (https://datastax-oss.atlassian.net/secure/BrowseProjects.jspa#all)
>>
>>
>> --
>> Tyler Hobbs
>> DataStax
--
Jon Haddad
http://www.rustyrazorblade.com
skype: rustyrazorblade
Re: too many open files
Posted by Kevin Burton <bu...@spinn3r.com>.
Another idea to detect this is when the number of open sessions exceeds the
number of threads.
On Aug 9, 2014 10:59 AM, "Andrew" <re...@gmail.com> wrote:
> I just had a generator that (in the incorrect way) had a cluster as a
> member variable, and would call .connect() repeatedly. I _thought_,
> incorrectly, that the Session was thread unsafe, and so I should request a
> separate Session each time—obviously wrong in hind sight.
>
> There was no special logic; I had a restriction of about 128 connections
> per host, but the connections were in the 100s of thousands, like the OP
> mentioned. Again, I’ll see about reproducing it on Monday, but just wanted
> the repro steps (overall) to live somewhere in case I can’t. :)
>
> Andrew
>
> On August 8, 2014 at 4:08:50 PM, Tyler Hobbs (tyler@datastax.com) wrote:
>
>
> On Fri, Aug 8, 2014 at 5:52 PM, Redmumba <re...@gmail.com> wrote:
>
>> Just to chime in, I also ran into this issue when I was migrating to the
>> Datastax client. Instead of reusing the session, I was opening a new
>> session each time. For some reason, even though I was still closing the
>> session on the client side, I was getting the same error.
>
>
> Which driver? If you can still reproduce this, would you mind opening a
> ticket? (https://datastax-oss.atlassian.net/secure/BrowseProjects.jspa#all
> )
>
>
> --
> Tyler Hobbs
> DataStax <http://datastax.com/>
>
>
Re: too many open files
Posted by Andrew <re...@gmail.com>.
I just had a generator that (in the incorrect way) had a cluster as a member variable, and would call .connect() repeatedly. I _thought_, incorrectly, that the Session was thread unsafe, and so I should request a separate Session each time—obviously wrong in hind sight.
There was no special logic; I had a restriction of about 128 connections per host, but the connections were in the 100s of thousands, like the OP mentioned. Again, I’ll see about reproducing it on Monday, but just wanted the repro steps (overall) to live somewhere in case I can’t. :)
Andrew
On August 8, 2014 at 4:08:50 PM, Tyler Hobbs (tyler@datastax.com) wrote:
On Fri, Aug 8, 2014 at 5:52 PM, Redmumba <re...@gmail.com> wrote:
Just to chime in, I also ran into this issue when I was migrating to the Datastax client. Instead of reusing the session, I was opening a new session each time. For some reason, even though I was still closing the session on the client side, I was getting the same error.
Which driver? If you can still reproduce this, would you mind opening a ticket? (https://datastax-oss.atlassian.net/secure/BrowseProjects.jspa#all)
--
Tyler Hobbs
DataStax
Re: too many open files
Posted by Andrew <re...@gmail.com>.
Tyler,
I’ll see if I can reproduce this on a local instance, but just in case, the error was basically—instead of storing the session in my connection factory, I stored a cluster and called “connect” each time I requested a Session. I had defined a max/min number of connections for the connection itself, maxing out at 128 local/remote. I’m not sure if a Session results in a new file handle on the server side, but I saw the same issue (hundreds of thousands of sockets opened on the server).
The cluster was also using hsha; most of the other settings were default in 2.0.7.
Andrew
On August 8, 2014 at 4:08:50 PM, Tyler Hobbs (tyler@datastax.com) wrote:
On Fri, Aug 8, 2014 at 5:52 PM, Redmumba <re...@gmail.com> wrote:
Just to chime in, I also ran into this issue when I was migrating to the Datastax client. Instead of reusing the session, I was opening a new session each time. For some reason, even though I was still closing the session on the client side, I was getting the same error.
Which driver? If you can still reproduce this, would you mind opening a ticket? (https://datastax-oss.atlassian.net/secure/BrowseProjects.jspa#all)
--
Tyler Hobbs
DataStax
Re: too many open files
Posted by Tyler Hobbs <ty...@datastax.com>.
On Fri, Aug 8, 2014 at 5:52 PM, Redmumba <re...@gmail.com> wrote:
> Just to chime in, I also ran into this issue when I was migrating to the
> Datastax client. Instead of reusing the session, I was opening a new
> session each time. For some reason, even though I was still closing the
> session on the client side, I was getting the same error.
Which driver? If you can still reproduce this, would you mind opening a
ticket? (https://datastax-oss.atlassian.net/secure/BrowseProjects.jspa#all)
--
Tyler Hobbs
DataStax <http://datastax.com/>
Re: too many open files
Posted by Marcelo Elias Del Valle <ma...@s1mbi0se.com.br>.
IMHO, I think the drivers are fine. It was a dumb mistake of mine to use
sessions as connections and not as connection pools.
What was harder to figure, in my opinion, was that too many connections
from the client would increase the amount of file descriptors used by the
server. I didn't know Linux open a FD for each connection received and
honestly I still don't know much about the details of this. When I got a
"too many open files" error it took a good while to think about checking
the connections.
I think the documentation could point this fact, it would help other people
with the same problem.
There could be something talking about it here:
http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
[]s
2014-08-09 12:55 GMT-03:00 Jack Krupansky <ja...@basetechnology.com>:
> Maybe the drivers should have two modes: few sessions, and lots of
> sessions. The former would give you a developer-friendly driver error if
> you leave more than say a dozen or two dozen sessions open (or whatever is
> considered a best practice for parallel threads in a client), on the theory
> that you probably used the anti-pattern of failing to reuse sessions. The
> latter would be more for expert apps that have some good reason for having
> hundreds or thousands of simultaneous sessions open. Whether the latter
> also has some (configurable) limit that is simply a lot higher than the
> former or is unlimited, is probably not so important. Or, maybe, simply
> have a single limit, without the modes and default it to 10 or 25 or some
> other relatively low number for “normal” apps.
>
> This would be more developer-friendly, for both new and “normal”
> developers... I think.
>
> -- Jack Krupansky
>
> *From:* Marcelo Elias Del Valle <ma...@s1mbi0se.com.br>
> *Sent:* Saturday, August 9, 2014 12:41 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: too many open files
>
> Indeed, that was my mistake, that was exactly what we were doing in the
> code.
> []s
>
>
> 2014-08-09 0:56 GMT-03:00 Brian Zhang <yi...@gmail.com>:
>
>> For cassandra driver,session is just like database connection pool,it
>> maybe contains many tcp connections,if you create a new session every
>> time,more and more tcp connections will be created,till surpass the max
>> file description limit of os.
>>
>> You should create one session,use it repeatedly ,session can manage
>> connections automatically,create new connection or close old connection for
>> your requests.
>>
>> 在 2014年8月9日，6:52，Redmumba <re...@gmail.com> 写道：
>>
>> Just to chime in, I also ran into this issue when I was migrating to
>> the Datastax client. Instead of reusing the session, I was opening a new
>> session each time. For some reason, even though I was still closing the
>> session on the client side, I was getting the same error.
>>
>> Plus, the only way I could recover was by restarting Cassandra. I did not
>> really see the connections timeout over a period of a few minutes.
>>
>> Andrew
>> On Aug 8, 2014 3:19 PM, "Andrey Ilinykh" <ai...@gmail.com> wrote:
>>
>>> You may have this problem if your client doesn't reuse the connection
>>> but opens new every type. So, run netstat and check the number of
>>> established connections. This number should not be big.
>>>
>>> Thank you,
>>> Andrey
>>>
>>>
>>> On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle <
>>> marcelo@s1mbi0se.com.br> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having
>>>> "too many open files" exceptions when I try to perform a large number of
>>>> operations in my 10 node cluster.
>>>>
>>>> I saw the documentation
>>>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
>>>> and I have set everything to the recommended settings, but I keep getting
>>>> the errors.
>>>>
>>>> In the documentation it says: "Another, much less likely possibility,
>>>> is a file descriptor leak in Cassandra. Run lsof -n | grep java to
>>>> check that the number of file descriptors opened by Java is reasonable and
>>>> reports the error if the number is greater than a few thousand."
>>>>
>>>> I guess it's not the case, or else a lot of people would be complaining
>>>> about it, but I am not sure what I could do to solve the problem.
>>>>
>>>> Any hint about how to solve it?
>>>>
>>>> My client is written in python and uses Cassandra Python Driver. Here
>>>> are the exceptions I am having in the client:
>>>> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
>>>> seconds: [Errno 24] Too many open files
>>>> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
>>>> seconds: Timed out connecting to 200.200.200.144
>>>> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
>>>> attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
>>>> seconds: Timed out connecting to 200.200.200.77
>>>>
>>>>
>>>> And here is the exception I am having in the server:
>>>>
>>>> WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
>>>> BatchStatement.java (line 223) Batch of prepared statements for
>>>> [identification.entity_lookup, identification.entity] is of size 25216,
>>>> exceeding specified threshold of 5120 by 20096.
>>>> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
>>>> ErrorMessage.java (line 222) Unexpected exception during request
>>>> java.io.IOException: Connection reset by peer
>>>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>>> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
>>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>>>> at
>>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>>
>>>> Here is the amount of open files before and after I restart Cassandra:
>>>>
>>>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>>>> 936580
>>>> root@h:/etc/security/limits.d# sudo service cassandra restart
>>>> [ ok ] Restarting Cassandra: cassandra.
>>>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>>>> 80295
>>>>
>>>>
>>>> Best regards,
>>>> Marcelo Valle.
>>>>
>>>
>>>
>>
>>
>
>
Re: too many open files
Posted by Jack Krupansky <ja...@basetechnology.com>.
Maybe the drivers should have two modes: few sessions, and lots of sessions. The former would give you a developer-friendly driver error if you leave more than say a dozen or two dozen sessions open (or whatever is considered a best practice for parallel threads in a client), on the theory that you probably used the anti-pattern of failing to reuse sessions. The latter would be more for expert apps that have some good reason for having hundreds or thousands of simultaneous sessions open. Whether the latter also has some (configurable) limit that is simply a lot higher than the former or is unlimited, is probably not so important. Or, maybe, simply have a single limit, without the modes and default it to 10 or 25 or some other relatively low number for “normal” apps.
This would be more developer-friendly, for both new and “normal” developers... I think.
-- Jack Krupansky
From: Marcelo Elias Del Valle
Sent: Saturday, August 9, 2014 12:41 AM
To: user@cassandra.apache.org
Subject: Re: too many open files
Indeed, that was my mistake, that was exactly what we were doing in the code.
[]s
2014-08-09 0:56 GMT-03:00 Brian Zhang <yi...@gmail.com>:
For cassandra driver,session is just like database connection pool,it maybe contains many tcp connections,if you create a new session every time,more and more tcp connections will be created,till surpass the max file description limit of os.
You should create one session,use it repeatedly ,session can manage connections automatically,create new connection or close old connection for your requests.
在 2014年8月9日，6:52，Redmumba <re...@gmail.com> 写道：
Just to chime in, I also ran into this issue when I was migrating to the Datastax client. Instead of reusing the session, I was opening a new session each time. For some reason, even though I was still closing the session on the client side, I was getting the same error.
Plus, the only way I could recover was by restarting Cassandra. I did not really see the connections timeout over a period of a few minutes.
Andrew
On Aug 8, 2014 3:19 PM, "Andrey Ilinykh" <ai...@gmail.com> wrote:
You may have this problem if your client doesn't reuse the connection but opens new every type. So, run netstat and check the number of established connections. This number should not be big.
Thank you,
Andrey
On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle <ma...@s1mbi0se.com.br> wrote:
Hi,
I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having "too many open files" exceptions when I try to perform a large number of operations in my 10 node cluster.
I saw the documentation http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html and I have set everything to the recommended settings, but I keep getting the errors.
In the documentation it says: "Another, much less likely possibility, is a file descriptor leak in Cassandra. Run lsof -n | grep java to check that the number of file descriptors opened by Java is reasonable and reports the error if the number is greater than a few thousand."
I guess it's not the case, or else a lot of people would be complaining about it, but I am not sure what I could do to solve the problem.
Any hint about how to solve it?
My client is written in python and uses Cassandra Python Driver. Here are the exceptions I am having in the client:
[s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.151, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.143, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.145, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.144, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.148, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.146, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.77, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.76, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.75, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.185, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
[s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.144, scheduling retry in 512.0 seconds: Timed out connecting to 200.200.200.144
[s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.77, scheduling retry in 512.0 seconds: Timed out connecting to 200.200.200.77
And here is the exception I am having in the server:
WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499 BatchStatement.java (line 223) Batch of prepared statements for [identification.entity_lookup, identification.entity] is of size 25216, exceeding specified threshold of 5120 by 20096.
ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611 ErrorMessage.java (line 222) Unexpected exception during request
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Here is the amount of open files before and after I restart Cassandra:
root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
936580
root@h:/etc/security/limits.d# sudo service cassandra restart
[ ok ] Restarting Cassandra: cassandra.
root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
80295
Best regards,
Marcelo Valle.
Re: too many open files
Posted by Marcelo Elias Del Valle <ma...@s1mbi0se.com.br>.
Indeed, that was my mistake, that was exactly what we were doing in the
code.
[]s
2014-08-09 0:56 GMT-03:00 Brian Zhang <yi...@gmail.com>:
> For cassandra driver,session is just like database connection pool,it
> maybe contains many tcp connections,if you create a new session every
> time,more and more tcp connections will be created,till surpass the max
> file description limit of os.
>
> You should create one session,use it repeatedly ,session can manage
> connections automatically,create new connection or close old connection for
> your requests.
>
> 在 2014年8月9日，6:52，Redmumba <re...@gmail.com> 写道：
>
> Just to chime in, I also ran into this issue when I was migrating to the
> Datastax client. Instead of reusing the session, I was opening a new
> session each time. For some reason, even though I was still closing the
> session on the client side, I was getting the same error.
>
> Plus, the only way I could recover was by restarting Cassandra. I did not
> really see the connections timeout over a period of a few minutes.
>
> Andrew
> On Aug 8, 2014 3:19 PM, "Andrey Ilinykh" <ai...@gmail.com> wrote:
>
>> You may have this problem if your client doesn't reuse the connection but
>> opens new every type. So, run netstat and check the number of established
>> connections. This number should not be big.
>>
>> Thank you,
>> Andrey
>>
>>
>> On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle <
>> marcelo@s1mbi0se.com.br> wrote:
>>
>>> Hi,
>>>
>>> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having
>>> "too many open files" exceptions when I try to perform a large number of
>>> operations in my 10 node cluster.
>>>
>>> I saw the documentation
>>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
>>> and I have set everything to the recommended settings, but I keep getting
>>> the errors.
>>>
>>> In the documentation it says: "Another, much less likely possibility,
>>> is a file descriptor leak in Cassandra. Run lsof -n | grep java to
>>> check that the number of file descriptors opened by Java is reasonable and
>>> reports the error if the number is greater than a few thousand."
>>>
>>> I guess it's not the case, or else a lot of people would be complaining
>>> about it, but I am not sure what I could do to solve the problem.
>>>
>>> Any hint about how to solve it?
>>>
>>> My client is written in python and uses Cassandra Python Driver. Here
>>> are the exceptions I am having in the client:
>>> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
>>> seconds: [Errno 24] Too many open files
>>> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
>>> seconds: Timed out connecting to 200.200.200.144
>>> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
>>> attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
>>> seconds: Timed out connecting to 200.200.200.77
>>>
>>>
>>> And here is the exception I am having in the server:
>>>
>>> WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
>>> BatchStatement.java (line 223) Batch of prepared statements for
>>> [identification.entity_lookup, identification.entity] is of size 25216,
>>> exceeding specified threshold of 5120 by 20096.
>>> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
>>> ErrorMessage.java (line 222) Unexpected exception during request
>>> java.io.IOException: Connection reset by peer
>>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>>> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>>> at
>>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>>> at
>>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> Here is the amount of open files before and after I restart Cassandra:
>>>
>>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>>> 936580
>>> root@h:/etc/security/limits.d# sudo service cassandra restart
>>> [ ok ] Restarting Cassandra: cassandra.
>>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>>> 80295
>>>
>>>
>>> Best regards,
>>> Marcelo Valle.
>>>
>>
>>
>
Re: too many open files
Posted by Brian Zhang <yi...@gmail.com>.
For cassandra driver,session is just like database connection pool,it maybe contains many tcp connections,if you create a new session every time,more and more tcp connections will be created,till surpass the max file description limit of os.
You should create one session,use it repeatedly ,session can manage connections automatically,create new connection or close old connection for your requests.
在 2014年8月9日，6:52，Redmumba <re...@gmail.com> 写道：
> Just to chime in, I also ran into this issue when I was migrating to the Datastax client. Instead of reusing the session, I was opening a new session each time. For some reason, even though I was still closing the session on the client side, I was getting the same error.
>
> Plus, the only way I could recover was by restarting Cassandra. I did not really see the connections timeout over a period of a few minutes.
>
> Andrew
>
> On Aug 8, 2014 3:19 PM, "Andrey Ilinykh" <ai...@gmail.com> wrote:
> You may have this problem if your client doesn't reuse the connection but opens new every type. So, run netstat and check the number of established connections. This number should not be big.
>
> Thank you,
> Andrey
>
>
> On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle <ma...@s1mbi0se.com.br> wrote:
> Hi,
>
> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having "too many open files" exceptions when I try to perform a large number of operations in my 10 node cluster.
>
> I saw the documentation http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html and I have set everything to the recommended settings, but I keep getting the errors.
>
> In the documentation it says: "Another, much less likely possibility, is a file descriptor leak in Cassandra. Run lsof -n | grep java to check that the number of file descriptors opened by Java is reasonable and reports the error if the number is greater than a few thousand."
>
> I guess it's not the case, or else a lot of people would be complaining about it, but I am not sure what I could do to solve the problem.
>
> Any hint about how to solve it?
>
> My client is written in python and uses Cassandra Python Driver. Here are the exceptions I am having in the client:
> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.151, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.143, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.145, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.144, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.148, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.146, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.77, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.76, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.75, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.142, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.185, scheduling retry in 600.0 seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.144, scheduling retry in 512.0 seconds: Timed out connecting to 200.200.200.144
> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error attempting to reconnect to 200.200.200.77, scheduling retry in 512.0 seconds: Timed out connecting to 200.200.200.77
>
>
> And here is the exception I am having in the server:
>
> WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499 BatchStatement.java (line 223) Batch of prepared statements for [identification.entity_lookup, identification.entity] is of size 25216, exceeding specified threshold of 5120 by 20096.
> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611 ErrorMessage.java (line 222) Unexpected exception during request
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
> at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
> at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
> at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> Here is the amount of open files before and after I restart Cassandra:
>
> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
> 936580
> root@h:/etc/security/limits.d# sudo service cassandra restart
> [ ok ] Restarting Cassandra: cassandra.
> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
> 80295
>
>
> Best regards,
> Marcelo Valle.
>
Re: too many open files
Posted by Redmumba <re...@gmail.com>.
Just to chime in, I also ran into this issue when I was migrating to the
Datastax client. Instead of reusing the session, I was opening a new
session each time. For some reason, even though I was still closing the
session on the client side, I was getting the same error.
Plus, the only way I could recover was by restarting Cassandra. I did not
really see the connections timeout over a period of a few minutes.
Andrew
On Aug 8, 2014 3:19 PM, "Andrey Ilinykh" <ai...@gmail.com> wrote:
> You may have this problem if your client doesn't reuse the connection but
> opens new every type. So, run netstat and check the number of established
> connections. This number should not be big.
>
> Thank you,
> Andrey
>
>
> On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle <
> marcelo@s1mbi0se.com.br> wrote:
>
>> Hi,
>>
>> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having "too
>> many open files" exceptions when I try to perform a large number of
>> operations in my 10 node cluster.
>>
>> I saw the documentation
>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
>> and I have set everything to the recommended settings, but I keep getting
>> the errors.
>>
>> In the documentation it says: "Another, much less likely possibility, is
>> a file descriptor leak in Cassandra. Run lsof -n | grep java to check
>> that the number of file descriptors opened by Java is reasonable and
>> reports the error if the number is greater than a few thousand."
>>
>> I guess it's not the case, or else a lot of people would be complaining
>> about it, but I am not sure what I could do to solve the problem.
>>
>> Any hint about how to solve it?
>>
>> My client is written in python and uses Cassandra Python Driver. Here are
>> the exceptions I am having in the client:
>> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
>> seconds: Timed out connecting to 200.200.200.144
>> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
>> seconds: Timed out connecting to 200.200.200.77
>>
>>
>> And here is the exception I am having in the server:
>>
>> WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
>> BatchStatement.java (line 223) Batch of prepared statements for
>> [identification.entity_lookup, identification.entity] is of size 25216,
>> exceeding specified threshold of 5120 by 20096.
>> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
>> ErrorMessage.java (line 222) Unexpected exception during request
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Here is the amount of open files before and after I restart Cassandra:
>>
>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>> 936580
>> root@h:/etc/security/limits.d# sudo service cassandra restart
>> [ ok ] Restarting Cassandra: cassandra.
>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>> 80295
>>
>>
>> Best regards,
>> Marcelo Valle.
>>
>
>
Re: too many open files
Posted by "J. Ryan Earl" <os...@jryanearl.us>.
Yes, definitely look how many open files are actual file handles versus
networks sockets. We found a file handle leak in 2.0 but it was patched in
2.0.3 or .5 I think. A million open files is way too high.
On Fri, Aug 8, 2014 at 5:19 PM, Andrey Ilinykh <ai...@gmail.com> wrote:
> You may have this problem if your client doesn't reuse the connection but
> opens new every type. So, run netstat and check the number of established
> connections. This number should not be big.
>
> Thank you,
> Andrey
>
>
> On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle <
> marcelo@s1mbi0se.com.br> wrote:
>
>> Hi,
>>
>> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having "too
>> many open files" exceptions when I try to perform a large number of
>> operations in my 10 node cluster.
>>
>> I saw the documentation
>> http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
>> and I have set everything to the recommended settings, but I keep getting
>> the errors.
>>
>> In the documentation it says: "Another, much less likely possibility, is
>> a file descriptor leak in Cassandra. Run lsof -n | grep java to check
>> that the number of file descriptors opened by Java is reasonable and
>> reports the error if the number is greater than a few thousand."
>>
>> I guess it's not the case, or else a lot of people would be complaining
>> about it, but I am not sure what I could do to solve the problem.
>>
>> Any hint about how to solve it?
>>
>> My client is written in python and uses Cassandra Python Driver. Here are
>> the exceptions I am having in the client:
>> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
>> seconds: [Errno 24] Too many open files
>> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
>> seconds: Timed out connecting to 200.200.200.144
>> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
>> attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
>> seconds: Timed out connecting to 200.200.200.77
>>
>>
>> And here is the exception I am having in the server:
>>
>> WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
>> BatchStatement.java (line 223) Batch of prepared statements for
>> [identification.entity_lookup, identification.entity] is of size 25216,
>> exceeding specified threshold of 5120 by 20096.
>> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
>> ErrorMessage.java (line 222) Unexpected exception during request
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
>> at
>> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
>> at
>> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>>
>> Here is the amount of open files before and after I restart Cassandra:
>>
>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>> 936580
>> root@h:/etc/security/limits.d# sudo service cassandra restart
>> [ ok ] Restarting Cassandra: cassandra.
>> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
>> 80295
>>
>>
>> Best regards,
>> Marcelo Valle.
>>
>
>
Re: too many open files
Posted by Andrey Ilinykh <ai...@gmail.com>.
You may have this problem if your client doesn't reuse the connection but
opens new every type. So, run netstat and check the number of established
connections. This number should not be big.
Thank you,
Andrey
On Fri, Aug 8, 2014 at 12:35 PM, Marcelo Elias Del Valle <
marcelo@s1mbi0se.com.br> wrote:
> Hi,
>
> I am using Cassandra 2.0.9 running on Debian Wheezy, and I am having "too
> many open files" exceptions when I try to perform a large number of
> operations in my 10 node cluster.
>
> I saw the documentation
> http://www.datastax.com/documentation/cassandra/2.0/cassandra/troubleshooting/trblshootTooManyFiles_r.html
> and I have set everything to the recommended settings, but I keep getting
> the errors.
>
> In the documentation it says: "Another, much less likely possibility, is
> a file descriptor leak in Cassandra. Run lsof -n | grep java to check
> that the number of file descriptors opened by Java is reasonable and
> reports the error if the number is greater than a few thousand."
>
> I guess it's not the case, or else a lot of people would be complaining
> about it, but I am not sure what I could do to solve the problem.
>
> Any hint about how to solve it?
>
> My client is written in python and uses Cassandra Python Driver. Here are
> the exceptions I am having in the client:
> [s1log] 2014-08-08 12:16:09,631 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.151, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,632 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,633 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.143, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,634 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.145, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.144, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,635 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.148, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,732 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.146, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,733 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.77, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.76, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,734 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.75, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,735 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.142, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,736 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.185, scheduling retry in 600.0
> seconds: [Errno 24] Too many open files
> [s1log] 2014-08-08 12:16:09,942 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.144, scheduling retry in 512.0
> seconds: Timed out connecting to 200.200.200.144
> [s1log] 2014-08-08 12:16:09,998 - cassandra.pool - WARNING - Error
> attempting to reconnect to 200.200.200.77, scheduling retry in 512.0
> seconds: Timed out connecting to 200.200.200.77
>
>
> And here is the exception I am having in the server:
>
> WARN [Native-Transport-Requests:163] 2014-08-08 14:27:30,499
> BatchStatement.java (line 223) Batch of prepared statements for
> [identification.entity_lookup, identification.entity] is of size 25216,
> exceeding specified threshold of 5120 by 20096.
> ERROR [Native-Transport-Requests:150] 2014-08-08 14:27:31,611
> ErrorMessage.java (line 222) Unexpected exception during request
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:192)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
> at
> org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> Here is the amount of open files before and after I restart Cassandra:
>
> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
> 936580
> root@h:/etc/security/limits.d# sudo service cassandra restart
> [ ok ] Restarting Cassandra: cassandra.
> root@h:/etc/security/limits.d# lsof -n | grep java | wc -l
> 80295
>
>
> Best regards,
> Marcelo Valle.
>