You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Dr. Martin Grabmüller" <Ma...@eleven.de> on 2010/01/22 11:27:09 UTC

Too many open files

Hello all,

I am using Cassandra for storing mail data, and after
filling my test installation with data over the night,
I got "Too many open files" errors.  I checked JIRA for
known bugs, but found nothing that matched my setup.

I am using version 0.5.0 from http://people.apache.org/~eevans/
and the exception looks like this:

ERROR - Fatal exception in thread Thread[ROW-READ-STAGE:33,5,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/MessageIndex-844-Data.db (Too many open files)
        at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:117)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/MessageIndex-844-Data.db (Too many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
        at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:142)
        at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:123)
        at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:58)
        at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamilyInternal(ColumnFamilyStore.java:1245)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1203)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1172)
        at org.apache.cassandra.db.Table.getRow(Table.java:422)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
        at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:79)
        ... 4 more

As can be seen from the trace, the exception happens when trying to
read from Cassandra.

I have 789 data files in my data directory (+ the same number
of index and filter files.

The obvious call to lsof did not give me any insight (with 2271 being
my Cassandra instance's pid):

  (env)cassandra@archive00001:~$ lsof -p 2271|wc -l
  101

Maybe the file limit is reached while scanning all data files?

Nevertheless, when working with a development snapshot yesterday,
before upgrading, Cassandra was happily handling more than 8000
data files in its data directory.  That was when I was using snapshot
apache-cassandra-incubating-2010-01-06_12-47-49 from the snapshot
download page.

I am looking for advice on how to debug this.

Thanks,
  Martin

RE: Too many open files

Posted by "Dr. Martin Grabmüller" <Ma...@eleven.de>.
 

> -----Original Message-----
> From: Jonathan Ellis [mailto:jbellis@gmail.com] 
> Sent: Friday, January 22, 2010 2:30 PM
> To: cassandra-user@incubator.apache.org
> Subject: Re: Too many open files
> 
> increase the number of files you allow cassandra to open

Thanks, that works.  That means that Cassandra needs many file
descriptors when there are many data files, right?  That's something
one has to keep in mind.

Martin

Re: Too many open files

Posted by Jonathan Ellis <jb...@gmail.com>.
increase the number of files you allow cassandra to open

On Fri, Jan 22, 2010 at 4:27 AM, Dr. Martin Grabmüller
<Ma...@eleven.de> wrote:
> Hello all,
>
> I am using Cassandra for storing mail data, and after
> filling my test installation with data over the night,
> I got "Too many open files" errors.  I checked JIRA for
> known bugs, but found nothing that matched my setup.
>
> I am using version 0.5.0 from http://people.apache.org/~eevans/
> and the exception looks like this:
>
> ERROR - Fatal exception in thread Thread[ROW-READ-STAGE:33,5,main]
> java.lang.RuntimeException: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/MessageIndex-844-Data.db (Too many open files)
>        at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:117)
>        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:38)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>        at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.FileNotFoundException: /mnt/data000/cassandra/data/Archive/MessageIndex-844-Data.db (Too many open files)
>        at java.io.RandomAccessFile.open(Native Method)
>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>        at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
>        at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:142)
>        at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:123)
>        at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:58)
>        at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
>        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamilyInternal(ColumnFamilyStore.java:1245)
>        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1203)
>        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1172)
>        at org.apache.cassandra.db.Table.getRow(Table.java:422)
>        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
>        at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:79)
>        ... 4 more
>
> As can be seen from the trace, the exception happens when trying to
> read from Cassandra.
>
> I have 789 data files in my data directory (+ the same number
> of index and filter files.
>
> The obvious call to lsof did not give me any insight (with 2271 being
> my Cassandra instance's pid):
>
>  (env)cassandra@archive00001:~$ lsof -p 2271|wc -l
>  101
>
> Maybe the file limit is reached while scanning all data files?
>
> Nevertheless, when working with a development snapshot yesterday,
> before upgrading, Cassandra was happily handling more than 8000
> data files in its data directory.  That was when I was using snapshot
> apache-cassandra-incubating-2010-01-06_12-47-49 from the snapshot
> download page.
>
> I am looking for advice on how to debug this.
>
> Thanks,
>  Martin
>

RE: Too many open files

Posted by "Dr. Martin Grabmüller" <Ma...@eleven.de>.
Hi all, 

another thing I forgot in my first post:

Thank you all for the great work!  Cassandra is an amazing piece of
software.

Martin

Re: Too many open files

Posted by Ted Zlatanov <tz...@lifelogs.com>.
On Fri, 22 Jan 2010 11:27:09 +0100 Dr. Martin Grabmüller <Ma...@eleven.de> wrote: 

MG> The obvious call to lsof did not give me any insight (with 2271 being
MG> my Cassandra instance's pid):

MG>   (env)cassandra@archive00001:~$ lsof -p 2271|wc -l
MG>   101

MG> Maybe the file limit is reached while scanning all data files?

Looks like it, 100 is a nice round number (you're counting the header
too to get 101).  What does `ulimit -a' say?  You need to adjust it but
the way depends on the way your specific OS distro configures login.

Ted

RE: Too many open files

Posted by "Dr. Martin Grabmüller" <Ma...@eleven.de>.
Sorry, I forgot the following:

I am using a single node, replication factor 1, random partitioner,
and I am doing a multiget_slice with 10 keys.  Yesterday, in the old
setup, I only used get_slice [1].

Maybe Cassandra opens files in parallel for all the keys?

Thanks,
  Martin

[1] The change to version 0.5.0 (pre) was due to that change, because
    of a NPE in the older version for multiget_slice.

> -----Original Message-----
> From: Dr. Martin Grabmüller [mailto:Martin.Grabmueller@eleven.de] 
> Sent: Friday, January 22, 2010 11:27 AM
> To: cassandra-user@incubator.apache.org
> Subject: Too many open files
> 
> Hello all,
> 
> I am using Cassandra for storing mail data, and after
> filling my test installation with data over the night,
> I got "Too many open files" errors.  I checked JIRA for
> known bugs, but found nothing that matched my setup.
> 
> I am using version 0.5.0 from http://people.apache.org/~eevans/
> and the exception looks like this:
> 
> ERROR - Fatal exception in thread Thread[ROW-READ-STAGE:33,5,main]
> java.lang.RuntimeException: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/MessageIndex-844-Data.db 
> (Too many open files)
>         at 
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler
> .java:117)
>         at 
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDelive
> ryTask.java:38)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolEx
> ecutor.java:1110)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE
> xecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.FileNotFoundException: 
> /mnt/data000/cassandra/data/Archive/MessageIndex-844-Data.db 
> (Too many open files)
>         at java.io.RandomAccessFile.open(Native Method)
>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
>         at java.io.RandomAccessFile.<init>(RandomAccessFile.java:118)
>         at 
> org.apache.cassandra.io.BufferedRandomAccessFile.<init>(Buffer
> edRandomAccessFile.java:142)
>         at 
> org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGrou
> pReader.<init>(SSTableSliceIterator.java:123)
>         at 
> org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SST
> ableSliceIterator.java:58)
>         at 
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColu
> mnIterator(SliceQueryFilter.java:63)
>         at 
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamilyInter
> nal(ColumnFamilyStore.java:1245)
>         at 
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(Colu
> mnFamilyStore.java:1203)
>         at 
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(Colu
> mnFamilyStore.java:1172)
>         at org.apache.cassandra.db.Table.getRow(Table.java:422)
>         at 
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromR
> eadCommand.java:59)
>         at 
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler
> .java:79)
>         ... 4 more
> 
> As can be seen from the trace, the exception happens when trying to
> read from Cassandra.
> 
> I have 789 data files in my data directory (+ the same number
> of index and filter files.
> 
> The obvious call to lsof did not give me any insight (with 2271 being
> my Cassandra instance's pid):
> 
>   (env)cassandra@archive00001:~$ lsof -p 2271|wc -l
>   101
> 
> Maybe the file limit is reached while scanning all data files?
> 
> Nevertheless, when working with a development snapshot yesterday,
> before upgrading, Cassandra was happily handling more than 8000
> data files in its data directory.  That was when I was using snapshot
> apache-cassandra-incubating-2010-01-06_12-47-49 from the snapshot
> download page.
> 
> I am looking for advice on how to debug this.
> 
> Thanks,
>   Martin
>