You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Ian Eure (JIRA)" <ji...@apache.org> on 2009/07/08 23:35:14 UTC
[jira] Created: (CASSANDRA-283) Cassandra leaks FDs
Cassandra leaks FDs
-------------------
Key: CASSANDRA-283
URL: https://issues.apache.org/jira/browse/CASSANDRA-283
Project: Cassandra
Issue Type: Bug
Environment: Debian Etch, J2SE 1.6.0
Reporter: Ian Eure
Cassandra leaks file descriptors like crazy. I started getting these errors after a few hours of uptime:
java.lang.RuntimeException: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:84)
at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:181)
at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:859)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:141)
at org.apache.cassandra.io.SequenceFile$BufferReader.init(SequenceFile.java:811)
at org.apache.cassandra.io.SequenceFile$Reader.<init>(SequenceFile.java:743)
at org.apache.cassandra.io.SequenceFile$BufferReader.<init>(SequenceFile.java:805)
at org.apache.cassandra.io.SequenceFile$ColumnGroupReader.<init>(SequenceFile.java:248)
at org.apache.cassandra.io.SSTableReader.getColumnGroupReader(SSTableReader.java:346)
at org.apache.cassandra.db.SSTableColumnIterator.<init>(ColumnIterator.java:61)
at org.apache.cassandra.db.ColumnFamilyStore.getSliceFrom(ColumnFamilyStore.java:1589)
at org.apache.cassandra.db.Table.getRow(Table.java:596)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:60)
at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:600)
at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:303)
at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:80)
I have an open file limit of 1024. Examining the lsof output for Cassandra shows 975 FDs for the same file: /var/cassandra/data/Digg-Items-2-Data.db
Clearly, these FDs are leaking somewhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-283) Cassandra leaks FDs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729993#action_12729993 ]
Jonathan Ellis commented on CASSANDRA-283:
------------------------------------------
CASSANDRA-287 is committed.
Ian, do you still see leaking?
> Cassandra leaks FDs
> -------------------
>
> Key: CASSANDRA-283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-283
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Etch, J2SE 1.6.0
> Reporter: Ian Eure
>
> Cassandra leaks file descriptors like crazy. I started getting these errors after a few hours of uptime:
> java.lang.RuntimeException: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:84)
> at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:181)
> at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:859)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:141)
> at org.apache.cassandra.io.SequenceFile$BufferReader.init(SequenceFile.java:811)
> at org.apache.cassandra.io.SequenceFile$Reader.<init>(SequenceFile.java:743)
> at org.apache.cassandra.io.SequenceFile$BufferReader.<init>(SequenceFile.java:805)
> at org.apache.cassandra.io.SequenceFile$ColumnGroupReader.<init>(SequenceFile.java:248)
> at org.apache.cassandra.io.SSTableReader.getColumnGroupReader(SSTableReader.java:346)
> at org.apache.cassandra.db.SSTableColumnIterator.<init>(ColumnIterator.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getSliceFrom(ColumnFamilyStore.java:1589)
> at org.apache.cassandra.db.Table.getRow(Table.java:596)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:60)
> at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:600)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:303)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:80)
> I have an open file limit of 1024. Examining the lsof output for Cassandra shows 975 FDs for the same file: /var/cassandra/data/Digg-Items-2-Data.db
> Clearly, these FDs are leaking somewhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-283) Cassandra leaks FDs
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729676#action_12729676 ]
Jun Rao commented on CASSANDRA-283:
-----------------------------------
There exists code in get_slice() that closes all files it opens at the end. Could you describe your scenario a bit more? Are you just issuing get_slice calls? How many requests were issued?
> Cassandra leaks FDs
> -------------------
>
> Key: CASSANDRA-283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-283
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Etch, J2SE 1.6.0
> Reporter: Ian Eure
>
> Cassandra leaks file descriptors like crazy. I started getting these errors after a few hours of uptime:
> java.lang.RuntimeException: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:84)
> at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:181)
> at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:859)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:141)
> at org.apache.cassandra.io.SequenceFile$BufferReader.init(SequenceFile.java:811)
> at org.apache.cassandra.io.SequenceFile$Reader.<init>(SequenceFile.java:743)
> at org.apache.cassandra.io.SequenceFile$BufferReader.<init>(SequenceFile.java:805)
> at org.apache.cassandra.io.SequenceFile$ColumnGroupReader.<init>(SequenceFile.java:248)
> at org.apache.cassandra.io.SSTableReader.getColumnGroupReader(SSTableReader.java:346)
> at org.apache.cassandra.db.SSTableColumnIterator.<init>(ColumnIterator.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getSliceFrom(ColumnFamilyStore.java:1589)
> at org.apache.cassandra.db.Table.getRow(Table.java:596)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:60)
> at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:600)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:303)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:80)
> I have an open file limit of 1024. Examining the lsof output for Cassandra shows 975 FDs for the same file: /var/cassandra/data/Digg-Items-2-Data.db
> Clearly, these FDs are leaking somewhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-283) Cassandra leaks FDs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729682#action_12729682 ]
Jonathan Ellis commented on CASSANDRA-283:
------------------------------------------
Actually now that I think of it I did fix a FD leak in 287 already. In the code
iter = filter.getSSTableColumnIterator(sstable);
if (iter.hasNext())
{
returnCF.delete(iter.getColumnFamily());
iterators.add(iter);
}
else
{
iter.close();
}
the "else" clause didn't exist before, and since it wasn't added to the list of iterators it wouldn't get closed at the end.
> Cassandra leaks FDs
> -------------------
>
> Key: CASSANDRA-283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-283
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Etch, J2SE 1.6.0
> Reporter: Ian Eure
>
> Cassandra leaks file descriptors like crazy. I started getting these errors after a few hours of uptime:
> java.lang.RuntimeException: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:84)
> at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:181)
> at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:859)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:141)
> at org.apache.cassandra.io.SequenceFile$BufferReader.init(SequenceFile.java:811)
> at org.apache.cassandra.io.SequenceFile$Reader.<init>(SequenceFile.java:743)
> at org.apache.cassandra.io.SequenceFile$BufferReader.<init>(SequenceFile.java:805)
> at org.apache.cassandra.io.SequenceFile$ColumnGroupReader.<init>(SequenceFile.java:248)
> at org.apache.cassandra.io.SSTableReader.getColumnGroupReader(SSTableReader.java:346)
> at org.apache.cassandra.db.SSTableColumnIterator.<init>(ColumnIterator.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getSliceFrom(ColumnFamilyStore.java:1589)
> at org.apache.cassandra.db.Table.getRow(Table.java:596)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:60)
> at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:600)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:303)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:80)
> I have an open file limit of 1024. Examining the lsof output for Cassandra shows 975 FDs for the same file: /var/cassandra/data/Digg-Items-2-Data.db
> Clearly, these FDs are leaking somewhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-283) Cassandra leaks FDs
Posted by "Michael Greene (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Greene updated CASSANDRA-283:
-------------------------------------
Component/s: Core
> Cassandra leaks FDs
> -------------------
>
> Key: CASSANDRA-283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Debian Etch, J2SE 1.6.0
> Reporter: Ian Eure
> Assignee: Jonathan Ellis
> Fix For: 0.4
>
>
> Cassandra leaks file descriptors like crazy. I started getting these errors after a few hours of uptime:
> java.lang.RuntimeException: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:84)
> at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:181)
> at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:859)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:141)
> at org.apache.cassandra.io.SequenceFile$BufferReader.init(SequenceFile.java:811)
> at org.apache.cassandra.io.SequenceFile$Reader.<init>(SequenceFile.java:743)
> at org.apache.cassandra.io.SequenceFile$BufferReader.<init>(SequenceFile.java:805)
> at org.apache.cassandra.io.SequenceFile$ColumnGroupReader.<init>(SequenceFile.java:248)
> at org.apache.cassandra.io.SSTableReader.getColumnGroupReader(SSTableReader.java:346)
> at org.apache.cassandra.db.SSTableColumnIterator.<init>(ColumnIterator.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getSliceFrom(ColumnFamilyStore.java:1589)
> at org.apache.cassandra.db.Table.getRow(Table.java:596)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:60)
> at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:600)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:303)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:80)
> I have an open file limit of 1024. Examining the lsof output for Cassandra shows 975 FDs for the same file: /var/cassandra/data/Digg-Items-2-Data.db
> Clearly, these FDs are leaking somewhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-283) Cassandra leaks FDs
Posted by "Ian Eure (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ian Eure updated CASSANDRA-283:
-------------------------------
We're not seeing this right now, though I suspect the issue still exists somewhere.
We determined that the problem was caused because Cassandra corrupted it's data files in some way which caused it to leak FDs when calling get_slice. After removing the data files, we couldn't reproduce the issue anymore.
So there's the issue with the initial corruption, which we haven't reproduced, and the issue with the FDs, which we might be able to reproduce at some point if the corruption can be reproduced.
> Cassandra leaks FDs
> -------------------
>
> Key: CASSANDRA-283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-283
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Etch, J2SE 1.6.0
> Reporter: Ian Eure
>
> Cassandra leaks file descriptors like crazy. I started getting these errors after a few hours of uptime:
> java.lang.RuntimeException: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:84)
> at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:181)
> at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:859)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:141)
> at org.apache.cassandra.io.SequenceFile$BufferReader.init(SequenceFile.java:811)
> at org.apache.cassandra.io.SequenceFile$Reader.<init>(SequenceFile.java:743)
> at org.apache.cassandra.io.SequenceFile$BufferReader.<init>(SequenceFile.java:805)
> at org.apache.cassandra.io.SequenceFile$ColumnGroupReader.<init>(SequenceFile.java:248)
> at org.apache.cassandra.io.SSTableReader.getColumnGroupReader(SSTableReader.java:346)
> at org.apache.cassandra.db.SSTableColumnIterator.<init>(ColumnIterator.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getSliceFrom(ColumnFamilyStore.java:1589)
> at org.apache.cassandra.db.Table.getRow(Table.java:596)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:60)
> at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:600)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:303)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:80)
> I have an open file limit of 1024. Examining the lsof output for Cassandra shows 975 FDs for the same file: /var/cassandra/data/Digg-Items-2-Data.db
> Clearly, these FDs are leaking somewhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-283) Cassandra leaks FDs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-283:
----------------------------------------
Assignee: Jonathan Ellis
> Cassandra leaks FDs
> -------------------
>
> Key: CASSANDRA-283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-283
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Etch, J2SE 1.6.0
> Reporter: Ian Eure
> Assignee: Jonathan Ellis
>
> Cassandra leaks file descriptors like crazy. I started getting these errors after a few hours of uptime:
> java.lang.RuntimeException: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:84)
> at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:181)
> at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:859)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:141)
> at org.apache.cassandra.io.SequenceFile$BufferReader.init(SequenceFile.java:811)
> at org.apache.cassandra.io.SequenceFile$Reader.<init>(SequenceFile.java:743)
> at org.apache.cassandra.io.SequenceFile$BufferReader.<init>(SequenceFile.java:805)
> at org.apache.cassandra.io.SequenceFile$ColumnGroupReader.<init>(SequenceFile.java:248)
> at org.apache.cassandra.io.SSTableReader.getColumnGroupReader(SSTableReader.java:346)
> at org.apache.cassandra.db.SSTableColumnIterator.<init>(ColumnIterator.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getSliceFrom(ColumnFamilyStore.java:1589)
> at org.apache.cassandra.db.Table.getRow(Table.java:596)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:60)
> at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:600)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:303)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:80)
> I have an open file limit of 1024. Examining the lsof output for Cassandra shows 975 FDs for the same file: /var/cassandra/data/Digg-Items-2-Data.db
> Clearly, these FDs are leaking somewhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-283) Cassandra leaks FDs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12729680#action_12729680 ]
Jonathan Ellis commented on CASSANDRA-283:
------------------------------------------
Let's re-test after committing CASSANDRA-287 so there's only one code path to debug. :)
> Cassandra leaks FDs
> -------------------
>
> Key: CASSANDRA-283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-283
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Etch, J2SE 1.6.0
> Reporter: Ian Eure
>
> Cassandra leaks file descriptors like crazy. I started getting these errors after a few hours of uptime:
> java.lang.RuntimeException: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:84)
> at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:181)
> at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:859)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:141)
> at org.apache.cassandra.io.SequenceFile$BufferReader.init(SequenceFile.java:811)
> at org.apache.cassandra.io.SequenceFile$Reader.<init>(SequenceFile.java:743)
> at org.apache.cassandra.io.SequenceFile$BufferReader.<init>(SequenceFile.java:805)
> at org.apache.cassandra.io.SequenceFile$ColumnGroupReader.<init>(SequenceFile.java:248)
> at org.apache.cassandra.io.SSTableReader.getColumnGroupReader(SSTableReader.java:346)
> at org.apache.cassandra.db.SSTableColumnIterator.<init>(ColumnIterator.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getSliceFrom(ColumnFamilyStore.java:1589)
> at org.apache.cassandra.db.Table.getRow(Table.java:596)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:60)
> at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:600)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:303)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:80)
> I have an open file limit of 1024. Examining the lsof output for Cassandra shows 975 FDs for the same file: /var/cassandra/data/Digg-Items-2-Data.db
> Clearly, these FDs are leaking somewhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-283) Cassandra leaks FDs
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-283.
--------------------------------------
Resolution: Fixed
I'm going to close this then because CASSANDRA-287 fixes the most obvious FD leak. Feel free to reopen if you have any more problems.
(Did you save a copy of the bad sstable files?)
> Cassandra leaks FDs
> -------------------
>
> Key: CASSANDRA-283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-283
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Etch, J2SE 1.6.0
> Reporter: Ian Eure
>
> Cassandra leaks file descriptors like crazy. I started getting these errors after a few hours of uptime:
> java.lang.RuntimeException: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:84)
> at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:181)
> at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:859)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:817)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException: /var/cassandra/data/Digg-Items-2-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> at org.apache.cassandra.io.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:141)
> at org.apache.cassandra.io.SequenceFile$BufferReader.init(SequenceFile.java:811)
> at org.apache.cassandra.io.SequenceFile$Reader.<init>(SequenceFile.java:743)
> at org.apache.cassandra.io.SequenceFile$BufferReader.<init>(SequenceFile.java:805)
> at org.apache.cassandra.io.SequenceFile$ColumnGroupReader.<init>(SequenceFile.java:248)
> at org.apache.cassandra.io.SSTableReader.getColumnGroupReader(SSTableReader.java:346)
> at org.apache.cassandra.db.SSTableColumnIterator.<init>(ColumnIterator.java:61)
> at org.apache.cassandra.db.ColumnFamilyStore.getSliceFrom(ColumnFamilyStore.java:1589)
> at org.apache.cassandra.db.Table.getRow(Table.java:596)
> at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:60)
> at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:600)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:303)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:80)
> I have an open file limit of 1024. Examining the lsof output for Cassandra shows 975 FDs for the same file: /var/cassandra/data/Digg-Items-2-Data.db
> Clearly, these FDs are leaking somewhere.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.