You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Dominic Williams (JIRA)" <ji...@apache.org> on 2011/06/18 16:43:47 UTC
[jira] [Created] (CASSANDRA-2794) Scrub fails on system with
blocked compaction with java.io.FileNotFoundException / Too many open files
Scrub fails on system with blocked compaction with java.io.FileNotFoundException / Too many open files
------------------------------------------------------------------------------------------------------
Key: CASSANDRA-2794
URL: https://issues.apache.org/jira/browse/CASSANDRA-2794
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.6
Environment: Ubuntu
Reporter: Dominic Williams
Fix For: 0.7.7
A node is suffering from CASSANDRA-2793. Scrub is run to try and fix the problem. Although ulimit shows unlimited file handles allowed scrub eventually fails with java.io.FileNotFoundException / Too many open files:
INFO [CompactionExecutor:1] 2011-06-17 00:46:26,115 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7018-Data.db')
INFO [CompactionExecutor:1] 2011-06-17 00:46:26,225 CompactionManager.java (line 652) Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7018-Data.db') complete: 275 rows in new sstable and 0 empty (tombstoned) rows dropped
INFO [CompactionExecutor:1] 2011-06-17 00:46:26,226 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7580-Data.db')
INFO [CompactionExecutor:1] 2011-06-17 00:46:28,383 CompactionManager.java (line 652) Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7580-Data.db') complete: 297 rows in new sstable and 0 empty (tombstoned) rows dropped
INFO [CompactionExecutor:1] 2011-06-17 00:46:28,384 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7574-Data.db')
INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 CompactionManager.java (line 652) Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7574-Data.db') complete: 347 rows in new sstable and 0 empty (tombstoned) rows dropped
INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7010-Data.db')
ERROR [CompactionExecutor:1] 2011-06-17 00:46:29,374 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/UserMonsters-tmp-f-7823-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78)
at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2243)
at org.apache.cassandra.db.CompactionManager.maybeCreateWriter(CompactionManager.java:794)
at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:534)
at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
This error eventually "takes over":
ERROR [ReadStage:328] 2011-06-17 00:46:38,634 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[ReadStage:328,5,main]
java.io.IOError: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3466-Data.db (Too many open files)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78)
at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:553)
at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:95)
at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1442)
at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:49)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3466-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:108)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:93)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:74)
... 8 more
ERROR [ReadStage:343] 2011-06-17 00:46:48,624 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[ReadStage:343,5,main]
java.io.IOError: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3468-Data.db (Too many open files)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78)
at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:553)
at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:95)
at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1442)
at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:49)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3468-Data.db (Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:108)
at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:93)
at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:74)
... 8 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2794) Scrub fails on system with
blocked compaction with java.io.FileNotFoundException / Too many open files
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2794.
---------------------------------------
Resolution: Duplicate
Fix Version/s: (was: 0.7.7)
scrub leaking file handles was fixed for 0.7.7 in CASSANDRA-2669
> Scrub fails on system with blocked compaction with java.io.FileNotFoundException / Too many open files
> ------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2794
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2794
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.6
> Environment: Ubuntu
> Reporter: Dominic Williams
>
> A node is suffering from CASSANDRA-2793. Scrub is run to try and fix the problem. Although ulimit shows unlimited file handles allowed scrub eventually fails with java.io.FileNotFoundException / Too many open files:
> INFO [CompactionExecutor:1] 2011-06-17 00:46:26,115 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7018-Data.db')
> INFO [CompactionExecutor:1] 2011-06-17 00:46:26,225 CompactionManager.java (line 652) Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7018-Data.db') complete: 275 rows in new sstable and 0 empty (tombstoned) rows dropped
> INFO [CompactionExecutor:1] 2011-06-17 00:46:26,226 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7580-Data.db')
> INFO [CompactionExecutor:1] 2011-06-17 00:46:28,383 CompactionManager.java (line 652) Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7580-Data.db') complete: 297 rows in new sstable and 0 empty (tombstoned) rows dropped
> INFO [CompactionExecutor:1] 2011-06-17 00:46:28,384 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7574-Data.db')
> INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 CompactionManager.java (line 652) Scrub of SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7574-Data.db') complete: 347 rows in new sstable and 0 empty (tombstoned) rows dropped
> INFO [CompactionExecutor:1] 2011-06-17 00:46:29,300 CompactionManager.java (line 511) Scrubbing SSTableReader(path='/var/opt/cassandra/data/FightMyMonster/UserMonsters-f-7010-Data.db')
> ERROR [CompactionExecutor:1] 2011-06-17 00:46:29,374 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/UserMonsters-tmp-f-7823-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
> at org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:78)
> at org.apache.cassandra.db.ColumnFamilyStore.createCompactionWriter(ColumnFamilyStore.java:2243)
> at org.apache.cassandra.db.CompactionManager.maybeCreateWriter(CompactionManager.java:794)
> at org.apache.cassandra.db.CompactionManager.doScrub(CompactionManager.java:534)
> at org.apache.cassandra.db.CompactionManager.access$600(CompactionManager.java:56)
> at org.apache.cassandra.db.CompactionManager$3.call(CompactionManager.java:195)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> This error eventually "takes over":
> ERROR [ReadStage:328] 2011-06-17 00:46:38,634 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[ReadStage:328,5,main]
> java.io.IOError: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3466-Data.db (Too many open files)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78)
> at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:553)
> at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:95)
> at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1442)
> at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:49)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3466-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
> at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:108)
> at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:93)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:74)
> ... 8 more
> ERROR [ReadStage:343] 2011-06-17 00:46:48,624 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[ReadStage:343,5,main]
> java.io.IOError: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3468-Data.db (Too many open files)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:78)
> at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReader.java:553)
> at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:95)
> at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1442)
> at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:49)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException: /var/opt/cassandra/data/FightMyMonster/MonsterMarket_3-f-3468-Data.db (Too many open files)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:113)
> at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:108)
> at org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:93)
> at org.apache.cassandra.io.sstable.SSTableScanner.<init>(SSTableScanner.java:74)
> ... 8 more
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira