You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mason Hale <ma...@onespot.com> on 2010/05/15 07:43:08 UTC
FileNotFoundException in ROW-READ-STAGE
One of our Cassandra nodes is throwing multiple of these errors:
ERROR [ROW-READ-STAGE:185] 2010-04-27 14:56:59,543 CassandraDaemon.java
(line 78) Fatal exception in thread Thread[ROW-READ-STAGE:185,5,main]
java.lang.RuntimeException: java.io.FileNotFoundException:
/data/cassandra/data/Onespot/Entry-28-Index.db (No such file or directory)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:118)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException:
/data/cassandra/data/Onespot/Entry-28-Index.db (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:143)
at
org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:138)
at org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:340)
at
org.apache.cassandra.io.SSTableReader.getFileDataInput(SSTableReader.java:482)
at
org.apache.cassandra.db.filter.SSTableNamesIterator.<init>(SSTableNamesIterator.java:50)
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:830)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
at org.apache.cassandra.db.Table.getRow(Table.java:381)
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:80)
... 4 more
I've checked, and the file in question is definitely not in that directory.
I'm not sure how to troubleshoot this issue, can anyone give me some
pointers?
thanks,
Mason
Re: FileNotFoundException in ROW-READ-STAGE
Posted by Jonathan Ellis <jb...@gmail.com>.
we really need to know what the file history is.
if it's a freshly flushed file, that's Bad.
if it's a file that was a compaction victim, that's annoying but not
something we need to worry about.
both flush and compaction are logged at INFO.
On Sun, May 16, 2010 at 11:13 AM, Mason Hale <ma...@onespot.com> wrote:
> Grepping through the logs, I haven't been able to deduce what was going on
> at the same time, but I have found many instances of these errors for
> multiple files going back for weeks nows. (Found 848 of these errors since
> 4-23-2010). The files not found include Index and Data files spanning
> multiple column families.
> Where should I go from here?
>
>
> On Sat, May 15, 2010 at 10:06 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> grep for Entry-28-Index.db in the logs, see what was going on before
>> the exceptions. (was it compacted?)
>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: FileNotFoundException in ROW-READ-STAGE
Posted by Mason Hale <ma...@onespot.com>.
Grepping through the logs, I haven't been able to deduce what was going on
at the same time, but I have found many instances of these errors for
multiple files going back for weeks nows. (Found 848 of these errors since
4-23-2010). The files not found include Index and Data files spanning
multiple column families.
Where should I go from here?
On Sat, May 15, 2010 at 10:06 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> grep for Entry-28-Index.db in the logs, see what was going on before
> the exceptions. (was it compacted?)
>
>
Re: FileNotFoundException in ROW-READ-STAGE
Posted by Jonathan Ellis <jb...@gmail.com>.
grep for Entry-28-Index.db in the logs, see what was going on before
the exceptions. (was it compacted?)
On Sat, May 15, 2010 at 12:43 AM, Mason Hale <ma...@onespot.com> wrote:
> One of our Cassandra nodes is throwing multiple of these errors:
> ERROR [ROW-READ-STAGE:185] 2010-04-27 14:56:59,543 CassandraDaemon.java
> (line 78) Fatal exception in thread Thread[ROW-READ-STAGE:185,5,main]
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /data/cassandra/data/Onespot/Entry-28-Index.db (No such file or directory)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:118)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.FileNotFoundException:
> /data/cassandra/data/Onespot/Entry-28-Index.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:98)
> at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:143)
> at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:138)
> at org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:340)
> at
> org.apache.cassandra.io.SSTableReader.getFileDataInput(SSTableReader.java:482)
> at
> org.apache.cassandra.db.filter.SSTableNamesIterator.<init>(SSTableNamesIterator.java:50)
> at
> org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:830)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
> at org.apache.cassandra.db.Table.getRow(Table.java:381)
> at
> org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:80)
> ... 4 more
> I've checked, and the file in question is definitely not in that directory.
> I'm not sure how to troubleshoot this issue, can anyone give me some
> pointers?
> thanks,
> Mason
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com