You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by David McNelis <dm...@gmail.com> on 2013/09/09 15:15:08 UTC

FSReadError

Morning,

I'm getting the following error (21 node cluster running 1.2.8)

FSReadError in /var/cassandra/data/et/http_request/ks-mycql3table-ic-1799-Data.db
  at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:93)
	at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:312)
	at org.apache.cassandra.io.sstable.SSTableScanner.seekTo(SSTableScanner.java:98)
	at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:73)
	at org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1424)
	at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1476)
	at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:46)
	at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:58)
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Input/output error
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
	at sun.nio.ch.IOUtil.read(IOUtil.java:198)
	at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:143)
	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:108)
	at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:85)
	... 11 more

This starts during compaction and compaction ends up stuck, the node
eventually winds up locked up with this error repeating throughout the
logs, and requires a restart.

On restart I tried running a nodetool scrub against the offending
column family, which itself failed.

Any suggestions on taking care of this? Should I just delete the
sstable and repair? The node keeps starting compactions and running
into this after a period of time.


David

Re: FSReadError

Posted by David McNelis <dm...@gmail.com>.
Looks to be the case, getting an IO error when trying to cp the file.  That
is unfortunate.  On the bright side, now we at least have a more narrow
scope of the problem's source.


On Mon, Sep 9, 2013 at 12:54 PM, Robert Coli <rc...@eventbrite.com> wrote:

> On Mon, Sep 9, 2013 at 6:15 AM, David McNelis <dm...@gmail.com> wrote:
>
>> FSReadError in /var/cassandra/data/et/http_request/ks-mycql3table-ic-1799-Data.db
>>
>>
>
>> Any suggestions on taking care of this? Should I just delete the sstable and repair? The node keeps starting compactions and running into this after a period of time.
>>
>>
> Can you interact with the SSTable not-via-Cassandra?
>
> Given that FSReadError very lightly extends FSError which itself is a very
> light extension of the Java builtin IOError, my hunch is that you have disk
> or filesystem level issues.
>
> =Rob
>
>

Re: FSReadError

Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Sep 9, 2013 at 6:15 AM, David McNelis <dm...@gmail.com> wrote:

> FSReadError in /var/cassandra/data/et/http_request/ks-mycql3table-ic-1799-Data.db
>
>

> Any suggestions on taking care of this? Should I just delete the sstable and repair? The node keeps starting compactions and running into this after a period of time.
>
>
Can you interact with the SSTable not-via-Cassandra?

Given that FSReadError very lightly extends FSError which itself is a very
light extension of the Java builtin IOError, my hunch is that you have disk
or filesystem level issues.

=Rob