You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by David McNelis <dm...@gmail.com> on 2013/09/09 15:15:08 UTC
FSReadError
Morning,
I'm getting the following error (21 node cluster running 1.2.8)
FSReadError in /var/cassandra/data/et/http_request/ks-mycql3table-ic-1799-Data.db
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:93)
at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:312)
at org.apache.cassandra.io.sstable.SSTableScanner.seekTo(SSTableScanner.java:98)
at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:73)
at org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1424)
at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1476)
at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:46)
at org.apache.cassandra.service.RangeSliceVerbHandler.doVerb(RangeSliceVerbHandler.java:58)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException: Input/output error
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.FileDispatcherImpl.read(FileDispatcherImpl.java:46)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
at sun.nio.ch.IOUtil.read(IOUtil.java:198)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:143)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:108)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:85)
... 11 more
This starts during compaction and compaction ends up stuck, the node
eventually winds up locked up with this error repeating throughout the
logs, and requires a restart.
On restart I tried running a nodetool scrub against the offending
column family, which itself failed.
Any suggestions on taking care of this? Should I just delete the
sstable and repair? The node keeps starting compactions and running
into this after a period of time.
David
Re: FSReadError
Posted by David McNelis <dm...@gmail.com>.
Looks to be the case, getting an IO error when trying to cp the file. That
is unfortunate. On the bright side, now we at least have a more narrow
scope of the problem's source.
On Mon, Sep 9, 2013 at 12:54 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Mon, Sep 9, 2013 at 6:15 AM, David McNelis <dm...@gmail.com> wrote:
>
>> FSReadError in /var/cassandra/data/et/http_request/ks-mycql3table-ic-1799-Data.db
>>
>>
>
>> Any suggestions on taking care of this? Should I just delete the sstable and repair? The node keeps starting compactions and running into this after a period of time.
>>
>>
> Can you interact with the SSTable not-via-Cassandra?
>
> Given that FSReadError very lightly extends FSError which itself is a very
> light extension of the Java builtin IOError, my hunch is that you have disk
> or filesystem level issues.
>
> =Rob
>
>
Re: FSReadError
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Sep 9, 2013 at 6:15 AM, David McNelis <dm...@gmail.com> wrote:
> FSReadError in /var/cassandra/data/et/http_request/ks-mycql3table-ic-1799-Data.db
>
>
> Any suggestions on taking care of this? Should I just delete the sstable and repair? The node keeps starting compactions and running into this after a period of time.
>
>
Can you interact with the SSTable not-via-Cassandra?
Given that FSReadError very lightly extends FSError which itself is a very
light extension of the Java builtin IOError, my hunch is that you have disk
or filesystem level issues.
=Rob