[jira] [Commented] (CASSANDRA-1248) ArrayIndexOutOfBoundsException thrown when reading partially-written sstable

["Patricio Echague (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050076#comment-13050076 ] 

Patricio Echague commented on CASSANDRA-1248:
---------------------------------------------

David, just out of curiosity. Are commits logs and Data partition all in the same volume?

if not, what partition ran out of space?

> ArrayIndexOutOfBoundsException thrown when reading partially-written sstable
> ----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-1248
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1248
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Cassandra 0.6.3
> java version "1.6.0_0"
> IcedTea6 1.3.1 (6b12-0ubuntu6.6) Runtime Environment (build 1.6.0_0-b12)
> OpenJDK 64-Bit Server VM (build 1.6.0_0-b12, mixed mode)
> Ubuntu 8.10
> 6 node cluster
> RF==3
>            Reporter: David King
>            Assignee: Patricio Echague
>             Fix For: 0.8.2
>
>
> A node died by running out of disk space. After growing the volume, trying to execute reads dies with:
> {quote}
> ERROR 11:02:51,957 Uncaught exception in thread Thread[ROW-READ-STAGE:10,5,main]
> java.lang.ArrayIndexOutOfBoundsException
>         at org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
>         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
>         at java.io.DataInputStream.readUTF(DataInputStream.java:609)
>         at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:938)
>         at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
>         at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
>         at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
>         at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859)
>         at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:747)
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:772)
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739)
>         at org.apache.cassandra.db.Table.getRow(Table.java:381)
>         at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
>         at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> {quote}
> The clients then die with TimeoutExceptions.
> It might be better to return an exception to the client immediately instead of letting them die, and it would be helpful if this exception were accompanied with a message indicating *which* SStable is malformed. 
> This command does *not* turn up the failed table (nor does the same with sstablekeys):
> {quote}
> ri@pmc08:/cassandra/data$ for x in $(ls -t */*-Data.db); do ls -lh $x; /usr/local/cassandra/bin/sstable2json $x > /dev/null; done
> {quote}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira