You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "David King (JIRA)" <ji...@apache.org> on 2010/07/05 21:02:49 UTC
[jira] Created: (CASSANDRA-1248) ArrayIndexOutOfBoundsException
thrown when reading partially-written sstable
ArrayIndexOutOfBoundsException thrown when reading partially-written sstable
----------------------------------------------------------------------------
Key: CASSANDRA-1248
URL: https://issues.apache.org/jira/browse/CASSANDRA-1248
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.6.3
Environment: Cassandra 0.6.3
java version "1.6.0_0"
IcedTea6 1.3.1 (6b12-0ubuntu6.6) Runtime Environment (build 1.6.0_0-b12)
OpenJDK 64-Bit Server VM (build 1.6.0_0-b12, mixed mode)
Ubuntu 8.10
6 node cluster
RF==3
Reporter: David King
A node died by running out of disk space. After growing the volume, trying to execute reads dies with:
{quote}
ERROR 11:02:51,957 Uncaught exception in thread Thread[ROW-READ-STAGE:10,5,main]
java.lang.ArrayIndexOutOfBoundsException
at org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
at java.io.DataInputStream.readUTF(DataInputStream.java:609)
at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:938)
at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859)
at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:747)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:772)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739)
at org.apache.cassandra.db.Table.getRow(Table.java:381)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
{quote}
The clients then die with TimeoutExceptions.
It might be better to return an exception to the client immediately instead of letting them die, and it would be helpful if this exception were accompanied with a message indicating *which* SStable is malformed.
This command does *not* turn up the failed table (nor does the same with sstablekeys):
{quote}
ri@pmc08:/cassandra/data$ for x in $(ls -t */*-Data.db); do ls -lh $x; /usr/local/cassandra/bin/sstable2json $x > /dev/null; done
{quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1248) ArrayIndexOutOfBoundsException
thrown when reading partially-written sstable
Posted by "David King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885316#action_12885316 ]
David King commented on CASSANDRA-1248:
---------------------------------------
Not sure how much of this is helpful, but here are the sorrounding logs with DEBUG logging:
{quote}
DEBUG [ROW-READ-STAGE:185] 2010-07-05 12:21:00,642 ReadVerbHandler.java (line 75) digest is d41d8cd98f00b204e9800998ecf8427e
DEBUG [ROW-READ-STAGE:185] 2010-07-05 12:21:00,672 ReadVerbHandler.java (line 93) Read key sr_pop_en-gb_no_over18; sending response to 14264920@/10.251.82.223
DEBUG [ROW-MUTATION-STAGE:8] 2010-07-05 12:21:00,642 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='reddit', key='3qtik_c0tjtwf', modifications=[ColumnFamily(CommentVote [date:false:13@1278355826446131,ip:false:13@1278355826446131,name:false:1@1278355826446131,thing1_id:false:5@1278355826446131,thing2_id:false:7@1278355826446131,valid_thing:false:1@1278355826446131,valid_user:false:1@1278355826446131,])])
DEBUG [ROW-MUTATION-STAGE:13] 2010-07-05 12:21:00,670 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='permacache', key='04a594c18be3804bf2ad140a18292aaef76f8641', modifications=[ColumnFamily(permacache [76616c7565:false:45@1278357660440485,])])
DEBUG [ROW-MUTATION-STAGE:13] 2010-07-05 12:21:00,673 RowMutationVerbHandler.java (line 73) RowMutation(table='permacache', key='04a594c18be3804bf2ad140a18292aaef76f8641', modifications=[ColumnFamily(permacache [76616c7565:false:45@1278357660440485,])]) applied. Sending response to 2890086@/10.250.138.47
DEBUG [ROW-MUTATION-STAGE:6] 2010-07-05 12:21:00,669 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='permacache', key='comments_sort_21188087__confidence', modifications=[ColumnFamily(permacache [76616c7565:false:1128@1278357660437987,])])
DEBUG [ROW-MUTATION-STAGE:6] 2010-07-05 12:21:00,673 RowMutationVerbHandler.java (line 73) RowMutation(table='permacache', key='comments_sort_21188087__confidence', modifications=[ColumnFamily(permacache [76616c7565:false:1128@1278357660437987,])]) applied. Sending response to 14264934@/10.251.82.223
ERROR [ROW-READ-STAGE:180] 2010-07-05 12:21:00,669 DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
java.lang.ArrayIndexOutOfBoundsException
at org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
at java.io.DataInputStream.readUTF(DataInputStream.java:609)
at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:938)
at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859)
at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:747)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:772)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739)
at org.apache.cassandra.db.Table.getRow(Table.java:381)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
DEBUG [ROW-MUTATION-STAGE:9] 2010-07-05 12:21:00,666 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='permacache', key='comments_sort_21188087__hot', modifications=[ColumnFamily(permacache [76616c7565:false:1076@1278357660434824,])])
DEBUG [ROW-MUTATION-STAGE:18] 2010-07-05 12:21:00,665 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='permacache', key='comments_sort_21186905__controversy', modifications=[ColumnFamily(permacache [76616c7565:false:26956@1278357660427622,])])
DEBUG [ROW-MUTATION-STAGE:18] 2010-07-05 12:21:00,676 RowMutationVerbHandler.java (line 73) RowMutation(table='permacache', key='comments_sort_21186905__controversy', modifications=[ColumnFamily(permacache [76616c7565:false:26956@1278357660427622,])]) applied. Sending response to 14264931@/10.251.82.223
DEBUG [pool-1-thread-205] 2010-07-05 12:21:00,665 CassandraServer.java (line 227) multiget_slice
DEBUG [ROW-MUTATION-STAGE:28] 2010-07-05 12:21:00,661 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='reddit', key='2jpa_c0tk7r8', modifications=[ColumnFamily(CommentVote [date:false:13@1278357660430268,ip:false:11@1278357660430268,name:false:2@1278357660430268,thing1_id:false:4@1278357660430268,thing2_id:false:7@1278357660430268,valid_thing:false:1@1278357660430268,valid_user:false:1@1278357660430268,])])
DEBUG [ROW-READ-STAGE:184] 2010-07-05 12:21:00,660 ReadVerbHandler.java (line 75) digest is d41d8cd98f00b204e9800998ecf8427e
DEBUG [ROW-MUTATION-STAGE:11] 2010-07-05 12:21:00,660 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='permacache', key='4247f98369d200f3a53bea7cf24461eb860e1a0f', modifications=[ColumnFamily(permacache [76616c7565:false:2471@1278357660417184,])])
DEBUG [ROW-MUTATION-STAGE:11] 2010-07-05 12:21:00,677 RowMutationVerbHandler.java (line 73) RowMutation(table='permacache', key='4247f98369d200f3a53bea7cf24461eb860e1a0f', modifications=[ColumnFamily(permacache [76616c7565:false:2471@1278357660417184,])]) applied. Sending response to 14264928@/10.251.82.223
ERROR [ROW-READ-STAGE:190] 2010-07-05 12:21:00,658 DebuggableThreadPoolExecutor.java (line 101) Error in ThreadPoolExecutor
java.lang.ArrayIndexOutOfBoundsException
at org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
at java.io.DataInputStream.readUTF(DataInputStream.java:609)
at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:938)
at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859)
at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:747)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:772)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739)
at org.apache.cassandra.db.Table.getRow(Table.java:381)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
DEBUG [ROW-MUTATION-STAGE:12] 2010-07-05 12:21:00,655 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='permacache', key='last_comments_t3_cm600', modifications=[ColumnFamily(permacache [76616c7565:false:101@1278357660544809,])])
DEBUG [ROW-MUTATION-STAGE:12] 2010-07-05 12:21:00,678 RowMutationVerbHandler.java (line 73) RowMutation(table='permacache', key='last_comments_t3_cm600', modifications=[ColumnFamily(permacache [76616c7565:false:101@1278357660544809,])]) applied. Sending response to 2890083@/10.250.138.47
ERROR [ROW-READ-STAGE:190] 2010-07-05 12:21:00,678 CassandraDaemon.java (line 83) Uncaught exception in thread Thread[ROW-READ-STAGE:190,5,main]
java.lang.ArrayIndexOutOfBoundsException
at org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
at java.io.DataInputStream.readUTF(DataInputStream.java:609)
at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:938)
at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859)
at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:747)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:772)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739)
at org.apache.cassandra.db.Table.getRow(Table.java:381)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
DEBUG [ROW-MUTATION-STAGE:4] 2010-07-05 12:21:00,651 RowMutationVerbHandler.java (line 47) Applying RowMutation(table='permacache', key='comments_sort_21188483__controversy', modifications=[ColumnFamily(permacache [76616c7565:false:10948@1278357660432250,])])
{quote}
> ArrayIndexOutOfBoundsException thrown when reading partially-written sstable
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-1248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1248
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 0.6.3
> java version "1.6.0_0"
> IcedTea6 1.3.1 (6b12-0ubuntu6.6) Runtime Environment (build 1.6.0_0-b12)
> OpenJDK 64-Bit Server VM (build 1.6.0_0-b12, mixed mode)
> Ubuntu 8.10
> 6 node cluster
> RF==3
> Reporter: David King
> Fix For: 0.8
>
>
> A node died by running out of disk space. After growing the volume, trying to execute reads dies with:
> {quote}
> ERROR 11:02:51,957 Uncaught exception in thread Thread[ROW-READ-STAGE:10,5,main]
> java.lang.ArrayIndexOutOfBoundsException
> at org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
> at java.io.DataInputStream.readUTF(DataInputStream.java:609)
> at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:938)
> at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
> at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:747)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:772)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739)
> at org.apache.cassandra.db.Table.getRow(Table.java:381)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {quote}
> The clients then die with TimeoutExceptions.
> It might be better to return an exception to the client immediately instead of letting them die, and it would be helpful if this exception were accompanied with a message indicating *which* SStable is malformed.
> This command does *not* turn up the failed table (nor does the same with sstablekeys):
> {quote}
> ri@pmc08:/cassandra/data$ for x in $(ls -t */*-Data.db); do ls -lh $x; /usr/local/cassandra/bin/sstable2json $x > /dev/null; done
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1248) ArrayIndexOutOfBoundsException
thrown when reading partially-written sstable
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1248:
--------------------------------------
Fix Version/s: 0.8
Affects Version/s: (was: 0.6.3)
What stacktrace did you get during the write?
What is confusing is that I would expect partially written sstables to be removed after restart (since they would be marked with tmp in the filename).
> ArrayIndexOutOfBoundsException thrown when reading partially-written sstable
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-1248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1248
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 0.6.3
> java version "1.6.0_0"
> IcedTea6 1.3.1 (6b12-0ubuntu6.6) Runtime Environment (build 1.6.0_0-b12)
> OpenJDK 64-Bit Server VM (build 1.6.0_0-b12, mixed mode)
> Ubuntu 8.10
> 6 node cluster
> RF==3
> Reporter: David King
> Fix For: 0.8
>
>
> A node died by running out of disk space. After growing the volume, trying to execute reads dies with:
> {quote}
> ERROR 11:02:51,957 Uncaught exception in thread Thread[ROW-READ-STAGE:10,5,main]
> java.lang.ArrayIndexOutOfBoundsException
> at org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
> at java.io.DataInputStream.readUTF(DataInputStream.java:609)
> at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:938)
> at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
> at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:747)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:772)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739)
> at org.apache.cassandra.db.Table.getRow(Table.java:381)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {quote}
> The clients then die with TimeoutExceptions.
> It might be better to return an exception to the client immediately instead of letting them die, and it would be helpful if this exception were accompanied with a message indicating *which* SStable is malformed.
> This command does *not* turn up the failed table (nor does the same with sstablekeys):
> {quote}
> ri@pmc08:/cassandra/data$ for x in $(ls -t */*-Data.db); do ls -lh $x; /usr/local/cassandra/bin/sstable2json $x > /dev/null; done
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1248) ArrayIndexOutOfBoundsException
thrown when reading partially-written sstable
Posted by "David King (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12885354#action_12885354 ]
David King commented on CASSANDRA-1248:
---------------------------------------
I tried forcing the node to compact (by bringing it up listening only on localhost to keep it from joining the other nodes, and telling it to compact), and it throws this exception:
{quote}
INFO 15:00:12,012 Deleted /cassandra/data/reddit/CommentVote-1015-Data.db
INFO 15:00:12,314 Deleted /cassandra/data/system/HintsColumnFamily-1239-Data.db
INFO 15:00:12,923 Deleted /cassandra/data/reddit/VotesByLink-1059-Data.db
INFO 15:00:19,785 Deleted /cassandra/data/reddit/LinkVote-991-Data.db
ERROR 15:10:11,307 Skipping row DecoratedKey(15495971576493891752174312399692978238, ac5b905cb26978367a0a777c6fb7a9d1f2f162b4) in /cassandra/data/permacache/permacache-45341-Data.db
java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:394)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:87)
at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:29)
at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:117)
at org.apache.cassandra.db.ColumnFamilySerializer.deserializeFromSSTable(ColumnFamilySerializer.java:160)
at org.apache.cassandra.io.IteratingRow.getColumnFamily(IteratingRow.java:79)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:112)
at org.apache.cassandra.io.CompactionIterator.getReduced(CompactionIterator.java:41)
at org.apache.cassandra.utils.ReducingIterator.computeNext(ReducingIterator.java:73)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:135)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:130)
at org.apache.commons.collections.iterators.FilterIterator.setNextObject(FilterIterator.java:183)
at org.apache.commons.collections.iterators.FilterIterator.hasNext(FilterIterator.java:94)
at org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:295)
at org.apache.cassandra.db.CompactionManager$4.call(CompactionManager.java:177)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
{quote}
Telling it to compact again throws the same exception.
On the up-side, assuming there's only one corrupt file at least I know which one it is now.
> ArrayIndexOutOfBoundsException thrown when reading partially-written sstable
> ----------------------------------------------------------------------------
>
> Key: CASSANDRA-1248
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1248
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Cassandra 0.6.3
> java version "1.6.0_0"
> IcedTea6 1.3.1 (6b12-0ubuntu6.6) Runtime Environment (build 1.6.0_0-b12)
> OpenJDK 64-Bit Server VM (build 1.6.0_0-b12, mixed mode)
> Ubuntu 8.10
> 6 node cluster
> RF==3
> Reporter: David King
> Fix For: 0.8
>
>
> A node died by running out of disk space. After growing the volume, trying to execute reads dies with:
> {quote}
> ERROR 11:02:51,957 Uncaught exception in thread Thread[ROW-READ-STAGE:10,5,main]
> java.lang.ArrayIndexOutOfBoundsException
> at org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:326)
> at java.io.RandomAccessFile.readFully(RandomAccessFile.java:414)
> at java.io.DataInputStream.readUTF(DataInputStream.java:609)
> at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:938)
> at org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
> at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
> at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:859)
> at org.apache.cassandra.db.ColumnFamilyStore.cacheRow(ColumnFamilyStore.java:747)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:772)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:739)
> at org.apache.cassandra.db.Table.getRow(Table.java:381)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:56)
> at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> {quote}
> The clients then die with TimeoutExceptions.
> It might be better to return an exception to the client immediately instead of letting them die, and it would be helpful if this exception were accompanied with a message indicating *which* SStable is malformed.
> This command does *not* turn up the failed table (nor does the same with sstablekeys):
> {quote}
> ri@pmc08:/cassandra/data$ for x in $(ls -t */*-Data.db); do ls -lh $x; /usr/local/cassandra/bin/sstable2json $x > /dev/null; done
> {quote}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.