You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Scott White <sc...@gmail.com> on 2010/03/26 20:53:41 UTC

ArrayIndexOutOfBoundsException on 0.6-betarc3

I don't know if this is from switching from 0.5 to 0.6-betarc3 just recently
or from doing a series of bootstrap and removeToken operations but I
recently started getting ArrayIndexOutOfBoundsException exceptions (centered
around reading UTF from SSTableSliceIterator) on one of the machines in my
cluster (see below). Obviously I could take the node out of the cluster and
replace it with a new one, but if anyone has suggestions for how to repair
the existing machine, that would be appreciated.

thanks!
Scott

Here are some examples:

ERROR [ROW-READ-STAGE:3503] 2010-03-26 06:04:11,421 CassandraDaemon.java
(line 78) Fatal exception in thread Thread[ROW-READ-STAGE:3503,5,main]
java.lang.ArrayIndexOutOfBoundsException
    at
org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:317)
    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
    at java.io.DataInputStream.readUTF(DataInputStream.java:592)
    at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887)
    at
org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
    at
org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
    at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
    at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:833)
    at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:753)
    at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:722)
    at org.apache.cassandra.db.Table.getRow(Table.java:381)
    at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
    at
org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:80)
    at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:619)

at
org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:317)
        at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
        at java.io.DataInputStream.readUTF(DataInputStream.java:592)
        at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887)
        at
org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
        at
org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
        at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
        at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:833)
        at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:753)
        at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:722)
        at
org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122)
        at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
        at
org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80)
        at
org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280)
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)

Re: ArrayIndexOutOfBoundsException on 0.6-betarc3

Posted by Jonathan Ellis <jb...@gmail.com>.

The quickest solution is definitely going to be just blowing away the
Hint files from the system keyspace data directories.

On Fri, Mar 26, 2010 at 5:36 PM, Scott White <sc...@gmail.com> wrote:
> Nope it's always been random.
>
> On Fri, Mar 26, 2010 at 2:13 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> Did you switch partitioner types at some point?
>>
>> On Fri, Mar 26, 2010 at 2:53 PM, Scott White <sc...@gmail.com> wrote:
>> > I don't know if this is from switching from 0.5 to 0.6-betarc3 just
>> > recently
>> > or from doing a series of bootstrap and removeToken operations but I
>> > recently started getting ArrayIndexOutOfBoundsException exceptions
>> > (centered
>> > around reading UTF from SSTableSliceIterator) on one of the machines in
>> > my
>> > cluster (see below). Obviously I could take the node out of the cluster
>> > and
>> > replace it with a new one, but if anyone has suggestions for how to
>> > repair
>> > the existing machine, that would be appreciated.
>> >
>> > thanks!
>> > Scott
>> >
>> > Here are some examples:
>> >
>> > ERROR [ROW-READ-STAGE:3503] 2010-03-26 06:04:11,421 CassandraDaemon.java
>> > (line 78) Fatal exception in thread Thread[ROW-READ-STAGE:3503,5,main]
>> > java.lang.ArrayIndexOutOfBoundsException
>> >     at
>> >
>> > org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:317)
>> >     at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
>> >     at java.io.DataInputStream.readUTF(DataInputStream.java:592)
>> >     at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887)
>> >     at
>> >
>> > org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
>> >     at
>> >
>> > org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
>> >     at
>> >
>> > org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
>> >     at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:833)
>> >     at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:753)
>> >     at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:722)
>> >     at org.apache.cassandra.db.Table.getRow(Table.java:381)
>> >     at
>> >
>> > org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
>> >     at
>> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:80)
>> >     at
>> >
>> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>> >     at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >     at
>> >
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >     at java.lang.Thread.run(Thread.java:619)
>> >
>> > at
>> >
>> > org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:317)
>> >         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
>> >         at java.io.DataInputStream.readUTF(DataInputStream.java:592)
>> >         at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887)
>> >         at
>> >
>> > org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
>> >         at
>> >
>> > org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
>> >         at
>> >
>> > org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:833)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:753)
>> >         at
>> >
>> > org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:722)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80)
>> >         at
>> >
>> > org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280)
>> >         at
>> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>
>

Re: ArrayIndexOutOfBoundsException on 0.6-betarc3

Posted by Scott White <sc...@gmail.com>.

Nope it's always been random.

On Fri, Mar 26, 2010 at 2:13 PM, Jonathan Ellis <jb...@gmail.com> wrote:

> Did you switch partitioner types at some point?
>
> On Fri, Mar 26, 2010 at 2:53 PM, Scott White <sc...@gmail.com> wrote:
> > I don't know if this is from switching from 0.5 to 0.6-betarc3 just
> recently
> > or from doing a series of bootstrap and removeToken operations but I
> > recently started getting ArrayIndexOutOfBoundsException exceptions
> (centered
> > around reading UTF from SSTableSliceIterator) on one of the machines in
> my
> > cluster (see below). Obviously I could take the node out of the cluster
> and
> > replace it with a new one, but if anyone has suggestions for how to
> repair
> > the existing machine, that would be appreciated.
> >
> > thanks!
> > Scott
> >
> > Here are some examples:
> >
> > ERROR [ROW-READ-STAGE:3503] 2010-03-26 06:04:11,421 CassandraDaemon.java
> > (line 78) Fatal exception in thread Thread[ROW-READ-STAGE:3503,5,main]
> > java.lang.ArrayIndexOutOfBoundsException
> >     at
> >
> org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:317)
> >     at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
> >     at java.io.DataInputStream.readUTF(DataInputStream.java:592)
> >     at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887)
> >     at
> >
> org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
> >     at
> >
> org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
> >     at
> >
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
> >     at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:833)
> >     at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:753)
> >     at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:722)
> >     at org.apache.cassandra.db.Table.getRow(Table.java:381)
> >     at
> >
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
> >     at
> > org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:80)
> >     at
> >
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >     at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >     at java.lang.Thread.run(Thread.java:619)
> >
> > at
> >
> org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:317)
> >         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
> >         at java.io.DataInputStream.readUTF(DataInputStream.java:592)
> >         at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887)
> >         at
> >
> org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
> >         at
> >
> org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
> >         at
> >
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
> >         at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:833)
> >         at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:753)
> >         at
> >
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:722)
> >         at
> >
> org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122)
> >         at
> >
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
> >         at
> >
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80)
> >         at
> >
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280)
> >         at
> > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>

Re: ArrayIndexOutOfBoundsException on 0.6-betarc3

Posted by Jonathan Ellis <jb...@gmail.com>.

Did you switch partitioner types at some point?

On Fri, Mar 26, 2010 at 2:53 PM, Scott White <sc...@gmail.com> wrote:
> I don't know if this is from switching from 0.5 to 0.6-betarc3 just recently
> or from doing a series of bootstrap and removeToken operations but I
> recently started getting ArrayIndexOutOfBoundsException exceptions (centered
> around reading UTF from SSTableSliceIterator) on one of the machines in my
> cluster (see below). Obviously I could take the node out of the cluster and
> replace it with a new one, but if anyone has suggestions for how to repair
> the existing machine, that would be appreciated.
>
> thanks!
> Scott
>
> Here are some examples:
>
> ERROR [ROW-READ-STAGE:3503] 2010-03-26 06:04:11,421 CassandraDaemon.java
> (line 78) Fatal exception in thread Thread[ROW-READ-STAGE:3503,5,main]
> java.lang.ArrayIndexOutOfBoundsException
>     at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:317)
>     at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
>     at java.io.DataInputStream.readUTF(DataInputStream.java:592)
>     at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887)
>     at
> org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
>     at
> org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
>     at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
>     at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:833)
>     at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:753)
>     at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:722)
>     at org.apache.cassandra.db.Table.getRow(Table.java:381)
>     at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
>     at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:80)
>     at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:619)
>
> at
> org.apache.cassandra.io.util.BufferedRandomAccessFile.read(BufferedRandomAccessFile.java:317)
>         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:381)
>         at java.io.DataInputStream.readUTF(DataInputStream.java:592)
>         at java.io.RandomAccessFile.readUTF(RandomAccessFile.java:887)
>         at
> org.apache.cassandra.db.filter.SSTableSliceIterator$ColumnGroupReader.<init>(SSTableSliceIterator.java:125)
>         at
> org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:59)
>         at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:63)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:833)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:753)
>         at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:722)
>         at
> org.apache.cassandra.db.HintedHandOffManager.sendMessage(HintedHandOffManager.java:122)
>         at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:250)
>         at
> org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffManager.java:80)
>         at
> org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffManager.java:280)
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)