You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Dane Miller <da...@optimalsocial.com> on 2013/04/02 01:04:33 UTC
IndexOutOfBoundsException during repair, streaming
I hit some more errors while running repair. Repair starts streaming,
and then hangs. On 2 of the 3 nodes involved in the repair, "nodetool
netstats" show no streams. On the 3rd node there is an error in the
logs and "nodetool netstats" show several streams that appear hung --
no progress in about 1 hour. The error message from the log is below,
also viewable here: http://pastebin.com/JTcXgkNR. Any ideas how to
fix this?
ERROR [Thread-233] 2013-04-01 22:22:21,759 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-233,5,main]
java.lang.RuntimeException: java.nio.channels.AsynchronousCloseException
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.nio.channels.AsynchronousCloseException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:270)
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176)
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
... 1 more
ERROR [Thread-232] 2013-04-01 22:22:21,760 CassandraDaemon.java (line
133) Exception in thread Thread[Thread-232,5,main]
java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:96)
at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:128)
at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:125)
at java.util.TreeMap.compare(TreeMap.java:1173)
at java.util.TreeMap$NavigableSubMap.<init>(TreeMap.java:1243)
at java.util.TreeMap$AscendingSubMap.<init>(TreeMap.java:1677)
at java.util.TreeMap.tailMap(TreeMap.java:890)
at java.util.TreeSet.tailSet(TreeSet.java:333)
at java.util.TreeSet.tailSet(TreeSet.java:366)
at org.apache.cassandra.db.RangeTombstone$Tracker.update(RangeTombstone.java:206)
at org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:180)
at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:276)
at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226)
at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166)
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
Details:
cassandra 1.2.2 - single token per node
Ubuntu 10.10 x86_64
EC2 m1.large
Cassandra max heap: 4G
Dane
Re: IndexOutOfBoundsException during repair, streaming
Posted by Dane Miller <da...@optimalsocial.com>.
On Wed, Apr 3, 2013 at 6:08 PM, aaron morton <aa...@thelastpickle.com> wrote:
> We deleted and recreated those CFs before moving into
> production mode.
>
> We have a wiener.
>
> The comparator is applying the current schema to the byte value read from
> disk (schema on read) which describes a value with more than 2 components.
> It's then trying to apply the current schema so it can type cast the bytes
> for comparison.
>
> Something must have gone wrong in the "deleted" part of your statement
> above. We do not store schema with data, so this a problem of changing the
> schema in an incompatible way with existing data.
>
> nodetool scrub is probably your best bet. I've not checked that it handles
> this specific problem, but in general it will drop rows from SSTables that
> cannot be read or have some other problem. Best thing to do is snapshot and
> copy the data from one prod node to a QA box and run some tests.
>
> hope that helps.
I scrubbed the CF and was able to complete the repair. Thanks Aaron!
Dane
Re: IndexOutOfBoundsException during repair, streaming
Posted by aaron morton <aa...@thelastpickle.com>.
> We deleted and recreated those CFs before moving into
> production mode.
We have a wiener.
The comparator is applying the current schema to the byte value read from disk (schema on read) which describes a value with more than 2 components. It's then trying to apply the current schema so it can type cast the bytes for comparison.
Something must have gone wrong in the "deleted" part of your statement above. We do not store schema with data, so this a problem of changing the schema in an incompatible way with existing data.
nodetool scrub is probably your best bet. I've not checked that it handles this specific problem, but in general it will drop rows from SSTables that cannot be read or have some other problem. Best thing to do is snapshot and copy the data from one prod node to a QA box and run some tests.
hope that helps.
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 3/04/2013, at 2:11 AM, Dane Miller <da...@optimalsocial.com> wrote:
> On Mon, Apr 1, 2013 at 10:19 PM, aaron morton <aa...@thelastpickle.com> wrote:
>> ERROR [Thread-232] 2013-04-01 22:22:21,760 CassandraDaemon.java (line
>> 133) Exception in thread Thread[Thread-232,5,main]
>> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
>> at
>> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
>> at
>> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
>> at
>> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
>> at
>> org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:96)
>>
>> Something odd in the schema world perhaps.
>>
>> Has the schema changed recently?
>
> No, not recently. But during development we experimented with other
> comparator types for those CFs. More info below.
>
>> Do yo have more than one schema in the cluster ? (describe cluster in
>> cassandra-cli)
>
> I don't think so, that command in cassandra-cli shows just a single
> schema in the cluster:
>
> Cluster Information:
> Snitch: org.apache.cassandra.locator.Ec2Snitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 126b31ad-3660-3831-9d4f-c6763c9acc97: [ ...ip list... ]
>
>
> This error happened on a CF where the composite is: (Integer, UTF8)
>
> I'm a bit stumped about how we could get the an index==2 in that code
> pathway. See here:
> https://github.com/apache/cassandra/blob/cassandra-1.2.1/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
>
> ...start at line 63 in compare.
>
> My Java is terrible, but all of our CompositeTypes are composites of
> only two types. Thus the counter 'i' should never get up to 2 (it is
> used to access by index to individual comparator types within the
> composite), unless the value of the first two components in each of
> the column names being compared are equal, which should be impossible.
>
> During development we experimented with other comparator types for
> those CFs. We deleted and recreated those CFs before moving into
> production mode. Is there a chance Cassandra 'remembers' these old
> types from a deleted CF that shares a name with an existing CF? Could
> that be causing improper parsing of comparator column names?
>
> That we enter this code pathway from a section that seems to want to
> clean up tombstones makes me think this is a possibility, that there
> is a tombstone somewhere whose composite column name is causing
> issues.
>
> Dane
Re: IndexOutOfBoundsException during repair, streaming
Posted by Dane Miller <da...@optimalsocial.com>.
On Mon, Apr 1, 2013 at 10:19 PM, aaron morton <aa...@thelastpickle.com> wrote:
> ERROR [Thread-232] 2013-04-01 22:22:21,760 CassandraDaemon.java (line
> 133) Exception in thread Thread[Thread-232,5,main]
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> at
> com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> at
> com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
> at
> org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:96)
>
> Something odd in the schema world perhaps.
>
> Has the schema changed recently?
No, not recently. But during development we experimented with other
comparator types for those CFs. More info below.
> Do yo have more than one schema in the cluster ? (describe cluster in
> cassandra-cli)
I don't think so, that command in cassandra-cli shows just a single
schema in the cluster:
Cluster Information:
Snitch: org.apache.cassandra.locator.Ec2Snitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
126b31ad-3660-3831-9d4f-c6763c9acc97: [ ...ip list... ]
This error happened on a CF where the composite is: (Integer, UTF8)
I'm a bit stumped about how we could get the an index==2 in that code
pathway. See here:
https://github.com/apache/cassandra/blob/cassandra-1.2.1/src/java/org/apache/cassandra/db/marshal/AbstractCompositeType.java
...start at line 63 in compare.
My Java is terrible, but all of our CompositeTypes are composites of
only two types. Thus the counter 'i' should never get up to 2 (it is
used to access by index to individual comparator types within the
composite), unless the value of the first two components in each of
the column names being compared are equal, which should be impossible.
During development we experimented with other comparator types for
those CFs. We deleted and recreated those CFs before moving into
production mode. Is there a chance Cassandra 'remembers' these old
types from a deleted CF that shares a name with an existing CF? Could
that be causing improper parsing of comparator column names?
That we enter this code pathway from a section that seems to want to
clean up tombstones makes me think this is a possibility, that there
is a tombstone somewhere whose composite column name is causing
issues.
Dane
Re: IndexOutOfBoundsException during repair, streaming
Posted by aaron morton <aa...@thelastpickle.com>.
> ERROR [Thread-232] 2013-04-01 22:22:21,760 CassandraDaemon.java (line
> 133) Exception in thread Thread[Thread-232,5,main]
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
> at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:96)
Something odd in the schema world perhaps.
Has the schema changed recently?
Do yo have more than one schema in the cluster ? (describe cluster in cassandra-cli)
Cheers
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 2/04/2013, at 4:34 AM, Dane Miller <da...@optimalsocial.com> wrote:
> I hit some more errors while running repair. Repair starts streaming,
> and then hangs. On 2 of the 3 nodes involved in the repair, "nodetool
> netstats" show no streams. On the 3rd node there is an error in the
> logs and "nodetool netstats" show several streams that appear hung --
> no progress in about 1 hour. The error message from the log is below,
> also viewable here: http://pastebin.com/JTcXgkNR. Any ideas how to
> fix this?
>
> ERROR [Thread-233] 2013-04-01 22:22:21,759 CassandraDaemon.java (line
> 133) Exception in thread Thread[Thread-233,5,main]
> java.lang.RuntimeException: java.nio.channels.AsynchronousCloseException
> at com.google.common.base.Throwables.propagate(Throwables.java:160)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.nio.channels.AsynchronousCloseException
> at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:270)
> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
> at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:151)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ... 1 more
> ERROR [Thread-232] 2013-04-01 22:22:21,760 CassandraDaemon.java (line
> 133) Exception in thread Thread[Thread-232,5,main]
> java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
> at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
> at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
> at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
> at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:96)
> at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
> at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
> at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:128)
> at org.apache.cassandra.db.RangeTombstone$Tracker$1.compare(RangeTombstone.java:125)
> at java.util.TreeMap.compare(TreeMap.java:1173)
> at java.util.TreeMap$NavigableSubMap.<init>(TreeMap.java:1243)
> at java.util.TreeMap$AscendingSubMap.<init>(TreeMap.java:1677)
> at java.util.TreeMap.tailMap(TreeMap.java:890)
> at java.util.TreeSet.tailSet(TreeSet.java:333)
> at java.util.TreeSet.tailSet(TreeSet.java:366)
> at org.apache.cassandra.db.RangeTombstone$Tracker.update(RangeTombstone.java:206)
> at org.apache.cassandra.db.ColumnIndex$Builder.add(ColumnIndex.java:180)
> at org.apache.cassandra.io.sstable.SSTableWriter.appendFromStream(SSTableWriter.java:276)
> at org.apache.cassandra.streaming.IncomingStreamReader.streamIn(IncomingStreamReader.java:179)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:122)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:226)
> at org.apache.cassandra.net.IncomingTcpConnection.handleStream(IncomingTcpConnection.java:166)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:66)
>
>
>
> Details:
> cassandra 1.2.2 - single token per node
> Ubuntu 10.10 x86_64
> EC2 m1.large
> Cassandra max heap: 4G
>
>
> Dane