You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Nethi, Manoj" <Ma...@fmr.com.INVALID> on 2019/12/20 18:45:46 UTC
Streaming Failed during bootstrap of a Replacement node
Hi,
We are seeing the following error while bootstrapping a node which is replacement of a failed node in a multi DC cluster.
WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:641 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Retrying for following error
java.lang.ArrayIndexOutOfBoundsException: -127
at org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:355) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:64) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.StreamReader$StreamDeserializer.hasNext(StreamReader.java:253) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:99) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:178) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:106) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
ERROR [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:520 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type 0
at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 StreamResultFuture.java:185 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Session with /**.***.**.** is complete
WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 StreamResultFuture.java:212 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Stream failed
WARN [Thread-738] 2019-12-19 23:36:40,126 CompressedInputStream.java:182 - Error while reading compressed input stream.
java.nio.channels.ClosedChannelException: null
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_74]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_74]
at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
Did any one faced a similar issue ? What should be done to resolve this ?
Thanks
Manoj
RE: Streaming Failed during bootstrap of a Replacement node
Posted by "Nethi, Manoj" <Ma...@fmr.com.INVALID>.
Hi All,
I see a different error as below this time:
WARN [STREAM-IN-/***] 2019-12-21 19:42:11,662 StreamSession.java:641 - [Stream #ec8d1000-2451-11ea-b38a-0f6cfab62953] Retrying for following error
java.lang.ArrayIndexOutOfBoundsException: null
ERROR [STREAM-IN-/***] 2019-12-21 19:42:12,034 StreamSession.java:520 - [Stream #ec8d1000-2451-11ea-b38a-0f6cfab62953] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type 0
at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO [STREAM-IN-/10.146.128.41] 2019-12-21 19:42:12,037 StreamResultFuture.java:185 - [Stream #ec8d1000-2451-11ea-b38a-0f6cfab62953] Session with /*** is complete
WARN [STREAM-IN-/10.146.128.41] 2019-12-21 19:42:12,038 StreamResultFuture.java:212 - [Stream #ec8d1000-2451-11ea-b38a-0f6cfab62953] Stream failed
WARN [Thread-862] 2019-12-21 19:42:12,039 CompressedInputStream.java:182 - Error while reading compressed input stream.
java.nio.channels.ClosedChannelException: null
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_74]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_74]
at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
Does anyone know a resolution for this ?
From: Jon Haddad <jo...@jonhaddad.com>
Sent: Friday, December 20, 2019 2:29 PM
To: user@cassandra.apache.org
Subject: Re: Streaming Failed during bootstrap of a Replacement node
This email is from an external source - exercise caution regarding links and attachments.
Without getting too in the weeds here - ideally yes, you'd be able to replace the node. However, the error you're getting looks (at a glance) to be one of the many range tombstone bugs that were fixed in later versions.
On Fri, Dec 20, 2019 at 11:23 AM Nethi, Manoj <Ma...@fmr.com.invalid>> wrote:
Hi Jon,
Yes we will upgrade it soon. But before we can upgrade shouldn’t we get this lost node in the cluster to be replaced ?
From: Jon Haddad <jo...@jonhaddad.com>>
Sent: Friday, December 20, 2019 2:13 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Streaming Failed during bootstrap of a Replacement node
This email is from an external source - exercise caution regarding links and attachments.
You should upgrade to Cassandra 3.11.5 before doing anything else. You're running a pretty old and buggy version. There's been hundreds (maybe thousands) of bugs fixed between 3.3 and 3.11.5.
On Fri, Dec 20, 2019 at 10:46 AM Nethi, Manoj <Ma...@fmr.com.invalid>> wrote:
Hi,
We are seeing the following error while bootstrapping a node which is replacement of a failed node in a multi DC cluster.
WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:641 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Retrying for following error
java.lang.ArrayIndexOutOfBoundsException: -127
at org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:355) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:64) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.StreamReader$StreamDeserializer.hasNext(StreamReader.java:253) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:99) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:178) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:106) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
ERROR [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:520 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type 0
at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 StreamResultFuture.java:185 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Session with /**.***.**.** is complete
WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 StreamResultFuture.java:212 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Stream failed
WARN [Thread-738] 2019-12-19 23:36:40,126 CompressedInputStream.java:182 - Error while reading compressed input stream.
java.nio.channels.ClosedChannelException: null
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_74]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_74]
at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
Did any one faced a similar issue ? What should be done to resolve this ?
Thanks
Manoj
Re: Streaming Failed during bootstrap of a Replacement node
Posted by Jon Haddad <jo...@jonhaddad.com>.
Without getting too in the weeds here - ideally yes, you'd be able to
replace the node. However, the error you're getting looks (at a glance) to
be one of the many range tombstone bugs that were fixed in later versions.
On Fri, Dec 20, 2019 at 11:23 AM Nethi, Manoj <Ma...@fmr.com.invalid>
wrote:
> Hi Jon,
>
> Yes we will upgrade it soon. But before we can upgrade shouldn’t we get
> this lost node in the cluster to be replaced ?
>
>
>
>
>
>
>
> *From:* Jon Haddad <jo...@jonhaddad.com>
> *Sent:* Friday, December 20, 2019 2:13 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Streaming Failed during bootstrap of a Replacement node
>
>
>
> *This email is from an external source - **exercise caution regarding
> links and attachments. *
>
>
>
> You should upgrade to Cassandra 3.11.5 before doing anything else. You're
> running a pretty old and buggy version. There's been hundreds (maybe
> thousands) of bugs fixed between 3.3 and 3.11.5.
>
>
>
> On Fri, Dec 20, 2019 at 10:46 AM Nethi, Manoj <Ma...@fmr.com.invalid>
> wrote:
>
> Hi,
>
>
>
> We are seeing the following error while bootstrapping a node which is
> replacement of a failed node in a multi DC cluster.
>
>
>
>
>
>
>
> WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120
> StreamSession.java:641 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953]
> Retrying for following error
>
> java.lang.ArrayIndexOutOfBoundsException: -127
>
> at
> org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:355)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:64)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.StreamReader$StreamDeserializer.hasNext(StreamReader.java:253)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:99)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:178)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:106)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
>
> ERROR [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120
> StreamSession.java:520 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953]
> Streaming error occurred
>
> java.lang.IllegalArgumentException: Unknown type 0
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
>
> INFO [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124
> StreamResultFuture.java:185 - [Stream
> #f231db30-22da-11ea-b38a-0f6cfab62953] Session with /**.***.**.** is
> complete
>
> WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124
> StreamResultFuture.java:212 - [Stream
> #f231db30-22da-11ea-b38a-0f6cfab62953] Stream failed
>
> WARN [Thread-738] 2019-12-19 23:36:40,126 CompressedInputStream.java:182
> - Error while reading compressed input stream.
>
> java.nio.channels.ClosedChannelException: null
>
> at
> sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)
> ~[na:1.8.0_74]
>
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)
> ~[na:1.8.0_74]
>
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
> ~[na:1.8.0_74]
>
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
> ~[na:1.8.0_74]
>
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_74]
>
> at
> org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
>
>
>
>
>
>
>
> Did any one faced a similar issue ? What should be done to resolve this ?
>
>
>
>
>
>
>
> Thanks
>
> Manoj
>
>
RE: Streaming Failed during bootstrap of a Replacement node
Posted by "Nethi, Manoj" <Ma...@fmr.com.INVALID>.
Hi Jon,
Yes we will upgrade it soon. But before we can upgrade shouldn’t we get this lost node in the cluster to be replaced ?
From: Jon Haddad <jo...@jonhaddad.com>
Sent: Friday, December 20, 2019 2:13 PM
To: user@cassandra.apache.org
Subject: Re: Streaming Failed during bootstrap of a Replacement node
This email is from an external source - exercise caution regarding links and attachments.
You should upgrade to Cassandra 3.11.5 before doing anything else. You're running a pretty old and buggy version. There's been hundreds (maybe thousands) of bugs fixed between 3.3 and 3.11.5.
On Fri, Dec 20, 2019 at 10:46 AM Nethi, Manoj <Ma...@fmr.com.invalid>> wrote:
Hi,
We are seeing the following error while bootstrapping a node which is replacement of a failed node in a multi DC cluster.
WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:641 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Retrying for following error
java.lang.ArrayIndexOutOfBoundsException: -127
at org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:355) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:64) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.StreamReader$StreamDeserializer.hasNext(StreamReader.java:253) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:99) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:178) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:106) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59) [apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
ERROR [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120 StreamSession.java:520 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Streaming error occurred
java.lang.IllegalArgumentException: Unknown type 0
at org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261) ~[apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 StreamResultFuture.java:185 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Session with /**.***.**.** is complete
WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124 StreamResultFuture.java:212 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953] Stream failed
WARN [Thread-738] 2019-12-19 23:36:40,126 CompressedInputStream.java:182 - Error while reading compressed input stream.
java.nio.channels.ClosedChannelException: null
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257) ~[na:1.8.0_74]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) ~[na:1.8.0_74]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_74]
at org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172) ~[apache-cassandra-3.3.0.jar:3.3.0]
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [apache-cassandra-3.3.0.jar:3.3.0]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
Did any one faced a similar issue ? What should be done to resolve this ?
Thanks
Manoj
Re: Streaming Failed during bootstrap of a Replacement node
Posted by Jon Haddad <jo...@jonhaddad.com>.
You should upgrade to Cassandra 3.11.5 before doing anything else. You're
running a pretty old and buggy version. There's been hundreds (maybe
thousands) of bugs fixed between 3.3 and 3.11.5.
On Fri, Dec 20, 2019 at 10:46 AM Nethi, Manoj <Ma...@fmr.com.invalid>
wrote:
> Hi,
>
>
>
> We are seeing the following error while bootstrapping a node which is
> replacement of a failed node in a multi DC cluster.
>
>
>
>
>
>
>
> WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120
> StreamSession.java:641 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953]
> Retrying for following error
>
> java.lang.ArrayIndexOutOfBoundsException: -127
>
> at
> org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:355)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:86)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:64)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.StreamReader$StreamDeserializer.hasNext(StreamReader.java:253)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:108)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:120)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:57)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:153)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.append(SimpleSSTableMultiWriter.java:48)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.io.sstable.format.RangeAwareSSTableWriter.append(RangeAwareSSTableWriter.java:99)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.StreamReader.writePartition(StreamReader.java:178)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:106)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:50)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:39)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:59)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
>
> ERROR [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,120
> StreamSession.java:520 - [Stream #f231db30-22da-11ea-b38a-0f6cfab62953]
> Streaming error occurred
>
> java.lang.IllegalArgumentException: Unknown type 0
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage$Type.get(StreamMessage.java:97)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:58)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:261)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
>
> INFO [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124
> StreamResultFuture.java:185 - [Stream
> #f231db30-22da-11ea-b38a-0f6cfab62953] Session with /**.***.**.** is
> complete
>
> WARN [STREAM-IN-/**.***.***.**] 2019-12-19 23:36:40,124
> StreamResultFuture.java:212 - [Stream
> #f231db30-22da-11ea-b38a-0f6cfab62953] Stream failed
>
> WARN [Thread-738] 2019-12-19 23:36:40,126 CompressedInputStream.java:182
> - Error while reading compressed input stream.
>
> java.nio.channels.ClosedChannelException: null
>
> at
> sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)
> ~[na:1.8.0_74]
>
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)
> ~[na:1.8.0_74]
>
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:59)
> ~[na:1.8.0_74]
>
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
> ~[na:1.8.0_74]
>
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_74]
>
> at
> org.apache.cassandra.streaming.compress.CompressedInputStream$Reader.runMayThrow(CompressedInputStream.java:172)
> ~[apache-cassandra-3.3.0.jar:3.3.0]
>
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> [apache-cassandra-3.3.0.jar:3.3.0]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
>
>
>
>
>
>
>
> Did any one faced a similar issue ? What should be done to resolve this ?
>
>
>
>
>
>
>
> Thanks
>
> Manoj
>