You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Varun Barala <va...@gmail.com> on 2017/10/10 02:54:24 UTC
Stream is failing while removing the node
Hi developers,
Recently, I was removing one node from the cluster without downtime.
Cluster size :: 3 node [test cluster with nodeA, nodeB, nodeC]
*Procedure:-*1 Change RF to 1
2 Check all the nodes are up
3 Run repair on all the nodes
4 Run decommission on node nodeC
5 After decommission
6 Check node status(status should be *UN*)
7 Clean up on all the nodes
* repeat steps[2..7] for nodeB
But we found some weird exception in the logs:-
*level:WARN thread:STREAM-IN-/10.142.1.11 <http://10.142.1.11>
@timestamp:2017-10-06T02:44:21.567+0000
hostname:ip-10-142-1-10.compute.internal
loggerName:org.apache.cassandra.streaming.compress.CompressedStreamReader
file:CompressedStreamReader.java:115 message:[Stream
cc2f84d0-aa3e-11e7-a2fc-fd45c9a98801] Error while reading partition
DecoratedKey(-9223372036854775808, ) from stream on ks='ks1' and
table='cf1'. throwable:*
*level:WARN thread:STREAM-IN-/10.142.1.11 <http://10.142.1.11>
@timestamp:2017-10-06T02:44:21.568+0000
hostname:ip-10-142-1-10.compute.internal
loggerName:org.apache.cassandra.streaming.StreamSession
file:StreamSession.java:626 message:[Stream
#cc2f84d0-aa3e-11e7-a2fc-fd45c9a98801] Retrying for following error
throwable:java.lang.AssertionError: null\nat
org.apache.cassandra.streaming.compress.CompressedInputStream.read(CompressedInputStream.java:106)\nat
java.io.InputStream.read(InputStream.java:170)\nat
java.io.InputStream.skip(InputStream.java:224)\nat
org.apache.cassandra.streaming.StreamReader.drain(StreamReader.java:158)\nat
org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:129)\nat
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:48)\nat
org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38)\nat
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56)\nat
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)\nat
java.lang.Thread.run(Thread.java:745)\n*
assertion failing at:-
https://github.com/apache/cassandra/blob/cassandra-2.1.13/src/java/org/apache/cassandra/streaming/compress/CompressedInputStream.java#L106
Could someone please tell me in which scenario this happens, Thanks!!
Thanks in advance!!
Regards
Varun Barala
Re: Stream is failing while removing the node
Posted by Jason Brown <ja...@gmail.com>.
Varun,
Please open a JIRA ticket with all the details of what you are seeing.
Thanks,
-Jason
On Mon, Oct 9, 2017 at 7:54 PM, Varun Barala <va...@gmail.com>
wrote:
> Hi developers,
>
> Recently, I was removing one node from the cluster without downtime.
>
> Cluster size :: 3 node [test cluster with nodeA, nodeB, nodeC]
>
>
> *Procedure:-*1 Change RF to 1
> 2 Check all the nodes are up
> 3 Run repair on all the nodes
> 4 Run decommission on node nodeC
> 5 After decommission
> 6 Check node status(status should be *UN*)
> 7 Clean up on all the nodes
> * repeat steps[2..7] for nodeB
>
>
> But we found some weird exception in the logs:-
> *level:WARN thread:STREAM-IN-/10.142.1.11 <http://10.142.1.11>
> @timestamp:2017-10-06T02:44:21.567+0000
> hostname:ip-10-142-1-10.compute.internal
> loggerName:org.apache.cassandra.streaming.compress.CompressedStreamReader
> file:CompressedStreamReader.java:115 message:[Stream
> cc2f84d0-aa3e-11e7-a2fc-fd45c9a98801] Error while reading partition
> DecoratedKey(-9223372036854775808, ) from stream on ks='ks1' and
> table='cf1'. throwable:*
> *level:WARN thread:STREAM-IN-/10.142.1.11 <http://10.142.1.11>
> @timestamp:2017-10-06T02:44:21.568+0000
> hostname:ip-10-142-1-10.compute.internal
> loggerName:org.apache.cassandra.streaming.StreamSession
> file:StreamSession.java:626 message:[Stream
> #cc2f84d0-aa3e-11e7-a2fc-fd45c9a98801] Retrying for following error
> throwable:java.lang.AssertionError: null\nat
> org.apache.cassandra.streaming.compress.CompressedInputStream.read(
> CompressedInputStream.java:106)\nat
> java.io.InputStream.read(InputStream.java:170)\nat
> java.io.InputStream.skip(InputStream.java:224)\nat
> org.apache.cassandra.streaming.StreamReader.drain(
> StreamReader.java:158)\nat
> org.apache.cassandra.streaming.compress.CompressedStreamReader.read(
> CompressedStreamReader.java:129)\nat
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(
> IncomingFileMessage.java:48)\nat
> org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(
> IncomingFileMessage.java:38)\nat
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(
> StreamMessage.java:56)\nat
> org.apache.cassandra.streaming.ConnectionHandler$
> IncomingMessageHandler.run(ConnectionHandler.java:250)\nat
> java.lang.Thread.run(Thread.java:745)\n*
>
> assertion failing at:-
> https://github.com/apache/cassandra/blob/cassandra-2.1.
> 13/src/java/org/apache/cassandra/streaming/compress/
> CompressedInputStream.java#L106
>
> Could someone please tell me in which scenario this happens, Thanks!!
>
> Thanks in advance!!
>
> Regards
> Varun Barala
>