You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Paulo Motta (Jira)" <ji...@apache.org> on 2025/08/29 15:29:00 UTC
[jira] [Comment Edited] (CASSANDRA-20873) Hanging streaming during repair
[ https://issues.apache.org/jira/browse/CASSANDRA-20873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18017046#comment-18017046 ]
Paulo Motta edited comment on CASSANDRA-20873 at 8/29/25 3:28 PM:
------------------------------------------------------------------
*Node1 log while draining*
* *node1 closing active connection with node2*
{noformat}
DEBUG [Messaging-EventLoop-3-1] 2025-08-28 18:49:40,117 StreamingMultiplexedChannel.java:514 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Closing stream connection channels on /www.xxx.yyy.zzz:1234
ERROR [Stream-Deserializer-/www.xxx.yyy.zzz:1234-46e95837] 2025-08-28 18:49:40,117 StreamSession.java:717 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Socket closed before session completion, peer www.xxx.yyy.zzz:1234 is probably down.
java.nio.channels.ClosedChannelException: null
at org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)
at org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:58)
{noformat}
* *node1 closing streaming session with node 2*
{noformat}
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,120 StreamSession.java:596 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Changing session state from STREAMING to FAILED
INFO [NonPeriodicTasks:1] 2025-08-28 18:49:40,120 StreamResultFuture.java:201 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Session with /www.xxx.yyy.zzz:1234 is failed
WARN [NonPeriodicTasks:1] 2025-08-28 18:49:40,120 StreamResultFuture.java:250 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Stream failed:
Session peer /www.xxx.yyy.zzz:1234 Failed because there was an java.nio.channels.ClosedChannelException with state=STREAMING
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,121 StreamSession.java:564 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Will close attached inbound {} and outbound {} channels
{noformat}
*node1 closing streaming with node 3 (even though node3 does not see this stream)*
{noformat}
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,121 StreamSession.java:596 - [Stream #37f2d540-62b1-11f0-afef-1340d58ed089] Changing session state from WAIT_COMPLETE to FAILED
INFO [NonPeriodicTasks:1] 2025-08-28 18:49:40,121 StreamResultFuture.java:201 - [Stream #37f2d540-62b1-11f0-afef-1340d58ed089] Session with /eee.fff.ggg.hhh:1234 is failed
WARN [NonPeriodicTasks:1] 2025-08-28 18:49:40,121 StreamResultFuture.java:250 - [Stream #37f2d540-62b1-11f0-afef-1340d58ed089] Stream failed:
Session peer /eee.fff.ggg.hhh:1234 Failed because there was an java.nio.channels.ClosedChannelException with state=WAIT_COMPLETE
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,121 StreamSession.java:564 - [Stream #37f2d540-62b1-11f0-afef-1340d58ed089] Will close attached inbound {eb5fe05c=org.apache.cassandra.streaming.async.NettyStreamingChannel@2cdae6ad} and outbound {ff4eaf92=org.apache.cassandra.streaming.async.NettyStreamingChannel@b66a5c0, 184ef185=org.apache.cassandra.streaming.async.NettyStreamingChannel@30b56f1, e16e60cd=org.apache.cassandra.streaming.async.NettyStreamingChannel@105f3d91, eb5fe05c=org.apache.cassandra.streaming.async.NettyStreamingChannel@2cdae6ad, 978d403e=org.apache.cassandra.streaming.async.NettyStreamingChannel@6bcb5eb9} channels
{noformat}
*node2 logs during node1 drain*
{noformat}
DEBUG [Messaging-EventLoop-3-6] 2025-08-28 18:49:40,119 StreamingMultiplexedChannel.java:514 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Closing stream connection channels on /aaa.bbb.ccc.ddd:1234
ERROR [Stream-Deserializer-/aaa.bbb.ccc.ddd:1234-d87ba681] 2025-08-28 18:49:40,119 StreamSession.java:717 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Socket closed before session completion, peer aaa.bbb.ccc.ddd:1234 is probably down.
ERROR [Stream-Deserializer-/aaa.bbb.ccc.ddd:1234-23d3c0ac] 2025-08-28 18:49:40,119 StreamSession.java:717 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Socket closed before session completion, peer aaa.bbb.ccc.ddd:1234 is probably down.
ERROR [Stream-Deserializer-/aaa.bbb.ccc.ddd:1234-678f744d] 2025-08-28 18:49:40,119 StreamSession.java:717 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Socket closed before session completion, peer aaa.bbb.ccc.ddd:1234 is probably down.
ERROR [Stream-Deserializer-/aaa.bbb.ccc.ddd:1234-60cfa387] 2025-08-28 18:49:40,120 StreamSession.java:717 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Socket closed before session completion, peer aaa.bbb.ccc.ddd:1234 is probably down.
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,123 StreamSession.java:596 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Changing session state from WAIT_COMPLETE to FAILED
INFO [NonPeriodicTasks:1] 2025-08-28 18:49:40,123 StreamResultFuture.java:201 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Session with /aaa.bbb.ccc.ddd:1234 is failed
WARN [NonPeriodicTasks:1] 2025-08-28 18:49:40,123 StreamResultFuture.java:250 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Stream failed:
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,125 StreamSession.java:564 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Will close attached inbound {} and outbound {} channels
{noformat}
was (Author: paulo):
*Node1 log while draining*
* *node1 closing active connection with node2*
{noformat}
DEBUG [Messaging-EventLoop-3-1] 2025-08-28 18:49:40,117 StreamingMultiplexedChannel.java:514 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Closing stream connection channels on /www.xxx.yyy.zzz:1234
ERROR [Stream-Deserializer-/www.xxx.yyy.zzz:1234-46e95837] 2025-08-28 18:49:40,117 StreamSession.java:717 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Socket closed before session completion, peer www.xxx.yyy.zzz:1234 is probably down.
java.nio.channels.ClosedChannelException: null
at org.apache.cassandra.net.AsyncStreamingInputPlus.reBuffer(AsyncStreamingInputPlus.java:119)
at org.apache.cassandra.io.util.RebufferingInputStream.readByte(RebufferingInputStream.java:178)
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:49)
at org.apache.cassandra.streaming.StreamDeserializingTask.run(StreamDeserializingTask.java:58)
{noformat}
* *node1 closing streaming session with node 2*
{noformat}
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,120 StreamSession.java:596 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Changing session state from STREAMING to FAILED
INFO [NonPeriodicTasks:1] 2025-08-28 18:49:40,120 StreamResultFuture.java:201 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Session with /www.xxx.yyy.zzz:1234 is failed
WARN [NonPeriodicTasks:1] 2025-08-28 18:49:40,120 StreamResultFuture.java:250 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Stream failed:
Session peer /www.xxx.yyy.zzz:1234 Failed because there was an java.nio.channels.ClosedChannelException with state=STREAMING
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,121 StreamSession.java:564 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Will close attached inbound {} and outbound {} channels
{noformat}
*node1 closing streaming with node 3 (even though node3 does not see this stream)*
{noformat}
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,121 StreamSession.java:596 - [Stream #37f2d540-62b1-11f0-afef-1340d58ed089] Changing session state from WAIT_COMPLETE to FAILED
INFO [NonPeriodicTasks:1] 2025-08-28 18:49:40,121 StreamResultFuture.java:201 - [Stream #37f2d540-62b1-11f0-afef-1340d58ed089] Session with /eee.fff.ggg.hhh:1234 is failed
WARN [NonPeriodicTasks:1] 2025-08-28 18:49:40,121 StreamResultFuture.java:250 - [Stream #37f2d540-62b1-11f0-afef-1340d58ed089] Stream failed:
Session peer /eee.fff.ggg.hhh:1234 Failed because there was an java.nio.channels.ClosedChannelException with state=WAIT_COMPLETE
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,121 StreamSession.java:564 - [Stream #37f2d540-62b1-11f0-afef-1340d58ed089] Will close attached inbound {eb5fe05c=org.apache.cassandra.streaming.async.NettyStreamingChannel@2cdae6ad} and outbound {ff4eaf92=org.apache.cassandra.streaming.async.NettyStreamingChannel@b66a5c0, 184ef185=org.apache.cassandra.streaming.async.NettyStreamingChannel@30b56f1, e16e60cd=org.apache.cassandra.streaming.async.NettyStreamingChannel@105f3d91, eb5fe05c=org.apache.cassandra.streaming.async.NettyStreamingChannel@2cdae6ad, 978d403e=org.apache.cassandra.streaming.async.NettyStreamingChannel@6bcb5eb9} channels
{noformat}
*node2 logs during node1 drain*
{noformat}
DEBUG [Messaging-EventLoop-3-6] 2025-08-28 18:49:40,119 StreamingMultiplexedChannel.java:514 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Closing stream connection channels on /aaa.bbb.ccc.ddd:3476
ERROR [Stream-Deserializer-/aaa.bbb.ccc.ddd:3476-d87ba681] 2025-08-28 18:49:40,119 StreamSession.java:717 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Socket closed before session completion, peer aaa.bbb.ccc.ddd:3476 is probably down.
ERROR [Stream-Deserializer-/aaa.bbb.ccc.ddd:3476-23d3c0ac] 2025-08-28 18:49:40,119 StreamSession.java:717 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Socket closed before session completion, peer aaa.bbb.ccc.ddd:3476 is probably down.
ERROR [Stream-Deserializer-/aaa.bbb.ccc.ddd:3476-678f744d] 2025-08-28 18:49:40,119 StreamSession.java:717 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Socket closed before session completion, peer aaa.bbb.ccc.ddd:3476 is probably down.
ERROR [Stream-Deserializer-/aaa.bbb.ccc.ddd:3476-60cfa387] 2025-08-28 18:49:40,120 StreamSession.java:717 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Socket closed before session completion, peer aaa.bbb.ccc.ddd:3476 is probably down.
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,123 StreamSession.java:596 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Changing session state from WAIT_COMPLETE to FAILED
INFO [NonPeriodicTasks:1] 2025-08-28 18:49:40,123 StreamResultFuture.java:201 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Session with /aaa.bbb.ccc.ddd:3476 is failed
WARN [NonPeriodicTasks:1] 2025-08-28 18:49:40,123 StreamResultFuture.java:250 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Stream failed:
DEBUG [NonPeriodicTasks:1] 2025-08-28 18:49:40,125 StreamSession.java:564 - [Stream #8bcef5b0-78b7-11f0-bfc8-7392e70ddefe] Will close attached inbound {} and outbound {} channels
{noformat}
> Hanging streaming during repair
> -------------------------------
>
> Key: CASSANDRA-20873
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20873
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Feature/Repair, Feature/Streaming, Messaging/Internode
> Reporter: Paulo Motta
> Priority: Normal
>
> Repair is not running but there are active streaming sessions. Logs will be posted as comments.
> * Node 1 (aaa.bbb.ccc.ddd)
> {noformat}
> $ nodetool netstats
> Mode: NORMAL
> Repair 8bcef5b0-78b7-11f0-bfc8-7392e70ddefe
> /www.xxx.yyy.zzz
> Receiving 105 files, 136257 bytes total. Already received 110 files (104.76%), 137014 bytes total (100.56%)
> $ nodetool repair_admin
> no sessions
> {noformat}
> * Node 2 (www.xxx.yyy.zzz)
> {noformat}
> $ nodetool netstats
> Mode: NORMAL
> Repair 8bcef5b0-78b7-11f0-bfc8-7392e70ddefe
> /aaa.bbb.ccc.ddd
> Sending 105 files, 136257 bytes total. Already sent 110 files (104.76%), 137014 bytes total (100.56%)
> $ nodetool repair_admin
> no sessions{noformat}
> * Node 3 (ccc.ddd.eee.fff)
> ** No streaming or repair running
> From the logs, we can gather the following streaming session states:
> Node1 view of the streaming session:
> * Node1 state: STREAMING
> * Node3 state: WAIT_COMPLETE
> Node2 view of the streaming session:
> * Node1 state: WAIT_COMPLETE
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org