You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Paulo Motta (JIRA)" <ji...@apache.org> on 2016/01/12 15:54:40 UTC
[jira] [Commented] (CASSANDRA-10992) Hanging streaming sessions
[ https://issues.apache.org/jira/browse/CASSANDRA-10992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15094002#comment-15094002 ]
Paulo Motta commented on CASSANDRA-10992:
-----------------------------------------
Unfortunately hanging stream sessions is a classic problem which can have many possible causes (including network problems), so it's difficult to troubleshoot without more information or reproduction steps. Some questions:
* Do you see any errors or warnings in your system logs?
* Could you try decreasing your streaming_socket_timeout_in_ms to 10000 ms and see if the problem persists?
If the problem persists and you're able to reproduce it fairly simply, I recommend you to replace the cassandra jar of all nodes involved in a specific repair with this [jar with more debug logging|https://issues.apache.org/jira/secure/attachment/12781836/apache-cassandra-2.1.12-SNAPSHOT.jar], and also set the log level of the {{org.apache.cassandra.streaming}} package to {{TRACE}} on {{conf/logback.xml}}, and post back the logs for better troubleshooting. If you prefer to build your own jar with {{ant jar}} you can do it from this [branch|https://github.com/pauloricardomg/cassandra/tree/2.1-10961].
> Hanging streaming sessions
> --------------------------
>
> Key: CASSANDRA-10992
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10992
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.12, Debian Wheezy
> Reporter: mlowicki
> Assignee: Paulo Motta
> Fix For: 2.1.12
>
> Attachments: apache-cassandra-2.1.12-SNAPSHOT.jar
>
>
> I've started recently running repair using [Cassandra Reaper|https://github.com/spotify/cassandra-reaper] (built-in {{nodetool repair}} doesn't work for me - CASSANDRA-9935). It behaves fine but I've noticed hanging streaming sessions:
> {code}
> root@db1:~# date
> Sat Jan 9 16:43:00 UTC 2016
> root@db1:~# nt netstats -H | grep total
> Receiving 5 files, 46.59 MB total. Already received 1 files, 11.32 MB total
> Sending 7 files, 46.28 MB total. Already sent 7 files, 46.28 MB total
> Receiving 6 files, 64.15 MB total. Already received 1 files, 12.14 MB total
> Sending 5 files, 61.15 MB total. Already sent 5 files, 61.15 MB total
> Receiving 4 files, 7.75 MB total. Already received 3 files, 7.58 MB total
> Sending 4 files, 4.29 MB total. Already sent 4 files, 4.29 MB total
> Receiving 12 files, 13.79 MB total. Already received 11 files, 7.66 MB total
> Sending 5 files, 15.32 MB total. Already sent 5 files, 15.32 MB total
> Receiving 8 files, 20.35 MB total. Already received 1 files, 13.63 MB total
> Sending 38 files, 125.34 MB total. Already sent 38 files, 125.34 MB total
> root@db1:~# date
> Sat Jan 9 17:45:42 UTC 2016
> root@db1:~# nt netstats -H | grep total
> Receiving 5 files, 46.59 MB total. Already received 1 files, 11.32 MB total
> Sending 7 files, 46.28 MB total. Already sent 7 files, 46.28 MB total
> Receiving 6 files, 64.15 MB total. Already received 1 files, 12.14 MB total
> Sending 5 files, 61.15 MB total. Already sent 5 files, 61.15 MB total
> Receiving 4 files, 7.75 MB total. Already received 3 files, 7.58 MB total
> Sending 4 files, 4.29 MB total. Already sent 4 files, 4.29 MB total
> Receiving 12 files, 13.79 MB total. Already received 11 files, 7.66 MB total
> Sending 5 files, 15.32 MB total. Already sent 5 files, 15.32 MB total
> Receiving 8 files, 20.35 MB total. Already received 1 files, 13.63 MB total
> Sending 38 files, 125.34 MB total. Already sent 38 files, 125.34 MB total
> {code}
> Such sessions are left even when repair job is long time done (confirmed by checking Reaper's and Cassandra's logs). {{streaming_socket_timeout_in_ms}} in cassandra.yaml is set to default value (3600000).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)