You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Paulo Motta (JIRA)" <ji...@apache.org> on 2016/06/23 20:59:17 UTC
[jira] [Commented] (CASSANDRA-10992) Hanging streaming sessions
[ https://issues.apache.org/jira/browse/CASSANDRA-10992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347174#comment-15347174 ]
Paulo Motta commented on CASSANDRA-10992:
-----------------------------------------
I created a [dtest|https://github.com/riptano/cassandra-dtest/compare/master...pauloricardomg:10992#diff-0b30b9f097df89d74be1d1af8205ac7eR147] to reproduce this with the following sequence of steps:
- create 2 node cluster with data on compressed table
- bootstrap new node
- fail one of the original nodes during bootstrap
- expect bootstrap to fail
Currently bootstrap never completes or fails because when {{CompressedStreamReader}} gets an {{IOException}} due to the node failure it tries to drain the {{CompressedInputStream}} for a possible retry, but the data buffer is never filled by the {{Reader}} thread because the socket is closed, so the stream session blocks on drain and never completes.
Since we never retry on {{IOException}} the fix is to avoid draining the socket on {{IOException}}. Furthermore, I updated {{CompressedInputStream}} to cache the exception and re-throw it in case someone tries to read it again from the failed input stream.
The patch and CI runs are available below:
||2.1||2.2||3.0||trunk||dtest||
|[branch|https://github.com/apache/cassandra/compare/cassandra-2.1...pauloricardomg:2.1-10992]|[branch|https://github.com/apache/cassandra/compare/cassandra-2.2...pauloricardomg:2.2-10992]|[branch|https://github.com/apache/cassandra/compare/cassandra-3.0...pauloricardomg:3.0-10992]|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-10992]|[branch|https://github.com/riptano/cassandra-dtest/compare/master...pauloricardomg:10992]|
|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-10992-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-10992-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-10992-testall/lastCompletedBuild/testReport/]|[testall|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-10992-testall/lastCompletedBuild/testReport/]|
|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.1-10992-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-2.2-10992-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-3.0-10992-dtest/lastCompletedBuild/testReport/]|[dtest|http://cassci.datastax.com/view/Dev/view/paulomotta/job/pauloricardomg-trunk-10992-dtest/lastCompletedBuild/testReport/]|
On a related note, it's not very clear to me from what conditions retry is trying to recover from, and draining based on consumed size is a bit fragile since we may block on the socket/CIS if there is a size mismatch due to a bug (a la CASSANDRA-10005). I think it's safer to drain based on a magic number, so we should probably consider that in the future. With that said, I wonder if we should disable retry altogether because I think that in its current state it brings more pain than benefits.
> Hanging streaming sessions
> --------------------------
>
> Key: CASSANDRA-10992
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10992
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.12, Debian Wheezy
> Reporter: mlowicki
> Assignee: Paulo Motta
> Fix For: 2.1.12
>
> Attachments: apache-cassandra-2.1.12-SNAPSHOT.jar, db1.ams.jstack, db6.analytics.jstack
>
>
> I've started recently running repair using [Cassandra Reaper|https://github.com/spotify/cassandra-reaper] (built-in {{nodetool repair}} doesn't work for me - CASSANDRA-9935). It behaves fine but I've noticed hanging streaming sessions:
> {code}
> root@db1:~# date
> Sat Jan 9 16:43:00 UTC 2016
> root@db1:~# nt netstats -H | grep total
> Receiving 5 files, 46.59 MB total. Already received 1 files, 11.32 MB total
> Sending 7 files, 46.28 MB total. Already sent 7 files, 46.28 MB total
> Receiving 6 files, 64.15 MB total. Already received 1 files, 12.14 MB total
> Sending 5 files, 61.15 MB total. Already sent 5 files, 61.15 MB total
> Receiving 4 files, 7.75 MB total. Already received 3 files, 7.58 MB total
> Sending 4 files, 4.29 MB total. Already sent 4 files, 4.29 MB total
> Receiving 12 files, 13.79 MB total. Already received 11 files, 7.66 MB total
> Sending 5 files, 15.32 MB total. Already sent 5 files, 15.32 MB total
> Receiving 8 files, 20.35 MB total. Already received 1 files, 13.63 MB total
> Sending 38 files, 125.34 MB total. Already sent 38 files, 125.34 MB total
> root@db1:~# date
> Sat Jan 9 17:45:42 UTC 2016
> root@db1:~# nt netstats -H | grep total
> Receiving 5 files, 46.59 MB total. Already received 1 files, 11.32 MB total
> Sending 7 files, 46.28 MB total. Already sent 7 files, 46.28 MB total
> Receiving 6 files, 64.15 MB total. Already received 1 files, 12.14 MB total
> Sending 5 files, 61.15 MB total. Already sent 5 files, 61.15 MB total
> Receiving 4 files, 7.75 MB total. Already received 3 files, 7.58 MB total
> Sending 4 files, 4.29 MB total. Already sent 4 files, 4.29 MB total
> Receiving 12 files, 13.79 MB total. Already received 11 files, 7.66 MB total
> Sending 5 files, 15.32 MB total. Already sent 5 files, 15.32 MB total
> Receiving 8 files, 20.35 MB total. Already received 1 files, 13.63 MB total
> Sending 38 files, 125.34 MB total. Already sent 38 files, 125.34 MB total
> {code}
> Such sessions are left even when repair job is long time done (confirmed by checking Reaper's and Cassandra's logs). {{streaming_socket_timeout_in_ms}} in cassandra.yaml is set to default value (3600000).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)