You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Alexey Burylov (JIRA)" <ji...@apache.org> on 2015/08/27 18:13:45 UTC
[jira] [Comment Edited] (CASSANDRA-10206) Incorrect handling of
end-of stream leading to infinite loop in streaming session
[ https://issues.apache.org/jira/browse/CASSANDRA-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716958#comment-14716958 ]
Alexey Burylov edited comment on CASSANDRA-10206 at 8/27/15 4:13 PM:
---------------------------------------------------------------------
{quote}
Are you observing streaming hang when this happens?
{quote}
Streaming session visible in JMX monitoring and node-tool unity:
For thread 17365 and 17633 of threaddump-1440686737505.tdump
{code}
nodetool **** netstats | grep -B1 "10.8.6.84"
Repair 834af7d0-4cb8-11e5-813b-273d25d97b13
/10.8.6.84
Receiving 3 files, 141990 bytes total. Already received 3 files, 141990 bytes total
***-tmp-ka-77703-Data.db 66507/66507 bytes(100%) received from idx:0/10.8.6.84
***-tmp-ka-77702-Data.db 55758/55758 bytes(100%) received from idx:0/10.8.6.84
***-tmp-ka-77701-Data.db 19725/19725 bytes(100%) received from idx:0/10.8.6.84
Sending 3 files, 151858 bytes total. Already sent 3 files, 151858 bytes total
***-ka-59374-Data.db 20716/20716 bytes(100%) sent to idx:0/10.8.6.84
***-ka-75986-Data.db 75224/75224 bytes(100%) sent to idx:0/10.8.6.84
***-ka-53876-Data.db 55918/55918 bytes(100%) sent to idx:0/10.8.6.84
--
Repair 7b088ec1-4cb8-11e5-813b-273d25d97b13
/10.8.6.84
Receiving 3 files, 124841 bytes total. Already received 3 files, 124841 bytes total
***-tmp-ka-495105-Data.db 49758/49758 bytes(100%) received from idx:0/10.8.6.84
***-tmp-ka-495104-Data.db 24711/24711 bytes(100%) received from idx:0/10.8.6.84
***-tmp-ka-495106-Data.db 50372/50372 bytes(100%) received from idx:0/10.8.6.84
Sending 4 files, 175697 bytes total. Already sent 4 files, 175697 bytes total
***-ka-489513-Data.db 49255/49255 bytes(100%) sent to idx:0/10.8.6.84
***-ka-483689-Data.db 74596/74596 bytes(100%) sent to idx:0/10.8.6.84
***-ka-494198-Data.db 25941/25941 bytes(100%) sent to idx:0/10.8.6.84
***-ka-493844-Data.db 25905/25905 bytes(100%) sent to idx:0/10.8.6.84
{code}
{quote}
Are you observing streaming hang when this happens?
{quote}
Yes it hang on 100%
{quote}
What does other end of stream (thread dump) looks like?
{quote}
I try to capture form live node. At now, i can say what stream session continue even after other site node rebooted.
was (Author: a.burylov):
{qoute}
Are you observing streaming hang when this happens?
{qoute}
Streaming session visible in JMX monitoring and node-tool unity:
For thread 17365 and 17633 of threaddump-1440686737505.tdump
{code}
nodetool **** netstats | grep -B1 "10.8.6.84"
Repair 834af7d0-4cb8-11e5-813b-273d25d97b13
/10.8.6.84
Receiving 3 files, 141990 bytes total. Already received 3 files, 141990 bytes total
***-tmp-ka-77703-Data.db 66507/66507 bytes(100%) received from idx:0/10.8.6.84
***-tmp-ka-77702-Data.db 55758/55758 bytes(100%) received from idx:0/10.8.6.84
***-tmp-ka-77701-Data.db 19725/19725 bytes(100%) received from idx:0/10.8.6.84
Sending 3 files, 151858 bytes total. Already sent 3 files, 151858 bytes total
***-ka-59374-Data.db 20716/20716 bytes(100%) sent to idx:0/10.8.6.84
***-ka-75986-Data.db 75224/75224 bytes(100%) sent to idx:0/10.8.6.84
***-ka-53876-Data.db 55918/55918 bytes(100%) sent to idx:0/10.8.6.84
--
Repair 7b088ec1-4cb8-11e5-813b-273d25d97b13
/10.8.6.84
Receiving 3 files, 124841 bytes total. Already received 3 files, 124841 bytes total
***-tmp-ka-495105-Data.db 49758/49758 bytes(100%) received from idx:0/10.8.6.84
***-tmp-ka-495104-Data.db 24711/24711 bytes(100%) received from idx:0/10.8.6.84
***-tmp-ka-495106-Data.db 50372/50372 bytes(100%) received from idx:0/10.8.6.84
Sending 4 files, 175697 bytes total. Already sent 4 files, 175697 bytes total
***-ka-489513-Data.db 49255/49255 bytes(100%) sent to idx:0/10.8.6.84
***-ka-483689-Data.db 74596/74596 bytes(100%) sent to idx:0/10.8.6.84
***-ka-494198-Data.db 25941/25941 bytes(100%) sent to idx:0/10.8.6.84
***-ka-493844-Data.db 25905/25905 bytes(100%) sent to idx:0/10.8.6.84
{code}
{qoute}
Are you observing streaming hang when this happens?
{qoute}
Yes it hang on 100%
{qoute}
What does other end of stream (thread dump) looks like?
{qoute}
I try to capture form live node. At now, i can say what stream session continue even after other site node rebooted.
> Incorrect handling of end-of stream leading to infinite loop in streaming session
> ---------------------------------------------------------------------------------
>
> Key: CASSANDRA-10206
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10206
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: cassndra 2.1.8 12 nodes cluster in two DC, with enabled ssl encryption.
> Reporter: Alexey Burylov
> Fix For: 2.1.x
>
> Attachments: cassandra-2.1-10206.txt, threaddump-1440601628223.tdump, threaddump-1440602926719.tdump, threaddump-1440672821181.tdump, threaddump-1440686737505.tdump
>
>
> StreamMessage.deserialize (StreamMessage.java:63) function return null when receive end of stream for ReadableByteChannel.read () function. IncomingMessageHandler.run (ConnectionHandler.java:257) interpreting null - as unsupported message and ignore it. It resulting infinite loop when IncomingMessageHandler try to read message from closed steram, and deserialize always return null - to indicate close stream.
> It not happen on normal socket, because when normal socket closed it throw IOException. But happen from time to time when iner-node ssl encryption enabled.
> I'm attach two tread dump from one node with two infinity lopped thread with id 614 and 583
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)