You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2012/12/22 16:25:13 UTC
[jira] [Commented] (CASSANDRA-5083) node locks up when streamed
hinted handoffs and sending node dies.
[ https://issues.apache.org/jira/browse/CASSANDRA-5083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13538826#comment-13538826 ]
Jonathan Ellis commented on CASSANDRA-5083:
-------------------------------------------
"Hint streaming" is just normal writes as far as the recipient is concerned; I'm not sure how that could cause a lockup. Is this reproducible?
> node locks up when streamed hinted handoffs and sending node dies.
> ------------------------------------------------------------------
>
> Key: CASSANDRA-5083
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5083
> Project: Cassandra
> Issue Type: Bug
> Reporter: Terrance Shepherd
> Attachments: failure.log
>
>
> Last night 2 nodes appeared to have died do to an issues while one node was streaming hinted hand offs to another node. Attached is the last 100 lines from the node that was receiving hints. The version of Cassandra is 1.1.6.
> The order of events appeared to be
> 1) a node was unavailable ( unknown reason ) for a period of time causing hinted hand offs to back up on another node.
> 2) the node that went down came back on line and hinted hand offs started transferring.
> 3) the node holding the hints's disk space grew exponentially and filled up the data directory holding the hinted hand off column family. the error long is full of unable to compact hinted column family keyspaces.
> 4) the node holding the hints became unavailable.
> 5) the node being streamed the hints locked up. cassandra was still "running" but not responding to requests, gossip or restarts
> Terrance
> Developer
> Dynamic Network Services, Inc.
> http://www.dyn.com
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira