[jira] [Created] (CASSANDRA-5083) node locks up when streamed hinted handoffs and sending node dies.

["Terrance Shepherd (JIRA)" <ji...@apache.org>]

Terrance Shepherd created CASSANDRA-5083:
--------------------------------------------

             Summary: node locks up when streamed hinted handoffs and sending node dies.
                 Key: CASSANDRA-5083
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5083
             Project: Cassandra
          Issue Type: Bug
            Reporter: Terrance Shepherd
         Attachments: failure.log

Last night 2 nodes appeared to have died do to an issues while one node was streaming hinted hand offs to another node. Attached is the last 100 lines from the node that was receiving hints. The version of Cassandra is 1.1.6.

The order of events appeared to be
1) a node was unavailable ( unknown reason ) for a period of time causing hinted hand offs to back up on another node.
2) the node that went down came back on line and hinted hand offs started transferring.
3) the node holding the hints's disk space grew exponentially and filled up the data directory holding the hinted hand off column family. the error long is full of unable to compact hinted column family keyspaces.
4) the node holding the hints became unavailable.
5) the node being streamed the hints locked up. cassandra was still "running" but not responding to requests, gossip or restarts

Terrance
Developer
Dynamic Network Services, Inc.
http://www.dyn.com


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira