[jira] [Created] (CASSANDRA-7262) During streaming: java.lang.AssertionError: Reference counter -1

["Duncan Sands (JIRA)" <ji...@apache.org>]

Duncan Sands created CASSANDRA-7262:
---------------------------------------

             Summary: During streaming: java.lang.AssertionError: Reference counter -1
                 Key: CASSANDRA-7262
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7262
             Project: Cassandra
          Issue Type: Bug
          Components: Core
         Environment: Cassandra 2.07, x86-64 Ubuntu 12.04.4, Oracle java 1.7.0_45
            Reporter: Duncan Sands
            Priority: Minor
         Attachments: system.log.gz

Got this assertion failure this weekend during repair:

ERROR [STREAM-IN-/192.168.21.14] 2014-05-17 01:17:52,332 StreamSession.java (line 420) [Stream #3a3ac8a2-dd50-11e3-b3c1-6bf6dccd6457] Streaming error occurred
java.lang.RuntimeException: Outgoing stream handler has been closed
        at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:170)
        at org.apache.cassandra.streaming.StreamSession.receive(StreamSession.java:483)
        at org.apache.cassandra.streaming.StreamSession.messageReceived(StreamSession.java:372)
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:289)
        at java.lang.Thread.run(Thread.java:744)
ERROR [STREAM-IN-/192.168.21.14] 2014-05-17 01:17:52,350 CassandraDaemon.java (line 198) Exception in thread Thread[STREAM-IN-/192.168.21.14,5,RMI Runtime]
java.lang.AssertionError: Reference counter -1 for /mnt/ssd1/cassandra/data/ldn_production/historical_accounts/ldn_production-historical_accounts-jb-79827-Data.db
        at org.apache.cassandra.io.sstable.SSTableReader.releaseReference(SSTableReader.java:1107)
        at org.apache.cassandra.streaming.StreamTransferTask.abort(StreamTransferTask.java:80)
        at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:322)
        at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:425)
        at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:300)
        at java.lang.Thread.run(Thread.java:744)

followed by a few more (the reference counter got down to -3).  Got the same kind of assertion failure on one other node (in a different data centre; there are 21 nodes altogether distributed over 4 data centres).

I've attached the relevant part of the log.  It starts quite a bit before the assertion failure at the first exception on this node ("Cannot proceed on repair because a neighbor ... is dead"), and finishes a few hours afterwards when the node was restarted.



--
This message was sent by Atlassian JIRA
(v6.2#6252)