You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2011/03/04 00:46:37 UTC

[jira] Created: (CASSANDRA-2269) OOM in the Thrift thread doesn't shut down server

OOM in the Thrift thread doesn't shut down server
-------------------------------------------------

                 Key: CASSANDRA-2269
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2269
             Project: Cassandra
          Issue Type: Bug
          Components: API
    Affects Versions: 0.6
            Reporter: Jonathan Ellis
            Assignee: Jonathan Ellis
            Priority: Minor
             Fix For: 0.7.4


Example:

{noformat}
java.lang.OutOfMemoryError: Java heap space
        at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.resize(NonBlockingHashMap.java:849)
        at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.access$200(NonBlockingHashMap.java:699)
        at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:634)
        at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:339)
        at org.cliffc.high_scale_lib.NonBlockingHashMap.put(NonBlockingHashMap.java:302)
        at org.apache.cassandra.utils.ExpiringMap.put(ExpiringMap.java:112)
        at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:237)
        at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:305)
        at org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:386)
        at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:347)
        at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:92)
        at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:175)
        at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:254)
        at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:215)
        at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:1272)
        at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1166)
        at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
{noformat}

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (CASSANDRA-2269) OOM in the Thrift thread doesn't shut down server

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-2269:
--------------------------------------

    Remaining Estimate: 4h
     Original Estimate: 4h

> OOM in the Thrift thread doesn't shut down server
> -------------------------------------------------
>
>                 Key: CASSANDRA-2269
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2269
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>    Affects Versions: 0.6
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>            Priority: Minor
>             Fix For: 0.6.13, 0.7.4
>
>         Attachments: 2269-0.6.txt
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> Example:
> {noformat}
> java.lang.OutOfMemoryError: Java heap space
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.resize(NonBlockingHashMap.java:849)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.access$200(NonBlockingHashMap.java:699)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:634)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:339)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.put(NonBlockingHashMap.java:302)
>         at org.apache.cassandra.utils.ExpiringMap.put(ExpiringMap.java:112)
>         at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:237)
>         at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:305)
>         at org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:386)
>         at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:347)
>         at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:92)
>         at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:175)
>         at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:254)
>         at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:215)
>         at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:1272)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1166)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2269) OOM in the Thrift thread doesn't shut down server

Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007113#comment-13007113 ] 

T Jake Luciani commented on CASSANDRA-2269:
-------------------------------------------

+1

> OOM in the Thrift thread doesn't shut down server
> -------------------------------------------------
>
>                 Key: CASSANDRA-2269
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2269
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>    Affects Versions: 0.6
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>            Priority: Minor
>             Fix For: 0.7.5
>
>         Attachments: 2269-0.6.txt
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> Example:
> {noformat}
> java.lang.OutOfMemoryError: Java heap space
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.resize(NonBlockingHashMap.java:849)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.access$200(NonBlockingHashMap.java:699)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:634)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:339)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.put(NonBlockingHashMap.java:302)
>         at org.apache.cassandra.utils.ExpiringMap.put(ExpiringMap.java:112)
>         at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:237)
>         at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:305)
>         at org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:386)
>         at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:347)
>         at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:92)
>         at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:175)
>         at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:254)
>         at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:215)
>         at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:1272)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1166)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Commented: (CASSANDRA-2269) OOM in the Thrift thread doesn't shut down server

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008584#comment-13008584 ] 

Hudson commented on CASSANDRA-2269:
-----------------------------------

Integrated in Cassandra-0.6 #63 (See [https://hudson.apache.org/hudson/job/Cassandra-0.6/63/])
    

> OOM in the Thrift thread doesn't shut down server
> -------------------------------------------------
>
>                 Key: CASSANDRA-2269
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2269
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>    Affects Versions: 0.6
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>            Priority: Minor
>             Fix For: 0.6.13, 0.7.5
>
>         Attachments: 2269-0.6.txt
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> Example:
> {noformat}
> java.lang.OutOfMemoryError: Java heap space
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.resize(NonBlockingHashMap.java:849)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.access$200(NonBlockingHashMap.java:699)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:634)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:339)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.put(NonBlockingHashMap.java:302)
>         at org.apache.cassandra.utils.ExpiringMap.put(ExpiringMap.java:112)
>         at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:237)
>         at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:305)
>         at org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:386)
>         at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:347)
>         at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:92)
>         at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:175)
>         at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:254)
>         at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:215)
>         at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:1272)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1166)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {noformat}

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] Updated: (CASSANDRA-2269) OOM in the Thrift thread doesn't shut down server

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-2269:
--------------------------------------

    Attachment: 2269-0.6.txt

patch to extract exception logging from DTPE and call from the Thrift executor.

(the actual shutdown is done by the default exception hook we set up -- it's not normally called on executor threads because both Future and executor afterExecute machinery swallow exceptions.)

> OOM in the Thrift thread doesn't shut down server
> -------------------------------------------------
>
>                 Key: CASSANDRA-2269
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2269
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>    Affects Versions: 0.6
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>            Priority: Minor
>             Fix For: 0.6.13, 0.7.4
>
>         Attachments: 2269-0.6.txt
>
>
> Example:
> {noformat}
> java.lang.OutOfMemoryError: Java heap space
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.resize(NonBlockingHashMap.java:849)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.access$200(NonBlockingHashMap.java:699)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:634)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:339)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.put(NonBlockingHashMap.java:302)
>         at org.apache.cassandra.utils.ExpiringMap.put(ExpiringMap.java:112)
>         at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:237)
>         at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:305)
>         at org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:386)
>         at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:347)
>         at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:92)
>         at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:175)
>         at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:254)
>         at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:215)
>         at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:1272)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1166)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Updated: (CASSANDRA-2269) OOM in the Thrift thread doesn't shut down server

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/CASSANDRA-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Ellis updated CASSANDRA-2269:
--------------------------------------

    Fix Version/s: 0.6.13

> OOM in the Thrift thread doesn't shut down server
> -------------------------------------------------
>
>                 Key: CASSANDRA-2269
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2269
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>    Affects Versions: 0.6
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>            Priority: Minor
>             Fix For: 0.6.13, 0.7.4
>
>
> Example:
> {noformat}
> java.lang.OutOfMemoryError: Java heap space
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.resize(NonBlockingHashMap.java:849)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.access$200(NonBlockingHashMap.java:699)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:634)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:339)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.put(NonBlockingHashMap.java:302)
>         at org.apache.cassandra.utils.ExpiringMap.put(ExpiringMap.java:112)
>         at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:237)
>         at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:305)
>         at org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:386)
>         at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:347)
>         at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:92)
>         at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:175)
>         at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:254)
>         at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:215)
>         at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:1272)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1166)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (CASSANDRA-2269) OOM in the Thrift thread doesn't shut down server

Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002443#comment-13002443 ] 

Jonathan Ellis commented on CASSANDRA-2269:
-------------------------------------------

Adding ByteBuffer.allocateDirect(256 * 1024 * 1024) to any of the CassandraServer methods should do it

> OOM in the Thrift thread doesn't shut down server
> -------------------------------------------------
>
>                 Key: CASSANDRA-2269
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2269
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>    Affects Versions: 0.6
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>            Priority: Minor
>             Fix For: 0.6.13, 0.7.4
>
>         Attachments: 2269-0.6.txt
>
>
> Example:
> {noformat}
> java.lang.OutOfMemoryError: Java heap space
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.resize(NonBlockingHashMap.java:849)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.access$200(NonBlockingHashMap.java:699)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:634)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:339)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.put(NonBlockingHashMap.java:302)
>         at org.apache.cassandra.utils.ExpiringMap.put(ExpiringMap.java:112)
>         at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:237)
>         at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:305)
>         at org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:386)
>         at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:347)
>         at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:92)
>         at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:175)
>         at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:254)
>         at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:215)
>         at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:1272)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1166)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] Commented: (CASSANDRA-2269) OOM in the Thrift thread doesn't shut down server

Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/CASSANDRA-2269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002441#comment-13002441 ] 

T Jake Luciani commented on CASSANDRA-2269:
-------------------------------------------

Makes sense... Is there any known way to reproduce it to verify the fix infact works?

> OOM in the Thrift thread doesn't shut down server
> -------------------------------------------------
>
>                 Key: CASSANDRA-2269
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-2269
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>    Affects Versions: 0.6
>            Reporter: Jonathan Ellis
>            Assignee: Jonathan Ellis
>            Priority: Minor
>             Fix For: 0.6.13, 0.7.4
>
>         Attachments: 2269-0.6.txt
>
>
> Example:
> {noformat}
> java.lang.OutOfMemoryError: Java heap space
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.resize(NonBlockingHashMap.java:849)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap$CHM.access$200(NonBlockingHashMap.java:699)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:634)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.putIfMatch(NonBlockingHashMap.java:339)
>         at org.cliffc.high_scale_lib.NonBlockingHashMap.put(NonBlockingHashMap.java:302)
>         at org.apache.cassandra.utils.ExpiringMap.put(ExpiringMap.java:112)
>         at org.apache.cassandra.net.MessagingService.addCallback(MessagingService.java:237)
>         at org.apache.cassandra.net.MessagingService.sendRR(MessagingService.java:305)
>         at org.apache.cassandra.service.StorageProxy.weakRead(StorageProxy.java:386)
>         at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:347)
>         at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:92)
>         at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:175)
>         at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:254)
>         at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:215)
>         at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:1272)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1166)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira