[jira] [Commented] (CASSANDRA-6604) AssertionError: Verb COUNTER_MUTATION should not legally be dropped

["Eric Lubow (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-6604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13893996#comment-13893996 ] 

Eric Lubow commented on CASSANDRA-6604:
---------------------------------------

I'm not entirely sure this is cosmetic.  We have a 20 node cluster running 1.2.13.2 (DSE 3.2.4) and it is not under a heavy load at all.  We are seeing this simultaneously with tens of thousands of dropped mutations under higher traffic scenarios and just thousands of mutations dropped under lower traffic scenarios.  However, it is only happening on a single node.  There is nothing (visibly) wrong with that node other than higher CPU utilization that the other nodes.  Even with the higher CPU utilization, it is still not a stressed node and yet we are seeing the dropped MUTATIONS and this error message (again only on this node).

> AssertionError: Verb COUNTER_MUTATION should not legally be dropped
> -------------------------------------------------------------------
>
>                 Key: CASSANDRA-6604
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6604
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Bartłomiej Romański
>             Fix For: 1.2.14, 2.0.5
>
>
> We're seeing the following errors in our logs from time to time (about once an hour):
> {code}
> ERROR [MutationStage:10] 2014-01-19 10:36:26,659 CassandraDaemon.java (line 187) Exception in thread Thread[MutationStage:10,5,main]
> java.lang.AssertionError: Verb COUNTER_MUTATION should not legally be dropped
>         at org.apache.cassandra.net.MessagingService.incrementDroppedMessages(MessagingService.java:779)
>         at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1925)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:724)
> {code}
> They always appear in groups. About 50-100 errors in a row.
> We've got a 12-nodes cluster recently upgraded from 1.2.10 to 2.0.4. It's under pretty heavy load.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)