[jira] [Commented] (CASSANDRA-6614) 2 hours loop flushing+compacting system/{schema_keyspaces,schema_columnfamilies,schema_columns} when upgrading

["Jason Brown (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-6614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13891436#comment-13891436 ] 

Jason Brown commented on CASSANDRA-6614:
----------------------------------------

[~cscetbon] I've been told "100% we're doing drain", but still running into this. What we do notice is that our clusters are multi-DC, and the problem only seems to occur within the the region we are upgrading (the other DCs remain unaffected wrt read/write latency). Further, we are using the Ec2MultiRegionSnitch. 

[~cscetbon] Are you running a multi-DC cluster, and/or are you using a 'reconnectable snitch, like Ec2MultiRegionSnitch?

> 2 hours loop flushing+compacting system/{schema_keyspaces,schema_columnfamilies,schema_columns} when upgrading
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-6614
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6614
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: ubuntu 12.04
>            Reporter: Cyril Scetbon
>
> It happens when we upgrade one node to 1.2.13 on a 1.2.2 cluster 
> see http://pastebin.com/YZKUQLXz
> If I grep for only InternalResponseStage logs I get http://pastebin.com/htnXZCiT which always displays same account of ops and serialized/live bytes per column family.
> When I upgrade one node from 1.2.2 to 1.2.13, for 2h I get the previous messages with a raise of CPU (as it flushes and compacts continually) on all nodes http://picpaste.com/pics/Screen_Shot_2014-01-24_at_09.18.50-ggcCDVqd.1390587562.png
> After that, everything is fine and I can upgrade other nodes without any raise of cpus load. when I start the upgrade, the more nodes I upgrade at the same time (at the beginning), the higher the cpu load is http://picpaste.com/pics/Screen_Shot_2014-01-23_at_17.45.56-I3fdEQ2T.1390587597.png



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)