[jira] [Comment Edited] (CASSANDRA-6614) 2 hours loop flushing+compacting system/{schema_keyspaces,schema_columnfamilies,schema_columns} when upgrading

["Jason Brown (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-6614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881182#comment-13881182 ] 

Jason Brown edited comment on CASSANDRA-6614 at 1/24/14 5:32 PM:
-----------------------------------------------------------------

We just started our upgrade of 1.1.7 to 1.2.12, and are seeing this, as well. It seems related to MigrationManager/MigrationTask where a node is endlessly looping getting retrieving the following system keyspaces from peers: schema_columnfamilies, schema_columns, schema_keyspaces. Once the entire cluster is upgraded, however, this behavior subsides.


was (Author: jasobrown):
We just started our upgrade of 1.1.7 to 1.2.12, and are seeing this, as well. It seems related to MigrationManager/MigrationTask where a node is endlessly looping getting retrieving the following system keyspaces from peers: schema_columnfamilies, schema_columns, schema_keyspaces

> 2 hours loop flushing+compacting system/{schema_keyspaces,schema_columnfamilies,schema_columns} when upgrading
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-6614
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6614
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: ubuntu 12.04
>            Reporter: Cyril Scetbon
>
> It happens when we upgrade one node to 1.2.13 on a 1.2.2 cluster 
> see http://pastebin.com/YZKUQLXz
> If I grep for only InternalResponseStage logs I get http://pastebin.com/htnXZCiT which always displays same account of ops and serialized/live bytes per column family.
> When I upgrade one node from 1.2.2 to 1.2.13, for 2h I get the previous messages with a raise of CPU (as it flushes and compacts continually) on all nodes http://picpaste.com/pics/Screen_Shot_2014-01-24_at_09.18.50-ggcCDVqd.1390551670.png
> After that, everything is fine and I can upgrade other nodes without any raise of cpus load. when I start the upgrade, the more nodes I upgrade at the same time (at the beginning), the higher the cpu load is http://picpaste.com/pics/Screen_Shot_2014-01-23_at_17.45.56-I3fdEQ2T.1390552036.png



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)