You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Aleksey Yeschenko (JIRA)" <ji...@apache.org> on 2015/11/11 17:58:10 UTC
[jira] [Resolved] (CASSANDRA-7240) Altering Keyspace Replication On
Large Cluster With vnodes Leads to Warns on All nodes
[ https://issues.apache.org/jira/browse/CASSANDRA-7240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aleksey Yeschenko resolved CASSANDRA-7240.
------------------------------------------
Resolution: Won't Fix
This seems to be related to schema propagation. We aren't changing anything there in 2.1 or 2.2 (or 3.0.x), and changes are already scheduled for 3.x, so closing this as won't fix.
> Altering Keyspace Replication On Large Cluster With vnodes Leads to Warns on All nodes
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-7240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-7240
> Project: Cassandra
> Issue Type: Bug
> Environment: 1000 Nodes M1.large ubuntu 12.04
> Reporter: Russell Alexander Spitzer
>
> 1000 Node cluster started with vnodes(256) on. 25 separate Nodes began an all write workload against the first 1000 nodes. During the test I attempted to alter the key-space from simple strategy to a network topology strategy.
> {code}
> cqlsh> ALTER KEYSPACE "Keyspace1" WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2':'3'} AND durable_writes = true;
> errors={}, last_host=127.0.0.1
> cqlsh> ALTER KEYSPACE "Keyspace1" WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '3', 'DC2':'3'} AND durable_writes = true;
> ('Unable to complete the operation against any hosts', {<Host: 127.0.0.1 DC1>: ConnectionShutdown('Connection to 127.0.0.1 is defunct',)})
> {code}
> All one thousand nodes then began to repeat the following in their respective logs
> {code}
> WARN [Thread-50131] 2014-05-14 23:34:07,631 IncomingTcpConnection.java:91 - UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=46b7b090-dbaf-11e3-8413-fffd4403e7d2
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164) ~[apache-cassandra-2.1.0-beta2.jar:2.1.0-beta2]
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97) ~[apache-cassandra-2.1.0-beta2.jar:2.1.0-beta2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:318) ~[apache-cassandra-2.1.0-beta2.jar:2.1.0-beta2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:298) ~[apache-cassandra-2.1.0-beta2.jar:2.1.0-beta2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:326) ~[apache-cassandra-2.1.0-beta2.jar:2.1.0-beta2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:268) ~[apache-cassandra-2.1.0-beta2.jar:2.1.0-beta2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.0-beta2.jar:2.1.0-beta2]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:165) ~[apache-cassandra-2.1.0-beta2.jar:2.1.0-beta2]
> at org.apache.cassandra.net.IncomingTcpConnection.handleModernVersion(IncomingTcpConnection.java:147) ~[apache-cassandra-2.1.0-beta2.jar:2.1.0-beta2]
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.0-beta2.jar:2.1.0-beta2]
> {code}
> Stress continued but at a decreased speed
> {code}
> Excerpt from one of the 25 Stress Nodes
> 83222847 , 14602, 14602, 6.7, 2.1, 23.1, 132.1, 292.3, 531.3, 5216.5, 0.00188
> 83239512 , 13888, 13888, 7.3, 2.1, 31.3, 129.9, 267.9, 555.8, 5217.7, 0.00188
> 83258520 , 14301, 14301, 7.0, 2.1, 28.8, 125.4, 297.2, 758.1, 5219.0, 0.00188
> 83277750 , 14023, 14023, 7.1, 2.1, 28.4, 132.8, 292.3, 703.6, 5220.4, 0.00188
> 83301413 , 14410, 14410, 6.9, 2.1, 24.5, 124.8, 391.4, 1010.1, 5222.0, 0.00188
> 83316846 , 12313, 12313, 8.1, 2.1, 35.1, 168.2, 275.3, 467.9, 5223.3, 0.00188
> 83332883 , 13753, 13753, 6.9, 2.1, 28.1, 132.2, 276.1, 498.9, 5224.4, 0.00188
> #ALTER REQUEST HERE
> 83351413 , 9981, 9981, 9.9, 2.1, 46.7, 172.0, 447.8, 1327.9, 5226.3, 0.00188
> 83358381 , 4464, 4464, 22.7, 2.2, 125.9, 257.8, 594.6, 1650.6, 5227.8, 0.00188
> 83363153 , 3186, 3186, 31.7, 2.5, 153.0, 300.3, 477.0, 566.1, 5229.3, 0.00189
> 83367341 , 2967, 2967, 33.7, 2.4, 173.9, 311.5, 465.8, 761.9, 5230.7, 0.00190
> 83370738 , 2392, 2392, 41.4, 2.9, 208.0, 308.1, 434.8, 839.6, 5232.2, 0.00191
> 83373651 , 2283, 2283, 43.0, 2.5, 213.9, 310.5, 409.3, 503.3, 5233.4, 0.00192
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)