You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (JIRA)" <ji...@apache.org> on 2015/01/15 17:55:35 UTC
[jira] [Commented] (CASSANDRA-8628) Stalling rebuild of new 2.1.2
node from cluster running 2.0.9
[ https://issues.apache.org/jira/browse/CASSANDRA-8628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278936#comment-14278936 ]
Brandon Williams commented on CASSANDRA-8628:
---------------------------------------------
bq. My reading of the upgrading compatibility document leads me to think that there should not be any problem in performing this kind of rebuild.
Actually, there is. Since the new node is 2.1, it won't pull schema from 2.0, which leads to your "Couldn't find cfId" problems. You should start the new node on 2.0, let it get the schema, then upgrade it to 2.1.
> Stalling rebuild of new 2.1.2 node from cluster running 2.0.9
> -------------------------------------------------------------
>
> Key: CASSANDRA-8628
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8628
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 14.04, 4 core machines, 8 and 16 GB RAM
> Cluster running cassandra 2.0.9, new node running 2.1.2,
> inter-node encryption: all
> Reporter: Lennart Liberg
>
> We want to add a new datacenter to our existing cluster consisting of two datacenters containing 2 nodes and 1 node respectively, running version 2.0.9
> We configured a node running cassandra 2.1.2 in the new datacenter (auto_bootstrap: false) and verified that it managed to join the cluster as expected (`nodetool status` saying its status was UN).
> Upon executing `nodetool rebuild ams1`, the system.log output below is observed, with a new UnknownColumnFamilyException and NullPointerException every few minutes, and no data being transfered.
> After downgrading to cassandra 2.0.11, the rebuild operation worked flawlessly, using a cassandra.yaml configured as identical as possible to the 2.1.2 yaml file (a lot of new items in 2.1.x).
> My reading of the upgrading compatibility document leads me to think that there should not be any problem in performing this kind of rebuild.
> http://www.datastax.com/documentation/upgrade/doc/upgrade/cassandra/upgradeDebC_t.html
> system.log output:
> {noformat}
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,228 StorageService.java:958 - rebuild from dc: ams1
> INFO [RMI TCP Connection(57622)-10.240.12.239] 2015-01-15 16:00:04,305 StreamResultFuture.java:86 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Executing streaming plan for Rebuild
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,306 StreamSession.java:213 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342] Starting streaming to 37.xx.xx.1/37.xx.xx.1
> INFO [StreamConnectionEstablisher:4] 2015-01-15 16:00:04,610 StreamCoordinator.java:209 - [Stream #91906440-9ccf-11e4-9cab-67625edb9342, ID#0] Beginning stream session with 37.xx.xx.1/37.xx.xx.1
> WARN [Thread-28935] 2015-01-15 16:03:40,458 IncomingTcpConnection.java:91 - UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-28974] 2015-01-15 16:05:54,164 CassandraDaemon.java:153 - Exception in thread Thread[Thread-28974,5,main]
> java.lang.NullPointerException: null
> at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-29011] 2015-01-15 16:07:58,349 CassandraDaemon.java:153 - Exception in thread Thread[Thread-29011,5,main]
> java.lang.NullPointerException: null
> at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.2.jar:2.1.2]
> ERROR [Thread-29013] 2015-01-15 16:08:01,037 CassandraDaemon.java:153 - Exception in thread Thread[Thread-29013,5,main]
> java.lang.NullPointerException: null
> at org.apache.cassandra.db.SliceFromReadCommandSerializer.deserialize(SliceFromReadCommand.java:153) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:157) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:131) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.2.jar:2.1.2]
> WARN [Thread-29065] 2015-01-15 16:10:56,374 IncomingTcpConnection.java:91 - UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=d1ea009c-feba-3d75-aa64-d022270842c4
> at org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150) ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82) ~[apache-cassandra-2.1.2.jar:2.1.2]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)