You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Bryce Godfrey (JIRA)" <ji...@apache.org> on 2012/08/19 06:32:38 UTC
[jira] [Commented] (CASSANDRA-4323) Joining new node to cluster
fails with error in add column family
[ https://issues.apache.org/jira/browse/CASSANDRA-4323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437454#comment-13437454 ]
Bryce Godfrey commented on CASSANDRA-4323:
------------------------------------------
What if I'm having this issue currently and I'm on 1.1.0 ? Do you think an upgrade to 1.1.3/4 will fix this and allow me to join new nodes again? What got me into this situation is that we pointed Opscenter at this cluster and it just went nuts after that. Some of the nodes were spamming errors about KS Opscenter does not exist, and killed my cluster. I removed opscenter and its agents, tried to do a drop keyspace on Opscenter (which is says does not exist), but now I'm here without an idea of how to fix the schema corruption.
I will try upgrading to latest though and keep my fingers crossed.
> Joining new node to cluster fails with error in add column family
> -----------------------------------------------------------------
>
> Key: CASSANDRA-4323
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4323
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.1.1
> Environment: CentOS 6, Java 1.6
> Reporter: Bryce Godfrey
>
> I tried joining a new node to the cluster, and before boostrap started it reported this error:
> INFO 08:20:51,584 Enqueuing flush of Memtable-schema_columns@1493418651(0/0 serialized/live bytes, 1 ops)
> INFO 08:20:51,584 Writing Memtable-schema_columns@1493418651(0/0 serialized/live bytes, 1 ops)
> INFO 08:20:51,589 Completed flushing /opt/cassandra/data/system/schema_columns/system-schema_columns-hc-1-Data.db (61 bytes)
> ERROR 08:20:51,889 Exception in thread Thread[MigrationStage:1,5,main]
> java.lang.IllegalArgumentException: value already present: 1015
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:115)
> at com.google.common.collect.AbstractBiMap.putInBothMaps(AbstractBiMap.java:111)
> at com.google.common.collect.AbstractBiMap.put(AbstractBiMap.java:96)
> at com.google.common.collect.HashBiMap.put(HashBiMap.java:84)
> at org.apache.cassandra.config.Schema.load(Schema.java:385)
> at org.apache.cassandra.db.DefsTable.addColumnFamily(DefsTable.java:426)
> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:361)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:270)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:248)
> at org.apache.cassandra.service.MigrationManager$MigrationTask.runMayThrow(MigrationManager.java:416)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> INFO 08:20:51,931 Enqueuing flush of Memtable-schema_keyspaces@833041663(943/1178 serialized/live bytes, 20 ops)
> INFO 08:20:51,932 Writing Memtable-schema_keyspaces@833041663(943/1178 serialized/live bytes, 20 ops)
> And continued on, then started writing these errors non-stop:
> ERROR 08:21:45,959 Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1019
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
> at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
> at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> ERROR 08:21:45,814 Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1019
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
> at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
> at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> ERROR 08:21:45,813 Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=1020
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
> at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
> at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> ERROR 08:21:45,813 Error in row mutation
> I had a problem a while ago of someone trying to create a new column family while a node was hung. The new node never picked up the new column family, so we deleted it and tried again and everything was fine. Not sure if its related though.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira