You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Hiller, Dean" <De...@nrel.gov> on 2012/09/26 18:04:34 UTC
any ideas on what these mean
We were consistently getting this exception over and over as we put data into the system. A reboot caused it to go away but we don't want to be rebooting in the future….
1. When does this occur?
2. Is it affecting my data put? (I have seen other weird validation exceptions where my data is still put and I can read it from cassandra and I get no exception client side)
3. How do I reverse engineer what column familsy 13740 and 13739 are? (ie. Their names?) so I can check for data corruption.
ERROR [MigrationStage:1] 2012-09-26 09:51:03,128 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]
java.lang.RuntimeException: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Column family ID mismatch (found 13740; expected 13739)
at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Column family ID mismatch (found 13740; expected 13739)
at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:676)
at org.apache.cassandra.db.DefsTable.updateColumnFamily(DefsTable.java:463)
at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:407)
at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:271)
at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:249)
at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:48)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
... 6 more
Caused by: org.apache.cassandra.config.ConfigurationException: Column family ID mismatch (found 13740; expected 13739)
at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:698)
at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:672) ... 12 more
We also see this exception in the same log which is ironic considering the above one says found 13740!!!!!! And this one says couldn't find it….
ERROR [MutationStage:27379] 2012-09-26 09:50:57,558 RowMutationVerbHandler.java (line 61) Error in row mutation
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=13740
at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Re: any ideas on what these mean
Posted by aaron morton <aa...@thelastpickle.com>.
> ERROR [MigrationStage:1] 2012-09-26 09:51:03,128 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]
I *think* this is due to concurrent schema modifications.
The found number is the one in the database for the matching KS / CF name, the second number is the one in memory at the time. That's why the reboot fixed it.
If it happens again try nodetool resetlocalschema
> 2. Is it affecting my data put? (I have seen other weird validation exceptions where my data is still put and I can read it from cassandra and I get no exception client side)
This write was dropped
> ERROR [MutationStage:27379] 2012-09-26 09:50:57,558 RowMutationVerbHandler.java (line 61) Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=13740
I would make sure repair is running to resolve any inconsistencies created during the period of wackyness.
> 3. How do I reverse engineer what column familsy 13740 and 13739 are? (ie. Their names?) so I can check for data corruption.
On 1.1 X schema is stored in the schema_columnfamilies CF in t he system KS. In previous versions it's in a different CF in the same KS.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 27/09/2012, at 4:04 AM, "Hiller, Dean" <De...@nrel.gov> wrote:
> We were consistently getting this exception over and over as we put data into the system. A reboot caused it to go away but we don't want to be rebooting in the future….
>
> 1. When does this occur?
> 2. Is it affecting my data put? (I have seen other weird validation exceptions where my data is still put and I can read it from cassandra and I get no exception client side)
> 3. How do I reverse engineer what column familsy 13740 and 13739 are? (ie. Their names?) so I can check for data corruption.
>
> ERROR [MigrationStage:1] 2012-09-26 09:51:03,128 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]
> java.lang.RuntimeException: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Column family ID mismatch (found 13740; expected 13739)
> at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Column family ID mismatch (found 13740; expected 13739)
> at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:676)
> at org.apache.cassandra.db.DefsTable.updateColumnFamily(DefsTable.java:463)
> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:407)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:271)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:249)
> at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:48)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 6 more
> Caused by: org.apache.cassandra.config.ConfigurationException: Column family ID mismatch (found 13740; expected 13739)
> at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:698)
> at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:672) ... 12 more
>
> We also see this exception in the same log which is ironic considering the above one says found 13740!!!!!! And this one says couldn't find it….
>
> ERROR [MutationStage:27379] 2012-09-26 09:50:57,558 RowMutationVerbHandler.java (line 61) Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=13740
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
> at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
> at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)