You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by aaron morton <aa...@thelastpickle.com> on 2012/10/01 03:12:26 UTC
Re: any ideas on what these mean
> ERROR [MigrationStage:1] 2012-09-26 09:51:03,128 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]
I *think* this is due to concurrent schema modifications.
The found number is the one in the database for the matching KS / CF name, the second number is the one in memory at the time. That's why the reboot fixed it.
If it happens again try nodetool resetlocalschema
> 2. Is it affecting my data put? (I have seen other weird validation exceptions where my data is still put and I can read it from cassandra and I get no exception client side)
This write was dropped
> ERROR [MutationStage:27379] 2012-09-26 09:50:57,558 RowMutationVerbHandler.java (line 61) Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=13740
I would make sure repair is running to resolve any inconsistencies created during the period of wackyness.
> 3. How do I reverse engineer what column familsy 13740 and 13739 are? (ie. Their names?) so I can check for data corruption.
On 1.1 X schema is stored in the schema_columnfamilies CF in t he system KS. In previous versions it's in a different CF in the same KS.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 27/09/2012, at 4:04 AM, "Hiller, Dean" <De...@nrel.gov> wrote:
> We were consistently getting this exception over and over as we put data into the system. A reboot caused it to go away but we don't want to be rebooting in the future….
>
> 1. When does this occur?
> 2. Is it affecting my data put? (I have seen other weird validation exceptions where my data is still put and I can read it from cassandra and I get no exception client side)
> 3. How do I reverse engineer what column familsy 13740 and 13739 are? (ie. Their names?) so I can check for data corruption.
>
> ERROR [MigrationStage:1] 2012-09-26 09:51:03,128 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[MigrationStage:1,5,main]
> java.lang.RuntimeException: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Column family ID mismatch (found 13740; expected 13739)
> at org.apache.cassandra.utils.FBUtilities.unchecked(FBUtilities.java:628)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: org.apache.cassandra.config.ConfigurationException: Column family ID mismatch (found 13740; expected 13739)
> at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:676)
> at org.apache.cassandra.db.DefsTable.updateColumnFamily(DefsTable.java:463)
> at org.apache.cassandra.db.DefsTable.mergeColumnFamilies(DefsTable.java:407)
> at org.apache.cassandra.db.DefsTable.mergeSchema(DefsTable.java:271)
> at org.apache.cassandra.db.DefsTable.mergeRemoteSchema(DefsTable.java:249)
> at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:48)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> ... 6 more
> Caused by: org.apache.cassandra.config.ConfigurationException: Column family ID mismatch (found 13740; expected 13739)
> at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:698)
> at org.apache.cassandra.config.CFMetaData.reload(CFMetaData.java:672) ... 12 more
>
> We also see this exception in the same log which is ironic considering the above one says found 13740!!!!!! And this one says couldn't find it….
>
> ERROR [MutationStage:27379] 2012-09-26 09:50:57,558 RowMutationVerbHandler.java (line 61) Error in row mutation
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find cfId=13740
> at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:126)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:439)
> at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:447)
> at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:395)
> at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:42)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)