Re: Attempt to assign id to existing column family.

[aaron morton <aa...@thelastpickle.com>]

There is no reason to change the RF on the system keyspace, it should probably not be allowed.

The system keyspace uses a LocalPartitioner and it's data is not replicated through the same mechanism as a user keyspace. 
 
Aaron
 
On 31 Mar 2011, at 10:22, Jeremy Stribling wrote:

> On 03/30/2011 02:54 PM, Jeremy Stribling wrote:
>> After restarting a Cassandra 0.7.2 node, the node catches an exception during initialization and refuses to start:
>> 
>> Caused by: org.apache.cassandra.config.ConfigurationException: Attempt to assign id to existing column family.
>>        at org.apache.cassandra.config.CFMetaData.map(CFMetaData.java:222)
>>        at org.apache.cassandra.config.DatabaseDescriptor.loadSchemas(DatabaseDescriptor.java:477) 
>>        ... 2 more
>> 
>> Unlike a previous thread about this topic (http://www.mail-archive.com/user@cassandra.apache.org/msg09024.html), we are not trying to preserve the JVM across restarts.  The restart comes up in an entirely fresh JVM.  We are, however, embedding Cassandra in our application, but we're using the same steps used by AbstractCassandraDaemon to bring it up.
>> 
>> Looking briefly through the code, the only way I see that this can happen is if loadSchemas tries to load information about the system table from storage (because the system table can be created in CFMetaData from the earlier DatabaseDescriptor.getTableMetaData(Table.SYSTEM_TABLE).values() call).  Or I guess the data on disk could have multiple entries under the same key, but the system table issue seems more likely to me.  Unfortunately the logging is not specific enough for me to tell which key it is failing with, and I haven't been able to reproduce this yet.
>> 
>> One relevant piece of information might be that, before the restart, our application changed the replication factor of all the tables, including the system table:
>> 
>> 2011-03-29 23:09:39,194 291146 [MigrationStage:1] INFO org.apache.cassandra.db.migration.Migration  - Applying migration 9f371026-5a59-11e0-b23f-65ed1eced995 Update keyspace systemrep factor:1rep  strategy:LocalStrategy{...} to systemrep factor:3rep strategy:LocalStrategy{...}
>> 
>> We're doing this in order to dynamically change the replication factor as new nodes are being added to the cluster (e.g., it starts off with one node and a repfactor of 1, and once there are three nodes, it increases the repfactor on all tables to 3).  Is it possible that migrations over the system table get written to disk in a way that would cause loadSchemas() during a restart to hit this exception?  Are we even allowed to change the replication factor of the system table?
>> 
> 
> I've confirmed that this happens when loading column family "IndexInfo" from the table "system" during the loadSchemas() call.  Does anyone know if there's a way to get around this?  Perhaps, like I theorized, it's not legit to change the replication factor on the system table.
> 
>