Unknown CF / Schema OK

[Tim Olson <ka...@gmail.com>]

​After upgrading a schema, I'm getting lots of UnknownColumnFamilyException in
the logs.  However, all nodes have the same schema as reported by nodetool
describecluster.   I queried the system tables for the given column family
UUID, but it doesn't appear in any of the schemas on any of the nodes.  I
restarted all clients, but that didn't help either.

The cluster was running 2.1.2 but I recently upgraded to 2.1.3.

Any ideas?  This is basically making our production cluster highly
unresponsive.

Tim

Re: Unknown CF / Schema OK

[Tim Olson <ka...@gmail.com>]

I did figure this out:

When adding a columnfamily, the query timed out before all nodes replied,
and I sent the schema out again.  Half the nodes ended up with the CF
having UUID A and half the nodes ended up with the new CF but UUID B.
UnknownColumnFamilyExceptions were thrown until the enqueued data exceeded
memory.  Eventually one half of the nodes crashed, with the other half
having a consistent view of the CF.  At this point I just dropped the
offending CF schema in the active cluster, then the downed nodes could be
re-added successfully.  We lost some data.  :(



On Sun, Mar 22, 2015 at 11:39 AM, Tim Olson <ka...@gmail.com> wrote:

> ​After upgrading a schema, I'm getting lots of
> UnknownColumnFamilyException in the logs.  However, all nodes have the
> same schema as reported by nodetool describecluster.   I queried the
> system tables for the given column family UUID, but it doesn't appear in
> any of the schemas on any of the nodes.  I restarted all clients, but that
> didn't help either.
>
> The cluster was running 2.1.2 but I recently upgraded to 2.1.3.
>
> Any ideas?  This is basically making our production cluster highly
> unresponsive.
>
> Tim
>