You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Maciek Sakrejda <ma...@heroku.com> on 2015/11/11 00:01:13 UTC
UnknownColumnFamily exception / schema inconsistencies
Hello,
I've been having some strange issues with one of our test clusters
(4-day-old, 3-node, 2.1.10 cluster on AWS). I saw a number of messages like
the following:
[0000] 10 Nov 20:21:00.406 * pri=WARN t=MessagingService-Incoming-/
192.168.168.202 at=IncomingTcpConnection.run UnknownColumnFamilyException
reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
cfId=3ecce750-84d3-11e5-bdd9-dd7717dcdbd5
A colleague suggested I run repair, but that failed with:
[2015-11-10 20:06:54,329] Nothing to repair for keyspace 'eventPipesState'
[2015-11-10 20:06:54,348] Starting repair command #1, repairing 768 ranges
for keyspace dbs8okvd7jcurj (parallelism=SEQUENTIAL, full=true)
[2015-11-10 20:06:55,599] Repair command #1 finished
[2015-11-10 20:06:55,610] Starting repair command #2, repairing 487 ranges
for keyspace context (parallelism=SEQUENTIAL, full=true)
[2015-11-10 20:11:21,213] Lost notification. You should check server log
for repair status of keyspace context
[2015-11-10 20:11:21,288] Lost notification. You should check server log
for repair status of keyspace context
Exception occurred during clean-up.
java.lang.reflect.UndeclaredThrowableException
error: JMX connection closed. You should check server log for repair status
of keyspace context(Subsequent keyspaces are not going to be repaired).
-- StackTrace --
java.io.IOException: JMX connection closed. You should check server log for
repair status of keyspace context(Subsequent keyspaces are not going to be
repaired).
I searched for other cases of similar issues, and found some posts (e.g.,
http://stackoverflow.com/questions/22783577/org-apache-cassandra-db-unknowncolumnfamilyexception-couldnt-find-cfid
), but nothing that seemed directly relevant. Still, I tried `nodetool
describecluster` and all the nodes showed up as being on the same schema
version.
The server log did not include any more info. I asked about this on IRC and
got the suggestion to run `nodetool resetlocalschema`. I tried running
that, and it completed (and `nodetool describecluster` now shows this node
as having a different schema version from the other two nodes) but now I
still get the original error in the server logs but also
[0000] 10 Nov 22:51:10.466 * pri=ERROR t=Thrift:12
at=CustomTThreadPoolServer.run Error occurred during processing of message.
java.lang.IllegalArgumentException: Unknown keyspace/cf pair
(system_auth.credentials)
Further `nodetool repair`s on the same node do complete, but only seem to
process the `system` keyspace (and don't do anything with it):
[2015-11-10 22:38:07,415] Nothing to repair for keyspace 'system'
I also tried running `nodetool repair` from another node in the cluster,
but that just seems to hang:
[2015-11-10 22:53:11,830] Starting repair command #7, repairing 768 ranges
for keyspace dbs8okvd7jcurj (parallelism=SEQUENTIAL, full=true)
[2015-11-10 22:53:12,943] Repair command #7 finished
[2015-11-10 22:53:12,958] Starting repair command #8, repairing 534 ranges
for keyspace context (parallelism=SEQUENTIAL, full=true)
How can I restore this cluster? And ideally, how can I figure out what went
wrong here in the first place?
Re: UnknownColumnFamily exception / schema inconsistencies
Posted by Maciek Sakrejda <ma...@heroku.com>.
Just wanted to follow up and say thanks: I went through this process (as
per Robert's suggestion, with the node stopped and no refresh) on an
affected cluster and was able to resolve the issue.
Re: UnknownColumnFamily exception / schema inconsistencies
Posted by Robert Coli <rc...@eventbrite.com>.
On Fri, Nov 13, 2015 at 12:31 PM, Maciek Sakrejda <ma...@heroku.com> wrote:
> On Fri, Nov 13, 2015 at 9:56 AM, Sebastian Estevez <
> sebastian.estevez@datastax.com> wrote:
>
>> I think you're just missing the steps in *Bold*:
>>
>> Thanks, but I wasn't clear on what to do if the "new" directory does not
> exist at all on some of the nodes (only the old). Can I just rename the
> "old" to the "new" or is there more to it?
>
Either create it or rename the old directory (and files) to the new name.
Unlike the linked doc, I recommend against using "nodetool refresh" for
this, and recommend doing the operation with the node down.
https://issues.apache.org/jira/browse/CASSANDRA-6245 (et al)
Note also that if the sstables in the "bad" directory are older than
gc_grace_seconds, it may not be safe for consistency purposes to shotgun
them into the "good" directory.
=Rob
Re: UnknownColumnFamily exception / schema inconsistencies
Posted by Maciek Sakrejda <ma...@heroku.com>.
On Fri, Nov 13, 2015 at 9:56 AM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:
> I think you're just missing the steps in *Bold*:
>
> Thanks, but I wasn't clear on what to do if the "new" directory does not
exist at all on some of the nodes (only the old). Can I just rename the
"old" to the "new" or is there more to it?
Re: UnknownColumnFamily exception / schema inconsistencies
Posted by Sebastian Estevez <se...@datastax.com>.
I think you're just missing the steps in *Bold*:
If THERE ARE TWO OR MORE DIRECTORIES:
4)Identify from schema_column_families which cf ID is the "new" one
(currently in use).
cqlsh -e "select * from system.schema_column_families"|grep <table name>
*5) Move the data from the "old" one to the "new" one and remove the
old directory. *
*6) If there are multiple "old" ones repeat 5 for every "old" directory.*
*7) run nodetool refresh*
All the best,
[image: datastax_logo.png] <http://www.datastax.com/>
Sebastián Estévez
Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://goog_410786983>
<http://www.datastax.com/gartner-magic-quadrant-odbms>
DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.
On Fri, Nov 13, 2015 at 12:37 PM, Maciek Sakrejda <ma...@heroku.com> wrote:
> Any advice on how to proceed here? Sebastian seems to have guessed
> correctly at the underlying issue, but I'm still not sure how to resolve
> this given what I see in the data directory and the catalogs.
>
> On Wed, Nov 11, 2015 at 12:15 PM, Maciek Sakrejda <ma...@heroku.com>
> wrote:
>
>> On Wed, Nov 11, 2015 at 9:55 AM, Sebastian Estevez <
>> sebastian.estevez@datastax.com> wrote:
>>
>>> Stupid question, but how do I find the problem table? The error message
>>>> complains about a keyspace (by uuid); I haven't seen errors relating to a
>>>> specific table. I've poked around in the data directory, but I'm not sure
>>>> what I'm looking for.
>>>
>>>
>>> Is the message complaining about a *keyspace* or abou*t a table (cfid)*?
>>> You'r original was complaining about a table:
>>>
>>
>>> at=IncomingTcpConnection.run UnknownColumnFamilyException reading from
>>>> socket; closing org.apache.cassandra.db.UnknownColumnFamilyException:
>>>> Couldn't find *cfId=3ecce750-84d3-11e5-bdd9-**dd7717dcdbd5*
>>>
>>>
>> Sorry, you're absolutely right--it's the table from this error message. I
>> confused myself. But now I was able to find it:
>>
>> cursors-3ecce75084d311e5bdd9dd7717dcdbd5
>> cursors-3ed23e8084d311e583b30fc0205655f5
>>
>> The second uuid is the one that shows up via the schema_columnfamilies
>> query, but on two of the nodes, the directory with the *other* uuid exists.
>> Can I just rename the directory on these two nodes? Or how should I proceed?
>>
>
>
Re: UnknownColumnFamily exception / schema inconsistencies
Posted by Maciek Sakrejda <ma...@heroku.com>.
Any advice on how to proceed here? Sebastian seems to have guessed
correctly at the underlying issue, but I'm still not sure how to resolve
this given what I see in the data directory and the catalogs.
On Wed, Nov 11, 2015 at 12:15 PM, Maciek Sakrejda <ma...@heroku.com> wrote:
> On Wed, Nov 11, 2015 at 9:55 AM, Sebastian Estevez <
> sebastian.estevez@datastax.com> wrote:
>
>> Stupid question, but how do I find the problem table? The error message
>>> complains about a keyspace (by uuid); I haven't seen errors relating to a
>>> specific table. I've poked around in the data directory, but I'm not sure
>>> what I'm looking for.
>>
>>
>> Is the message complaining about a *keyspace* or abou*t a table (cfid)*?
>> You'r original was complaining about a table:
>>
>
>> at=IncomingTcpConnection.run UnknownColumnFamilyException reading from
>>> socket; closing org.apache.cassandra.db.UnknownColumnFamilyException:
>>> Couldn't find *cfId=3ecce750-84d3-11e5-bdd9-**dd7717dcdbd5*
>>
>>
> Sorry, you're absolutely right--it's the table from this error message. I
> confused myself. But now I was able to find it:
>
> cursors-3ecce75084d311e5bdd9dd7717dcdbd5
> cursors-3ed23e8084d311e583b30fc0205655f5
>
> The second uuid is the one that shows up via the schema_columnfamilies
> query, but on two of the nodes, the directory with the *other* uuid exists.
> Can I just rename the directory on these two nodes? Or how should I proceed?
>
Re: UnknownColumnFamily exception / schema inconsistencies
Posted by Maciek Sakrejda <ma...@heroku.com>.
On Wed, Nov 11, 2015 at 9:55 AM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:
> Stupid question, but how do I find the problem table? The error message
>> complains about a keyspace (by uuid); I haven't seen errors relating to a
>> specific table. I've poked around in the data directory, but I'm not sure
>> what I'm looking for.
>
>
> Is the message complaining about a *keyspace* or abou*t a table (cfid)*?
> You'r original was complaining about a table:
>
> at=IncomingTcpConnection.run UnknownColumnFamilyException reading from
>> socket; closing org.apache.cassandra.db.UnknownColumnFamilyException:
>> Couldn't find *cfId=3ecce750-84d3-11e5-bdd9-**dd7717dcdbd5*
>
>
Sorry, you're absolutely right--it's the table from this error message. I
confused myself. But now I was able to find it:
cursors-3ecce75084d311e5bdd9dd7717dcdbd5
cursors-3ed23e8084d311e583b30fc0205655f5
The second uuid is the one that shows up via the schema_columnfamilies
query, but on two of the nodes, the directory with the *other* uuid exists.
Can I just rename the directory on these two nodes? Or how should I proceed?
Re: UnknownColumnFamily exception / schema inconsistencies
Posted by Sebastian Estevez <se...@datastax.com>.
>
> Stupid question, but how do I find the problem table? The error message
> complains about a keyspace (by uuid); I haven't seen errors relating to a
> specific table. I've poked around in the data directory, but I'm not sure
> what I'm looking for.
Is the message complaining about a *keyspace* or abou*t a table (cfid)*?
You'r original was complaining about a table:
at=IncomingTcpConnection.run UnknownColumnFamilyException reading from
> socket; closing org.apache.cassandra.db.UnknownColumnFamilyException:
> Couldn't find *cfId=3ecce750-84d3-11e5-bdd9-**dd7717dcdbd5*
All the best,
[image: datastax_logo.png] <http://www.datastax.com/>
Sebastián Estévez
Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://goog_410786983>
<http://www.datastax.com/gartner-magic-quadrant-odbms>
DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.
On Wed, Nov 11, 2015 at 12:26 PM, Maciek Sakrejda <ma...@heroku.com> wrote:
> On Tue, Nov 10, 2015 at 3:20 PM, Sebastian Estevez <
> sebastian.estevez@datastax.com> wrote:
>
>> #1 The cause of this problem is a CREATE TABLE statement collision. Do not generate tables
>> dynamically from multiple clients, even with IF NOT EXISTS. First thing you need to do is
>> fix your code so that this does not happen. Just create your tables manually from cqlsh allowing
>> time for the schema to settle.
>>
>> #2 Here's the fix:
>>
>> 1) Change your code to not automatically re-create tables (even with IF NOT EXISTS).
>>
>> 2) Run a rolling restart to ensure schema matches across nodes. Run nodetool describecluster
>>
>> around your cluster. Check that there is only one schema version.
>>
>> Thanks, that seems to have resolved the schema version inconsistency
> (though I'm still getting the original error).
>
>> ON EACH NODE:
>>
>> 3) Check your filesystem and see if you have two directories for the table in
>>
>> question in the data directory.
>>
>> Stupid question, but how do I find the problem table? The error message
> complains about a keyspace (by uuid); I haven't seen errors relating to a
> specific table. I've poked around in the data directory, but I'm not sure
> what I'm looking for.
>
>
Re: UnknownColumnFamily exception / schema inconsistencies
Posted by Maciek Sakrejda <ma...@heroku.com>.
On Tue, Nov 10, 2015 at 3:20 PM, Sebastian Estevez <
sebastian.estevez@datastax.com> wrote:
> #1 The cause of this problem is a CREATE TABLE statement collision. Do not generate tables
> dynamically from multiple clients, even with IF NOT EXISTS. First thing you need to do is
> fix your code so that this does not happen. Just create your tables manually from cqlsh allowing
> time for the schema to settle.
>
> #2 Here's the fix:
>
> 1) Change your code to not automatically re-create tables (even with IF NOT EXISTS).
>
> 2) Run a rolling restart to ensure schema matches across nodes. Run nodetool describecluster
>
> around your cluster. Check that there is only one schema version.
>
> Thanks, that seems to have resolved the schema version inconsistency
(though I'm still getting the original error).
> ON EACH NODE:
>
> 3) Check your filesystem and see if you have two directories for the table in
>
> question in the data directory.
>
> Stupid question, but how do I find the problem table? The error message
complains about a keyspace (by uuid); I haven't seen errors relating to a
specific table. I've poked around in the data directory, but I'm not sure
what I'm looking for.
Re: UnknownColumnFamily exception / schema inconsistencies
Posted by Sebastian Estevez <se...@datastax.com>.
#1 The cause of this problem is a CREATE TABLE statement collision. Do
not generate tables
dynamically from multiple clients, even with IF NOT EXISTS. First
thing you need to do is
fix your code so that this does not happen. Just create your tables
manually from cqlsh allowing
time for the schema to settle.
#2 Here's the fix:
1) Change your code to not automatically re-create tables (even with
IF NOT EXISTS).
2) Run a rolling restart to ensure schema matches across nodes. Run
nodetool describecluster
around your cluster. Check that there is only one schema version.
ON EACH NODE:
3) Check your filesystem and see if you have two directories for the table in
question in the data directory.
If THERE ARE TWO OR MORE DIRECTORIES:
4)Identify from schema_column_families which cf ID is the "new" one
(currently in use).
cqlsh -e "select * from system.schema_column_families"|grep <table name>
5) Move the data from the "old" one to the "new" one and remove the
old directory.
6) If there are multiple "old" ones repeat 5 for every "old" directory.
7) run nodetool refresh
IF THERE IS ONLY ONE DIRECTORY:
No further action is needed.
All the best,
[image: datastax_logo.png] <http://www.datastax.com/>
Sebastián Estévez
Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://goog_410786983>
<http://www.datastax.com/gartner-magic-quadrant-odbms>
DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.
On Tue, Nov 10, 2015 at 6:09 PM, Maciek Sakrejda <ma...@heroku.com> wrote:
> Oh and for what it's worth, I've also looked through the logs for this
> node, and the oldest error in the logs seems to be:
>
> [0000] 06 Nov 22:10:53.260 * pri=ERROR t=Thrift:16
> at=CustomTThreadPoolServer.run Error occurred during processing of message.
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.cassandra.exceptions.ConfigurationException: Column family ID
> mismatch (found 3ed23e80-84d3-11e5-83b3-0fc0205655f5; expected
> 3ecce750-84d3-11e5-bdd9-dd7717dcdbd5)
>
> Then the logs show a compaction, and then the UnknownColumnFamilyException
> starts occuring.
>
>
Re: UnknownColumnFamily exception / schema inconsistencies
Posted by Maciek Sakrejda <ma...@heroku.com>.
Oh and for what it's worth, I've also looked through the logs for this
node, and the oldest error in the logs seems to be:
[0000] 06 Nov 22:10:53.260 * pri=ERROR t=Thrift:16
at=CustomTThreadPoolServer.run Error occurred during processing of message.
java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException:
org.apache.cassandra.exceptions.ConfigurationException: Column family ID
mismatch (found 3ed23e80-84d3-11e5-83b3-0fc0205655f5; expected
3ecce750-84d3-11e5-bdd9-dd7717dcdbd5)
Then the logs show a compaction, and then the UnknownColumnFamilyException
starts occuring.