You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by mm...@vis.at on 2018/06/05 09:32:37 UTC
Single Host: Fix "Unknown CF" issue
Hi all!
We're using cassandra since a couple of month to get familiar with it.
We're currently using only 1-node. Yesterday our server had to be
restarted and now cassandra does not start anymore.
It reports:
INFO [main] 2018-06-05 09:50:43,030 ColumnFamilyStore.java:406 -
Initializing system_schema.indexes
INFO [main] 2018-06-05 09:50:43,036 ViewManager.java:137 - Not
submitting build tasks for views in keyspace system_schema as storage
service is not initialized
INFO [main] 2018-06-05 09:50:43,283 ColumnFamilyStore.java:406 -
Initializing system_traces.events
INFO [main] 2018-06-05 09:50:43,286 ColumnFamilyStore.java:406 -
Initializing system_traces.sessions
INFO [main] 2018-06-05 09:50:43,287 ViewManager.java:137 - Not
submitting build tasks for views in keyspace system_traces as storage
service is not initialized
INFO [main] 2018-06-05 09:50:43,300 ColumnFamilyStore.java:406 -
Initializing m2m_auth.user
INFO [main] 2018-06-05 09:50:43,302 ColumnFamilyStore.java:406 -
Initializing m2m_auth.eventsbytag1
INFO [main] 2018-06-05 09:50:43,306 ColumnFamilyStore.java:406 -
Initializing m2m_auth.mail2user
ERROR [main] 2018-06-05 09:50:43,311 CassandraDaemon.java:706 -
Exception encountered during startup
java.lang.IllegalArgumentException: Unknown CF
0f6c8b36-5f34-11e8-a476-c93745f84272
at
org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:204)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.db.view.ViewManager.addView(ViewManager.java:152)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:125)
~[apache-cassandra-3.11.0.jar:3.11.0]
at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)
~[apache-cassandra-3.11.0.jar:3.11.0]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127)
~[apache-cassandra-3.11.0.jar:3.11.0]
at org.apache.cassandra.db.Keyspace.open(Keyspace.java:104)
~[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:292)
[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
[apache-cassandra-3.11.0.jar:3.11.0]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
[apache-cassandra-3.11.0.jar:3.11.0]
We do not know how to bring it up again and e.g. stackoverflow only
recommends to delete the node and let it rebuild (which we cannot do as
we've only 1 node currently).
Can anybody drop us a hint ?
What can we do if our node does not start?
We've found an entry in the data directory with the given UUID but at
the time when we restarted the node the whole keyspace was idle for a
couple of hours.
With the problem we're now _very_ concerned about data safety when we
read that it can be a problem if e.g. tables are created/deleted/updated
concurrently. But in our case we did not create/update tables
concurrently and got this problem anyway....
Thanks for any help!
greetings,
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Single Host: Fix "Unknown CF" issue
Posted by mm...@vis.at.
Hi,
we will follow the recommendation not to use materialized views.
Thanks a lot to both of you!
You helped me a lot.
Oh and besides: We are also using the lagom framework :) So we will also
be able to regenerate a Read-Side if have to.
greetings,
Michael
On 07.06.2018 13:45, Evelyn Smith wrote:
> Hey Michael,
>
> In the case that you have a production cluster set up with multiple
> nodes, assuming you have rf>1 it’s easier to just replace the broken
> node and restore it’s data. (For future reference)
>
> I wasn’t sure if view was referring to materialised view at the time
> although Pradeeps comment along with your own suggest it might (I
> didn’t get a chance to look through the code to confirm if view was
> MV or something else and I’m not that familiar with the code base).
>
> As far as the choice of using Materialised Views, they aren’t being
> deprecated they are currently marked as experimental and most people
> strongly advise you to not use them. If you can avoid it don’t do
> it. They’re associated with a lot of bugs and scalability issues.
> Also they’re just hard to do right if you aren’t exceptionally
> familiar with Cassandra.
>
> Regards,
> Evelyn.
>
>> On 7 Jun 2018, at 3:05 am, Pradeep Chhetri <pr...@stashaway.com>
>> wrote:
>>
>> Hi Michael,
>>
>> We have faced the same situation as yours in our production
>> environment where we suddenly got "Unknown CF Exception" for
>> materialized views too. We are using Lagom apps with cassandra for
>> persistence. In our case, since these views can be regenerated from
>> the original events, we were able to safely recover.
>>
>> Few suggestions from my operations experience:
>>
>> 1) Upgrade your cassandra cluster to 3.11.2 because there are lots
>> of bug fixes specific to materialized views.
>> 2) Never let your application create/update/delete cassandra
>> table/materialized views. Always create them manually to make sure
>> that only connection is doing the operation.
>>
>> Regards,
>> Pradeep
>>
>> On Wed, Jun 6, 2018 at 9:44 PM, <mm...@vis.at> wrote:
>> Hi Evelyn,
>>
>> thanks a lot for your detailed response message.
>>
>> The data is not important. We've already wiped the data and created
>> a new cassandra installation. The data re-import task is already
>> running. We've lost the data for a couple of months but in this case
>> this does not matter.
>>
>> Nevertheless we will try what you told us - just to be
>> smarter/faster if this happens in production (where we will setup a
>> cassandra cluster with multiple cassandra nodes anyway). I will drop
>> you a note when we are done.
>>
>> Hmmm... the problem is within a "View". Are this the materialized
>> views?
>>
>> I'm asking this because:
>> * Someone on the internet (stackoverflow if a recall correctly)
>> mentioned that using materialized views are to be deprecated.
>> * I had been on a datastax workshop in Zurich a couple of days ago
>> where a datastax employee told me that we should not use
>> materialized views - it is better to create & fill all tables
>> directly.
>>
>> Would you also recommend not to use materialized views? As this
>> problem is related to a view - maybe we could avoid this problem
>> simply by following this recommendation.
>>
>> Thanks a lot again!
>>
>> Greetings,
>> Michael
>>
>> On 06.06.2018 16:48, Evelyn Smith wrote:
>> Hi Michael,
>>
>> So I looked at the code, here are some stages of your error message:
>> 1. at
>>
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:292)
>> [apache-cassandra-3.11.0.jar:3.11.0
>> At this step Cassandra is running through the keyspaces in it’s
>> schema turning off compactions for all tables before it starts
>> rerunning the commit log (so it isn’t an issue with the commit
>> log).
>> 2. at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> Loading key space related to the column family that is erroring out
>> 3. at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> Cassandra has initialised the column family and is reloading the
>> view
>> 4. at
>>
> org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:204)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> At this point I haven’t had enough time to tell if Cassandra is
>> requesting info on a column specifically or still requesting
>> information on a column family. Regardless, given we already rule
>> out
>> issues with the SSTables and their directory and Cassandra is yet to
>> start processing the commit log this to me suggests it’s something
>> wrong in one of the system keyspaces storing the schema information.
>>
>> There should definitely be a way to resolve this with zero data loss
>> by either:
>> 1. Fixing the issue in the system keyspace SSTables (hard)
>> 2. Rerunning the commit log on a new Cassandra node that has been
>> restored from the current one (I’m not sure if this is possible
>> but
>> I’ll figure it out tomorrow)
>>
>> The alternative is if you are ok with losing the commitlog then you
>> can backup the data and restore it to a new node (or the same node
>> but
>> with everything blown away). This isn’t a trivial process though
>> I’ve done it a few times.
>>
>> How important is the data?
>>
>> Happy to come back to this tomorrow (need some sleep)
>>
>> Regards,
>> Eevee.
>>
>> On 5 Jun 2018, at 7:32 pm, mm@vis.at wrote:
>> Keyspace.getColumnFamilyStore
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Single Host: Fix "Unknown CF" issue
Posted by Evelyn Smith <u5...@gmail.com>.
Hey Michael,
In the case that you have a production cluster set up with multiple nodes, assuming you have rf>1 it’s easier to just replace the broken node and restore it’s data. (For future reference)
I wasn’t sure if view was referring to materialised view at the time although Pradeeps comment along with your own suggest it might (I didn’t get a chance to look through the code to confirm if view was MV or something else and I’m not that familiar with the code base).
As far as the choice of using Materialised Views, they aren’t being deprecated they are currently marked as experimental and most people strongly advise you to not use them. If you can avoid it don’t do it. They’re associated with a lot of bugs and scalability issues. Also they’re just hard to do right if you aren’t exceptionally familiar with Cassandra.
Regards,
Evelyn.
> On 7 Jun 2018, at 3:05 am, Pradeep Chhetri <pr...@stashaway.com> wrote:
>
> Hi Michael,
>
> We have faced the same situation as yours in our production environment where we suddenly got "Unknown CF Exception" for materialized views too. We are using Lagom apps with cassandra for persistence. In our case, since these views can be regenerated from the original events, we were able to safely recover.
>
> Few suggestions from my operations experience:
>
> 1) Upgrade your cassandra cluster to 3.11.2 because there are lots of bug fixes specific to materialized views.
> 2) Never let your application create/update/delete cassandra table/materialized views. Always create them manually to make sure that only connection is doing the operation.
>
> Regards,
> Pradeep
>
>
>
> On Wed, Jun 6, 2018 at 9:44 PM, <mm@vis.at <ma...@vis.at>> wrote:
> Hi Evelyn,
>
> thanks a lot for your detailed response message.
>
> The data is not important. We've already wiped the data and created a new cassandra installation. The data re-import task is already running. We've lost the data for a couple of months but in this case this does not matter.
>
> Nevertheless we will try what you told us - just to be smarter/faster if this happens in production (where we will setup a cassandra cluster with multiple cassandra nodes anyway). I will drop you a note when we are done.
>
> Hmmm... the problem is within a "View". Are this the materialized views?
>
> I'm asking this because:
> * Someone on the internet (stackoverflow if a recall correctly) mentioned that using materialized views are to be deprecated.
> * I had been on a datastax workshop in Zurich a couple of days ago where a datastax employee told me that we should not use materialized views - it is better to create & fill all tables directly.
>
> Would you also recommend not to use materialized views? As this problem is related to a view - maybe we could avoid this problem simply by following this recommendation.
>
> Thanks a lot again!
>
> Greetings,
> Michael
>
>
>
>
> On 06.06.2018 16:48, Evelyn Smith wrote:
> Hi Michael,
>
> So I looked at the code, here are some stages of your error message:
> 1. at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:292)
> [apache-cassandra-3.11.0.jar:3.11.0
> At this step Cassandra is running through the keyspaces in it’s
> schema turning off compactions for all tables before it starts
> rerunning the commit log (so it isn’t an issue with the commit log).
> 2. at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> Loading key space related to the column family that is erroring out
> 3. at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> Cassandra has initialised the column family and is reloading the view
> 4. at
> org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:204)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> At this point I haven’t had enough time to tell if Cassandra is
> requesting info on a column specifically or still requesting
> information on a column family. Regardless, given we already rule out
> issues with the SSTables and their directory and Cassandra is yet to
> start processing the commit log this to me suggests it’s something
> wrong in one of the system keyspaces storing the schema information.
>
> There should definitely be a way to resolve this with zero data loss
> by either:
> 1. Fixing the issue in the system keyspace SSTables (hard)
> 2. Rerunning the commit log on a new Cassandra node that has been
> restored from the current one (I’m not sure if this is possible but
> I’ll figure it out tomorrow)
>
> The alternative is if you are ok with losing the commitlog then you
> can backup the data and restore it to a new node (or the same node but
> with everything blown away). This isn’t a trivial process though
> I’ve done it a few times.
>
> How important is the data?
>
> Happy to come back to this tomorrow (need some sleep)
>
> Regards,
> Eevee.
>
> On 5 Jun 2018, at 7:32 pm, mm@vis.at <ma...@vis.at> wrote:
> Keyspace.getColumnFamilyStore
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org <ma...@cassandra.apache.org>
> For additional commands, e-mail: user-help@cassandra.apache.org <ma...@cassandra.apache.org>
>
>
Re: Single Host: Fix "Unknown CF" issue
Posted by Pradeep Chhetri <pr...@stashaway.com>.
Hi Michael,
We have faced the same situation as yours in our production environment
where we suddenly got "Unknown CF Exception" for materialized views too. We
are using Lagom apps with cassandra for persistence. In our case, since
these views can be regenerated from the original events, we were able to
safely recover.
Few suggestions from my operations experience:
1) Upgrade your cassandra cluster to 3.11.2 because there are lots of bug
fixes specific to materialized views.
2) Never let your application create/update/delete cassandra
table/materialized views. Always create them manually to make sure that
only connection is doing the operation.
Regards,
Pradeep
On Wed, Jun 6, 2018 at 9:44 PM, <mm...@vis.at> wrote:
> Hi Evelyn,
>
> thanks a lot for your detailed response message.
>
> The data is not important. We've already wiped the data and created a new
> cassandra installation. The data re-import task is already running. We've
> lost the data for a couple of months but in this case this does not matter.
>
> Nevertheless we will try what you told us - just to be smarter/faster if
> this happens in production (where we will setup a cassandra cluster with
> multiple cassandra nodes anyway). I will drop you a note when we are done.
>
> Hmmm... the problem is within a "View". Are this the materialized views?
>
> I'm asking this because:
> * Someone on the internet (stackoverflow if a recall correctly) mentioned
> that using materialized views are to be deprecated.
> * I had been on a datastax workshop in Zurich a couple of days ago where a
> datastax employee told me that we should not use materialized views - it is
> better to create & fill all tables directly.
>
> Would you also recommend not to use materialized views? As this problem is
> related to a view - maybe we could avoid this problem simply by following
> this recommendation.
>
> Thanks a lot again!
>
> Greetings,
> Michael
>
>
>
>
> On 06.06.2018 16:48, Evelyn Smith wrote:
>
>> Hi Michael,
>>
>> So I looked at the code, here are some stages of your error message:
>> 1. at
>> org.apache.cassandra.service.CassandraDaemon.setup(Cassandra
>> Daemon.java:292)
>> [apache-cassandra-3.11.0.jar:3.11.0
>> At this step Cassandra is running through the keyspaces in it’s
>> schema turning off compactions for all tables before it starts
>> rerunning the commit log (so it isn’t an issue with the commit log).
>> 2. at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> Loading key space related to the column family that is erroring out
>> 3. at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> Cassandra has initialised the column family and is reloading the view
>> 4. at
>> org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:204)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> At this point I haven’t had enough time to tell if Cassandra is
>> requesting info on a column specifically or still requesting
>> information on a column family. Regardless, given we already rule out
>> issues with the SSTables and their directory and Cassandra is yet to
>> start processing the commit log this to me suggests it’s something
>> wrong in one of the system keyspaces storing the schema information.
>>
>> There should definitely be a way to resolve this with zero data loss
>> by either:
>> 1. Fixing the issue in the system keyspace SSTables (hard)
>> 2. Rerunning the commit log on a new Cassandra node that has been
>> restored from the current one (I’m not sure if this is possible but
>> I’ll figure it out tomorrow)
>>
>> The alternative is if you are ok with losing the commitlog then you
>> can backup the data and restore it to a new node (or the same node but
>> with everything blown away). This isn’t a trivial process though
>> I’ve done it a few times.
>>
>> How important is the data?
>>
>> Happy to come back to this tomorrow (need some sleep)
>>
>> Regards,
>> Eevee.
>>
>> On 5 Jun 2018, at 7:32 pm, mm@vis.at wrote:
>>> Keyspace.getColumnFamilyStore
>>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
Re: Single Host: Fix "Unknown CF" issue
Posted by mm...@vis.at.
Hi Evelyn,
thanks a lot for your detailed response message.
The data is not important. We've already wiped the data and created a
new cassandra installation. The data re-import task is already running.
We've lost the data for a couple of months but in this case this does
not matter.
Nevertheless we will try what you told us - just to be smarter/faster if
this happens in production (where we will setup a cassandra cluster with
multiple cassandra nodes anyway). I will drop you a note when we are
done.
Hmmm... the problem is within a "View". Are this the materialized views?
I'm asking this because:
* Someone on the internet (stackoverflow if a recall correctly)
mentioned that using materialized views are to be deprecated.
* I had been on a datastax workshop in Zurich a couple of days ago where
a datastax employee told me that we should not use materialized views -
it is better to create & fill all tables directly.
Would you also recommend not to use materialized views? As this problem
is related to a view - maybe we could avoid this problem simply by
following this recommendation.
Thanks a lot again!
Greetings,
Michael
On 06.06.2018 16:48, Evelyn Smith wrote:
> Hi Michael,
>
> So I looked at the code, here are some stages of your error message:
> 1. at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:292)
> [apache-cassandra-3.11.0.jar:3.11.0
> At this step Cassandra is running through the keyspaces in it’s
> schema turning off compactions for all tables before it starts
> rerunning the commit log (so it isn’t an issue with the commit log).
> 2. at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> Loading key space related to the column family that is erroring out
> 3. at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> Cassandra has initialised the column family and is reloading the view
> 4. at
> org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:204)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> At this point I haven’t had enough time to tell if Cassandra is
> requesting info on a column specifically or still requesting
> information on a column family. Regardless, given we already rule out
> issues with the SSTables and their directory and Cassandra is yet to
> start processing the commit log this to me suggests it’s something
> wrong in one of the system keyspaces storing the schema information.
>
> There should definitely be a way to resolve this with zero data loss
> by either:
> 1. Fixing the issue in the system keyspace SSTables (hard)
> 2. Rerunning the commit log on a new Cassandra node that has been
> restored from the current one (I’m not sure if this is possible but
> I’ll figure it out tomorrow)
>
> The alternative is if you are ok with losing the commitlog then you
> can backup the data and restore it to a new node (or the same node but
> with everything blown away). This isn’t a trivial process though
> I’ve done it a few times.
>
> How important is the data?
>
> Happy to come back to this tomorrow (need some sleep)
>
> Regards,
> Eevee.
>
>> On 5 Jun 2018, at 7:32 pm, mm@vis.at wrote:
>> Keyspace.getColumnFamilyStore
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Single Host: Fix "Unknown CF" issue
Posted by Evelyn Smith <u5...@gmail.com>.
Hi Michael,
So I looked at the code, here are some stages of your error message:
1. at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:292) [apache-cassandra-3.11.0.jar:3.11.0
At this step Cassandra is running through the keyspaces in it’s schema turning off compactions for all tables before it starts rerunning the commit log (so it isn’t an issue with the commit log).
2. at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127) ~[apache-cassandra-3.11.0.jar:3.11.0]
Loading key space related to the column family that is erroring out
3. at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324) ~[apache-cassandra-3.11.0.jar:3.11.0]
Cassandra has initialised the column family and is reloading the view
4. at org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:204) ~[apache-cassandra-3.11.0.jar:3.11.0]
At this point I haven’t had enough time to tell if Cassandra is requesting info on a column specifically or still requesting information on a column family. Regardless, given we already rule out issues with the SSTables and their directory and Cassandra is yet to start processing the commit log this to me suggests it’s something wrong in one of the system keyspaces storing the schema information.
There should definitely be a way to resolve this with zero data loss by either:
1. Fixing the issue in the system keyspace SSTables (hard)
2. Rerunning the commit log on a new Cassandra node that has been restored from the current one (I’m not sure if this is possible but I’ll figure it out tomorrow)
The alternative is if you are ok with losing the commitlog then you can backup the data and restore it to a new node (or the same node but with everything blown away). This isn’t a trivial process though I’ve done it a few times.
How important is the data?
Happy to come back to this tomorrow (need some sleep)
Regards,
Eevee.
> On 5 Jun 2018, at 7:32 pm, mm@vis.at wrote:
>
> Keyspace.getColumnFamilyStore
Re: Single Host: Fix "Unknown CF" issue
Posted by mm...@vis.at.
Hello Evelyn,
if we do what you say cassandra creates a new, empty column family
directory on restart and the stops with the same exception.
One point i forgot to mention is that we've still got data in the commit
log. (99.9% for other keyspaces).
Are there any tools which we can use to examine the sstable or the
commitlog files to dig further into this issue?
Or:
Can we setup a new cassandra installation and copy the data of the other
keyspacs to the new instance?
The column family which shows this error does not contain important
data. We're just very uncertain now what to do if this happens with
important data.
(ok - maybe this problem is not that important if we use a cluster of
e.g. 3 nodes and a replication factor of 2 - but it leaves a strange
impression about cassandra anyway).
greetings,
Michael
On 05.06.2018 15:31, Evelyn Smith wrote:
> Hey Michael,
>
> I have a hunch.
>
> If the system doesn’t recognise the column family which is stopping
> the node from starting perhaps try copying the column family directory
> to a backup then deleting it.
>
> Then restart Cassandra. If it starts I’ll assume the schema didn’t
> have the column family:
> * Create the column family again (be careful to create it exactly how
> it was in the original schema);
> * Stop Cassandra again;
> * Move the SSTables from the column family backup into the new column
> family folder (you have to do this as the column family folder will
> have a UUID in it’s name that will have changed); and
> * Restart Cassandra.
> You should now have Cassandra running without losing your data.
>
> If Cassandra doesn’t restart after deleting the column family
> directory then just restore it from the backup and you are back to
> square one.
>
> Regards,
> Evelyn.
>
>> On 5 Jun 2018, at 7:32 pm, mm@vis.at wrote:
>>
>> Hi all!
>>
>> We're using cassandra since a couple of month to get familiar with it.
>> We're currently using only 1-node. Yesterday our server had to be
>> restarted and now cassandra does not start anymore.
>>
>> It reports:
>> INFO [main] 2018-06-05 09:50:43,030 ColumnFamilyStore.java:406 -
>> Initializing system_schema.indexes
>> INFO [main] 2018-06-05 09:50:43,036 ViewManager.java:137 - Not
>> submitting build tasks for views in keyspace system_schema as storage
>> service is not initialized
>> INFO [main] 2018-06-05 09:50:43,283 ColumnFamilyStore.java:406 -
>> Initializing system_traces.events
>> INFO [main] 2018-06-05 09:50:43,286 ColumnFamilyStore.java:406 -
>> Initializing system_traces.sessions
>> INFO [main] 2018-06-05 09:50:43,287 ViewManager.java:137 - Not
>> submitting build tasks for views in keyspace system_traces as storage
>> service is not initialized
>> INFO [main] 2018-06-05 09:50:43,300 ColumnFamilyStore.java:406 -
>> Initializing m2m_auth.user
>> INFO [main] 2018-06-05 09:50:43,302 ColumnFamilyStore.java:406 -
>> Initializing m2m_auth.eventsbytag1
>> INFO [main] 2018-06-05 09:50:43,306 ColumnFamilyStore.java:406 -
>> Initializing m2m_auth.mail2user
>> ERROR [main] 2018-06-05 09:50:43,311 CassandraDaemon.java:706 -
>> Exception encountered during startup
>> java.lang.IllegalArgumentException: Unknown CF
>> 0f6c8b36-5f34-11e8-a476-c93745f84272
>> at
>> org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:204)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> at
>> org.apache.cassandra.db.view.ViewManager.addView(ViewManager.java:152)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> at
>> org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:125)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:104)
>> ~[apache-cassandra-3.11.0.jar:3.11.0]
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:292)
>> [apache-cassandra-3.11.0.jar:3.11.0]
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600)
>> [apache-cassandra-3.11.0.jar:3.11.0]
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689)
>> [apache-cassandra-3.11.0.jar:3.11.0]
>>
>> We do not know how to bring it up again and e.g. stackoverflow only
>> recommends to delete the node and let it rebuild (which we cannot do
>> as we've only 1 node currently).
>> Can anybody drop us a hint ?
>>
>> What can we do if our node does not start?
>>
>> We've found an entry in the data directory with the given UUID but at
>> the time when we restarted the node the whole keyspace was idle for a
>> couple of hours.
>> With the problem we're now _very_ concerned about data safety when we
>> read that it can be a problem if e.g. tables are
>> created/deleted/updated concurrently. But in our case we did not
>> create/update tables concurrently and got this problem anyway....
>>
>> Thanks for any help!
>>
>> greetings,
>> Michael
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Single Host: Fix "Unknown CF" issue
Posted by Evelyn Smith <u5...@gmail.com>.
Hey Michael,
I have a hunch.
If the system doesn’t recognise the column family which is stopping the node from starting perhaps try copying the column family directory to a backup then deleting it.
Then restart Cassandra. If it starts I’ll assume the schema didn’t have the column family:
* Create the column family again (be careful to create it exactly how it was in the original schema);
* Stop Cassandra again;
* Move the SSTables from the column family backup into the new column family folder (you have to do this as the column family folder will have a UUID in it’s name that will have changed); and
* Restart Cassandra.
You should now have Cassandra running without losing your data.
If Cassandra doesn’t restart after deleting the column family directory then just restore it from the backup and you are back to square one.
Regards,
Evelyn.
> On 5 Jun 2018, at 7:32 pm, mm@vis.at wrote:
>
> Hi all!
>
> We're using cassandra since a couple of month to get familiar with it. We're currently using only 1-node. Yesterday our server had to be restarted and now cassandra does not start anymore.
>
> It reports:
> INFO [main] 2018-06-05 09:50:43,030 ColumnFamilyStore.java:406 - Initializing system_schema.indexes
> INFO [main] 2018-06-05 09:50:43,036 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_schema as storage service is not initialized
> INFO [main] 2018-06-05 09:50:43,283 ColumnFamilyStore.java:406 - Initializing system_traces.events
> INFO [main] 2018-06-05 09:50:43,286 ColumnFamilyStore.java:406 - Initializing system_traces.sessions
> INFO [main] 2018-06-05 09:50:43,287 ViewManager.java:137 - Not submitting build tasks for views in keyspace system_traces as storage service is not initialized
> INFO [main] 2018-06-05 09:50:43,300 ColumnFamilyStore.java:406 - Initializing m2m_auth.user
> INFO [main] 2018-06-05 09:50:43,302 ColumnFamilyStore.java:406 - Initializing m2m_auth.eventsbytag1
> INFO [main] 2018-06-05 09:50:43,306 ColumnFamilyStore.java:406 - Initializing m2m_auth.mail2user
> ERROR [main] 2018-06-05 09:50:43,311 CassandraDaemon.java:706 - Exception encountered during startup
> java.lang.IllegalArgumentException: Unknown CF 0f6c8b36-5f34-11e8-a476-c93745f84272
> at org.apache.cassandra.db.Keyspace.getColumnFamilyStore(Keyspace.java:204) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.db.view.ViewManager.addView(ViewManager.java:152) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.db.view.ViewManager.reload(ViewManager.java:125) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:324) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:127) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:104) ~[apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:292) [apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:600) [apache-cassandra-3.11.0.jar:3.11.0]
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:689) [apache-cassandra-3.11.0.jar:3.11.0]
>
> We do not know how to bring it up again and e.g. stackoverflow only recommends to delete the node and let it rebuild (which we cannot do as we've only 1 node currently).
> Can anybody drop us a hint ?
>
> What can we do if our node does not start?
>
> We've found an entry in the data directory with the given UUID but at the time when we restarted the node the whole keyspace was idle for a couple of hours.
> With the problem we're now _very_ concerned about data safety when we read that it can be a problem if e.g. tables are created/deleted/updated concurrently. But in our case we did not create/update tables concurrently and got this problem anyway....
>
> Thanks for any help!
>
> greetings,
> Michael
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org