You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Saladi Naidu <na...@yahoo.com> on 2015/07/10 18:15:49 UTC
DROP Table
My understanding is that Cassandra File Structure follows below naming convention
/cassandra/data/ <key-spaces> <table>
Whereas our file structure is as below, each table has multiple names and when we drop tables and recreate these directories remain. Also when we dropped the table one node was down, when it came back, we tried to do Nodetool repair and repair kept failing referring to CFID error listed below
drwxr-xr-x. 16 cass cass 4096 May 24 06:49 ../
drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09application_by_user-e0eec95019a211e58b954ffc8e9bfaa6/
drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15application_info-4dba2bf0054f11e58b954ffc8e9bfaa6/
drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09application_info-a0ee65d019a311e58b954ffc8e9bfaa6/
drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09configproperties-228ea2e0c13811e4aa1d4ffc8e9bfaa6/
drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09user_activation-95d005f019a311e58b954ffc8e9bfaa6/
drwxr-xr-x. 3 cass cass 4096 Jun 25 10:16user_app_permission-9fddcd62ffbe11e4a25a45259f96ec68/
drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09user_credential-86cfff1019a311e58b954ffc8e9bfaa6/
drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09user_info-2fa076221b1011e58b954ffc8e9bfaa6/
drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15user_info-36028c00054f11e58b954ffc8e9bfaa6/
drwxr-xr-x. 3 cass cass 4096 Jun 25 10:15user_info-fe1d7b101a5711e58b954ffc8e9bfaa6/
drwxr-xr-x. 4 cass cass 4096 Jun 25 10:16user_role-9ed0ca30ffbe11e4b71d09335ad2d5a9/
WARN [Thread-2579] 2015-07-02 16:02:27,523 IncomingTcpConnection.java:91 -UnknownColumnFamilyException reading from socket; closing
org.apache.cassandra.db.UnknownColumnFamilyException:Couldn't findcfId=218e3c90-1b0e-11e5-a34b-d7c17b3e318a
atorg.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)~[apache-cassandra-2.1.2.jar:2.1.2]
atorg.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)~[apache-cassandra-2.1.2.jar:2.1.2]
atorg.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)~[apache-cassandra-2.1.2.jar:2.1.2]
at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)~[apache-cassandra-2.1.2.jar:2.1.2]
atorg.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)~[apache-cassandra-2.1.2.jar:2.1.2]
atorg.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)~[apache-cassandra-2.1.2.jar:2.1.2]
Naidu Saladi
Re: DROP Table
Posted by Sebastian Estevez <se...@datastax.com>.
When you say schema to settle, do you means we provide proper consistency
level? I don't think there is a provision to do that in tool. Or I can
change the SYSTEM KEYSPACE definition of replication factor equal to number
of nodes?
I mean the nodes have to all be on the same schema version.
In the steps described below for correcting this problem - when you say
move data from old directory to new, do you mean move the .db file? It
will override the current file right?
Yes copy the db files. No they will not override existing data.
Do we have to rename the directory name to remove CFID i.e. just
column family name without CFID? After that, update the System table as
well?
No, follow my instructions precisely, do not rename the directories, just
copy to the new one and rm the old one.
On Jul 13, 2015 2:52 PM, "Saladi Naidu" <na...@yahoo.com> wrote:
> Sebastian,
> Thank you so much for providing detailed explanation. I still have some
> questions and I need to provide some clarifications
>
> 1. We do not have code that is creating the tables dynamically. All DDL
> operations are done through Datastax DevCenter tool. When you say schema to
> settle, do you means we provide proper consistency level? I don't think
> there is a provision to do that in tool. Or I can change the SYSTEM
> KEYSPACE definition of replication factor equal to number of nodes?
>
> 2. In the steps described below for correcting this problem - when you say
> move data from old directory to new, do you mean move the .db file? It
> will override the current file right?
>
> 3. Do we have to rename the directory name to remove CFID i.e. just
> column family name without CFID? After that, update the System table as
> well?
>
>
> Naidu Saladi
>
> ------------------------------
> *From:* Sebastian Estevez <se...@datastax.com>
> *To:* user@cassandra.apache.org; Saladi Naidu <na...@yahoo.com>
> *Sent:* Friday, July 10, 2015 5:25 PM
> *Subject:* Re: DROP Table
>
> #1 The cause of this problem is a CREATE TABLE statement collision. Do
> *not* generate tables dynamically from multiple clients, even with IF NOT
> EXISTS. First thing you need to do is fix your code so that this does not
> happen. Just create your tables manually from cqlsh allowing time for the
> schema to settle.
>
> #2 Here's the fix:
>
> 1) *Change your code to not automatically re-create tables (even with IF
> NOT EXISTS).*
>
> 2) Run a rolling restart to ensure schema matches across nodes. Run
> nodetool describecluster around your cluster. Check that there is only one
> schema version.
>
> ON EACH NODE:
> 3) Check your filesystem and see if you have two directories for the table
> in question in the data directory.
>
> If THERE ARE TWO OR MORE DIRECTORIES:
> 4)Identify from schema_column_families which cf ID is the "new" one
> (currently in use).
>
> cqlsh -e "select * from system.schema_column_families"|grep <table name>
>
>
> 5) Move the data from the "old" one to the "new" one and remove the old
> directory.
>
> 6) If there are multiple "old" ones repeat 5 for every "old" directory.
>
> 7) run nodetool refresh
>
> IF THERE IS ONLY ONE DIRECTORY:
>
> No further action is needed.
>
> All the best,
>
> [image: datastax_logo.png] <http://www.datastax.com/>
> Sebastián Estévez
> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
>
> <http://cassandrasummit-datastax.com/>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
>
>
> On Fri, Jul 10, 2015 at 12:15 PM, Saladi Naidu <na...@yahoo.com>
> wrote:
>
> My understanding is that Cassandra File Structure follows below naming
> convention
>
> /cassandra/*data/ <key-spaces> <table>*
>
>
>
> Whereas our file structure is as below, each table has multiple names and
> when we drop tables and recreate these directories remain. Also when we
> dropped the table one node was down, when it came back, we tried to do
> Nodetool repair and repair kept failing referring to CFID error listed
> below
>
>
> drwxr-xr-x. 16 cass cass 4096 May 24 06:49 ../
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> application_by_user-e0eec95019a211e58b954ffc8e9bfaa6/
> drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15 application_info-
> 4dba2bf0054f11e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> application_info-a0ee65d019a311e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> configproperties-228ea2e0c13811e4aa1d4ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> user_activation-95d005f019a311e58b954ffc8e9bfaa6/
> drwxr-xr-x. 3 cass cass 4096 Jun 25 10:16
> user_app_permission-9fddcd62ffbe11e4a25a45259f96ec68/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> user_credential-86cfff1019a311e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> user_info-2fa076221b1011e58b954ffc8e9bfaa6/
> drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15
> user_info-36028c00054f11e58b954ffc8e9bfaa6/
> drwxr-xr-x. 3 cass cass 4096 Jun 25 10:15
> user_info-fe1d7b101a5711e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jun 25 10:16
> user_role-9ed0ca30ffbe11e4b71d09335ad2d5a9/
>
>
> WARN [Thread-2579] 2015-07-02 16:02:27,523 IncomingTcpConnection.java:91
> - UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=218e3c90-1b0e-11e5-a34b-d7c17b3e318a
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
>
>
> Naidu Saladi
>
>
>
>
>
Re: DROP Table
Posted by Sebastian Estevez <se...@datastax.com>.
Thank you Mikhail. I'll take a look at your utility.
On Jul 13, 2015 5:30 PM, "Mikhail Strebkov" <st...@gmail.com> wrote:
> Hi Saladi,
>
> Recently I faced a similar problem, I had a lot of CFs to fix, so I wrote
> this: https://github.com/kluyg/cassandra-schema-fix
> I think it can be useful to you.
>
> Kind regards,
> Mikhail
>
> On Mon, Jul 13, 2015 at 11:51 AM, Saladi Naidu <na...@yahoo.com>
> wrote:
>
>> Sebastian,
>> Thank you so much for providing detailed explanation. I still have some
>> questions and I need to provide some clarifications
>>
>> 1. We do not have code that is creating the tables dynamically. All DDL
>> operations are done through Datastax DevCenter tool. When you say schema to
>> settle, do you means we provide proper consistency level? I don't think
>> there is a provision to do that in tool. Or I can change the SYSTEM
>> KEYSPACE definition of replication factor equal to number of nodes?
>>
>> 2. In the steps described below for correcting this problem - when you
>> say move data from old directory to new, do you mean move the .db file? It
>> will override the current file right?
>>
>> 3. Do we have to rename the directory name to remove CFID i.e. just
>> column family name without CFID? After that, update the System table as
>> well?
>>
>>
>> Naidu Saladi
>>
>> ------------------------------
>> *From:* Sebastian Estevez <se...@datastax.com>
>> *To:* user@cassandra.apache.org; Saladi Naidu <na...@yahoo.com>
>> *Sent:* Friday, July 10, 2015 5:25 PM
>> *Subject:* Re: DROP Table
>>
>> #1 The cause of this problem is a CREATE TABLE statement collision. Do
>> *not* generate tables dynamically from multiple clients, even with IF
>> NOT EXISTS. First thing you need to do is fix your code so that this does
>> not happen. Just create your tables manually from cqlsh allowing time for
>> the schema to settle.
>>
>> #2 Here's the fix:
>>
>> 1) *Change your code to not automatically re-create tables (even with IF
>> NOT EXISTS).*
>>
>> 2) Run a rolling restart to ensure schema matches across nodes. Run
>> nodetool describecluster around your cluster. Check that there is only one
>> schema version.
>>
>> ON EACH NODE:
>> 3) Check your filesystem and see if you have two directories for the
>> table in question in the data directory.
>>
>> If THERE ARE TWO OR MORE DIRECTORIES:
>> 4)Identify from schema_column_families which cf ID is the "new" one
>> (currently in use).
>>
>> cqlsh -e "select * from system.schema_column_families"|grep <table name>
>>
>>
>> 5) Move the data from the "old" one to the "new" one and remove the old
>> directory.
>>
>> 6) If there are multiple "old" ones repeat 5 for every "old" directory.
>>
>> 7) run nodetool refresh
>>
>> IF THERE IS ONLY ONE DIRECTORY:
>>
>> No further action is needed.
>>
>> All the best,
>>
>> [image: datastax_logo.png] <http://www.datastax.com/>
>> Sebastián Estévez
>> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
>> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
>> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
>> <https://twitter.com/datastax> [image: g+.png]
>> <https://plus.google.com/+Datastax/about>
>> <http://feeds.feedburner.com/datastax>
>>
>> <http://cassandrasummit-datastax.com/>
>>
>> DataStax is the fastest, most scalable distributed database technology,
>> delivering Apache Cassandra to the world’s most innovative enterprises.
>> Datastax is built to be agile, always-on, and predictably scalable to any
>> size. With more than 500 customers in 45 countries, DataStax is the
>> database technology and transactional backbone of choice for the worlds
>> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>>
>>
>>
>> On Fri, Jul 10, 2015 at 12:15 PM, Saladi Naidu <na...@yahoo.com>
>> wrote:
>>
>> My understanding is that Cassandra File Structure follows below naming
>> convention
>>
>> /cassandra/*data/ <key-spaces> <table>*
>>
>>
>>
>> Whereas our file structure is as below, each table has multiple names and
>> when we drop tables and recreate these directories remain. Also when we
>> dropped the table one node was down, when it came back, we tried to do
>> Nodetool repair and repair kept failing referring to CFID error listed
>> below
>>
>>
>> drwxr-xr-x. 16 cass cass 4096 May 24 06:49 ../
>> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
>> application_by_user-e0eec95019a211e58b954ffc8e9bfaa6/
>> drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15 application_info-
>> 4dba2bf0054f11e58b954ffc8e9bfaa6/
>> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
>> application_info-a0ee65d019a311e58b954ffc8e9bfaa6/
>> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
>> configproperties-228ea2e0c13811e4aa1d4ffc8e9bfaa6/
>> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
>> user_activation-95d005f019a311e58b954ffc8e9bfaa6/
>> drwxr-xr-x. 3 cass cass 4096 Jun 25 10:16
>> user_app_permission-9fddcd62ffbe11e4a25a45259f96ec68/
>> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
>> user_credential-86cfff1019a311e58b954ffc8e9bfaa6/
>> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
>> user_info-2fa076221b1011e58b954ffc8e9bfaa6/
>> drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15
>> user_info-36028c00054f11e58b954ffc8e9bfaa6/
>> drwxr-xr-x. 3 cass cass 4096 Jun 25 10:15
>> user_info-fe1d7b101a5711e58b954ffc8e9bfaa6/
>> drwxr-xr-x. 4 cass cass 4096 Jun 25 10:16
>> user_role-9ed0ca30ffbe11e4b71d09335ad2d5a9/
>>
>>
>> WARN [Thread-2579] 2015-07-02 16:02:27,523 IncomingTcpConnection.java:91
>> - UnknownColumnFamilyException reading from socket; closing
>> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
>> cfId=218e3c90-1b0e-11e5-a34b-d7c17b3e318a
>> at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
>> ~[apache-cassandra-2.1.2.jar:2.1.2]
>> at
>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
>> ~[apache-cassandra-2.1.2.jar:2.1.2]
>> at
>> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
>> ~[apache-cassandra-2.1.2.jar:2.1.2]
>> at
>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
>> ~[apache-cassandra-2.1.2.jar:2.1.2]
>> at
>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
>> ~[apache-cassandra-2.1.2.jar:2.1.2]
>> at
>> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
>> ~[apache-cassandra-2.1.2.jar:2.1.2]
>> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
>> ~[apache-cassandra-2.1.2.jar:2.1.2]
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
>> ~[apache-cassandra-2.1.2.jar:2.1.2]
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
>> ~[apache-cassandra-2.1.2.jar:2.1.2]
>> at
>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
>> ~[apache-cassandra-2.1.2.jar:2.1.2]
>>
>>
>> Naidu Saladi
>>
>>
>>
>>
>>
>
Re: DROP Table
Posted by Mikhail Strebkov <st...@gmail.com>.
Hi Saladi,
Recently I faced a similar problem, I had a lot of CFs to fix, so I wrote
this: https://github.com/kluyg/cassandra-schema-fix
I think it can be useful to you.
Kind regards,
Mikhail
On Mon, Jul 13, 2015 at 11:51 AM, Saladi Naidu <na...@yahoo.com>
wrote:
> Sebastian,
> Thank you so much for providing detailed explanation. I still have some
> questions and I need to provide some clarifications
>
> 1. We do not have code that is creating the tables dynamically. All DDL
> operations are done through Datastax DevCenter tool. When you say schema to
> settle, do you means we provide proper consistency level? I don't think
> there is a provision to do that in tool. Or I can change the SYSTEM
> KEYSPACE definition of replication factor equal to number of nodes?
>
> 2. In the steps described below for correcting this problem - when you say
> move data from old directory to new, do you mean move the .db file? It
> will override the current file right?
>
> 3. Do we have to rename the directory name to remove CFID i.e. just
> column family name without CFID? After that, update the System table as
> well?
>
>
> Naidu Saladi
>
> ------------------------------
> *From:* Sebastian Estevez <se...@datastax.com>
> *To:* user@cassandra.apache.org; Saladi Naidu <na...@yahoo.com>
> *Sent:* Friday, July 10, 2015 5:25 PM
> *Subject:* Re: DROP Table
>
> #1 The cause of this problem is a CREATE TABLE statement collision. Do
> *not* generate tables dynamically from multiple clients, even with IF NOT
> EXISTS. First thing you need to do is fix your code so that this does not
> happen. Just create your tables manually from cqlsh allowing time for the
> schema to settle.
>
> #2 Here's the fix:
>
> 1) *Change your code to not automatically re-create tables (even with IF
> NOT EXISTS).*
>
> 2) Run a rolling restart to ensure schema matches across nodes. Run
> nodetool describecluster around your cluster. Check that there is only one
> schema version.
>
> ON EACH NODE:
> 3) Check your filesystem and see if you have two directories for the table
> in question in the data directory.
>
> If THERE ARE TWO OR MORE DIRECTORIES:
> 4)Identify from schema_column_families which cf ID is the "new" one
> (currently in use).
>
> cqlsh -e "select * from system.schema_column_families"|grep <table name>
>
>
> 5) Move the data from the "old" one to the "new" one and remove the old
> directory.
>
> 6) If there are multiple "old" ones repeat 5 for every "old" directory.
>
> 7) run nodetool refresh
>
> IF THERE IS ONLY ONE DIRECTORY:
>
> No further action is needed.
>
> All the best,
>
> [image: datastax_logo.png] <http://www.datastax.com/>
> Sebastián Estévez
> Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
> [image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
> facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
> <https://twitter.com/datastax> [image: g+.png]
> <https://plus.google.com/+Datastax/about>
> <http://feeds.feedburner.com/datastax>
>
> <http://cassandrasummit-datastax.com/>
>
> DataStax is the fastest, most scalable distributed database technology,
> delivering Apache Cassandra to the world’s most innovative enterprises.
> Datastax is built to be agile, always-on, and predictably scalable to any
> size. With more than 500 customers in 45 countries, DataStax is the
> database technology and transactional backbone of choice for the worlds
> most innovative companies such as Netflix, Adobe, Intuit, and eBay.
>
>
>
> On Fri, Jul 10, 2015 at 12:15 PM, Saladi Naidu <na...@yahoo.com>
> wrote:
>
> My understanding is that Cassandra File Structure follows below naming
> convention
>
> /cassandra/*data/ <key-spaces> <table>*
>
>
>
> Whereas our file structure is as below, each table has multiple names and
> when we drop tables and recreate these directories remain. Also when we
> dropped the table one node was down, when it came back, we tried to do
> Nodetool repair and repair kept failing referring to CFID error listed
> below
>
>
> drwxr-xr-x. 16 cass cass 4096 May 24 06:49 ../
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> application_by_user-e0eec95019a211e58b954ffc8e9bfaa6/
> drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15 application_info-
> 4dba2bf0054f11e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> application_info-a0ee65d019a311e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> configproperties-228ea2e0c13811e4aa1d4ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> user_activation-95d005f019a311e58b954ffc8e9bfaa6/
> drwxr-xr-x. 3 cass cass 4096 Jun 25 10:16
> user_app_permission-9fddcd62ffbe11e4a25a45259f96ec68/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> user_credential-86cfff1019a311e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> user_info-2fa076221b1011e58b954ffc8e9bfaa6/
> drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15
> user_info-36028c00054f11e58b954ffc8e9bfaa6/
> drwxr-xr-x. 3 cass cass 4096 Jun 25 10:15
> user_info-fe1d7b101a5711e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jun 25 10:16
> user_role-9ed0ca30ffbe11e4b71d09335ad2d5a9/
>
>
> WARN [Thread-2579] 2015-07-02 16:02:27,523 IncomingTcpConnection.java:91
> - UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=218e3c90-1b0e-11e5-a34b-d7c17b3e318a
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
>
>
> Naidu Saladi
>
>
>
>
>
Re: DROP Table
Posted by Saladi Naidu <na...@yahoo.com>.
Sebastian,Thank you so much for providing detailed explanation. I still have some questions and I need to provide some clarifications
1. We do not have code that is creating the tables dynamically. All DDL operations are done through Datastax DevCenter tool. When you say schema to settle, do you means we provide proper consistency level? I don't think there is a provision to do that in tool. Or I can change the SYSTEM KEYSPACE definition of replication factor equal to number of nodes?
2. In the steps described below for correcting this problem - when you say move data from old directory to new, do you mean move the .db file? It will override the current file right?
3. Do we have to rename the directory name to remove CFID i.e. just column family name without CFID? After that, update the System table as well? Naidu Saladi
From: Sebastian Estevez <se...@datastax.com>
To: user@cassandra.apache.org; Saladi Naidu <na...@yahoo.com>
Sent: Friday, July 10, 2015 5:25 PM
Subject: Re: DROP Table
#1 The cause of this problem is a CREATE TABLE statement collision. Do not generate tables dynamically from multiple clients, even with IF NOT EXISTS. First thing you need to do is fix your code so that this does not happen. Just create your tables manually from cqlsh allowing time for the schema to settle.
#2 Here's the fix:
1) Change your code to not automatically re-create tables (even with IF NOT EXISTS).
2) Run a rolling restart to ensure schema matches across nodes. Run nodetool describecluster around your cluster. Check that there is only one schema version.
ON EACH NODE:3) Check your filesystem and see if you have two directories for the table in question in the data directory.
If THERE ARE TWO OR MORE DIRECTORIES:4)Identify from schema_column_families which cf ID is the "new" one (currently in use).
cqlsh -e "select * from system.schema_column_families"|grep <table name>
5) Move the data from the "old" one to the "new" one and remove the old directory.
6) If there are multiple "old" ones repeat 5 for every "old" directory.
7) run nodetool refresh
IF THERE IS ONLY ONE DIRECTORY:
No further action is needed.
All the best,
Sebastián EstévezSolutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
DataStax is the fastest, most scalable distributed database technology, delivering Apache Cassandra to the world’s most innovative enterprises. Datastax is built to be agile, always-on, and predictably scalable to any size. With more than 500 customers in 45 countries, DataStax is the database technology and transactional backbone of choice for the worlds most innovative companies such as Netflix, Adobe, Intuit, and eBay.
On Fri, Jul 10, 2015 at 12:15 PM, Saladi Naidu <na...@yahoo.com> wrote:
My understanding is that Cassandra File Structure follows below naming convention
/cassandra/data/ <key-spaces> <table>
Whereas our file structure is as below, each table has multiple names and when we drop tables and recreate these directories remain. Also when we dropped the table one node was down, when it came back, we tried to do Nodetool repair and repair kept failing referring to CFID error listed below
drwxr-xr-x. 16 cass cass 4096 May 24 06:49 ../drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09application_by_user-e0eec95019a211e58b954ffc8e9bfaa6/drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15application_info-4dba2bf0054f11e58b954ffc8e9bfaa6/drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09application_info-a0ee65d019a311e58b954ffc8e9bfaa6/drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09configproperties-228ea2e0c13811e4aa1d4ffc8e9bfaa6/drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09user_activation-95d005f019a311e58b954ffc8e9bfaa6/drwxr-xr-x. 3 cass cass 4096 Jun 25 10:16user_app_permission-9fddcd62ffbe11e4a25a45259f96ec68/drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09user_credential-86cfff1019a311e58b954ffc8e9bfaa6/drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09user_info-2fa076221b1011e58b954ffc8e9bfaa6/drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15user_info-36028c00054f11e58b954ffc8e9bfaa6/drwxr-xr-x. 3 cass cass 4096 Jun 25 10:15user_info-fe1d7b101a5711e58b954ffc8e9bfaa6/drwxr-xr-x. 4 cass cass 4096 Jun 25 10:16user_role-9ed0ca30ffbe11e4b71d09335ad2d5a9/
WARN [Thread-2579] 2015-07-02 16:02:27,523 IncomingTcpConnection.java:91 -UnknownColumnFamilyException reading from socket; closingorg.apache.cassandra.db.UnknownColumnFamilyException:Couldn't findcfId=218e3c90-1b0e-11e5-a34b-d7c17b3e318a atorg.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)~[apache-cassandra-2.1.2.jar:2.1.2] atorg.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)~[apache-cassandra-2.1.2.jar:2.1.2] atorg.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)~[apache-cassandra-2.1.2.jar:2.1.2] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)~[apache-cassandra-2.1.2.jar:2.1.2] atorg.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)~[apache-cassandra-2.1.2.jar:2.1.2] atorg.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)~[apache-cassandra-2.1.2.jar:2.1.2] Naidu Saladi
Re: DROP Table
Posted by Sebastian Estevez <se...@datastax.com>.
#1 The cause of this problem is a CREATE TABLE statement collision. Do
*not* generate
tables dynamically from multiple clients, even with IF NOT EXISTS. First
thing you need to do is fix your code so that this does not happen. Just
create your tables manually from cqlsh allowing time for the schema to
settle.
#2 Here's the fix:
1) *Change your code to not automatically re-create tables (even with IF
NOT EXISTS).*
2) Run a rolling restart to ensure schema matches across nodes. Run
nodetool describecluster around your cluster. Check that there is only one
schema version.
ON EACH NODE:
3) Check your filesystem and see if you have two directories for the table
in question in the data directory.
If THERE ARE TWO OR MORE DIRECTORIES:
4)Identify from schema_column_families which cf ID is the "new" one
(currently in use).
cqlsh -e "select * from system.schema_column_families"|grep <table name>
5) Move the data from the "old" one to the "new" one and remove the old
directory.
6) If there are multiple "old" ones repeat 5 for every "old" directory.
7) run nodetool refresh
IF THERE IS ONLY ONE DIRECTORY:
No further action is needed.
All the best,
[image: datastax_logo.png] <http://www.datastax.com/>
Sebastián Estévez
Solutions Architect | 954 905 8615 | sebastian.estevez@datastax.com
[image: linkedin.png] <https://www.linkedin.com/company/datastax> [image:
facebook.png] <https://www.facebook.com/datastax> [image: twitter.png]
<https://twitter.com/datastax> [image: g+.png]
<https://plus.google.com/+Datastax/about>
<http://feeds.feedburner.com/datastax>
<http://cassandrasummit-datastax.com/>
DataStax is the fastest, most scalable distributed database technology,
delivering Apache Cassandra to the world’s most innovative enterprises.
Datastax is built to be agile, always-on, and predictably scalable to any
size. With more than 500 customers in 45 countries, DataStax is the
database technology and transactional backbone of choice for the worlds
most innovative companies such as Netflix, Adobe, Intuit, and eBay.
On Fri, Jul 10, 2015 at 12:15 PM, Saladi Naidu <na...@yahoo.com>
wrote:
> My understanding is that Cassandra File Structure follows below naming
> convention
>
> /cassandra/*data/ <key-spaces> <table>*
>
>
>
> Whereas our file structure is as below, each table has multiple names and
> when we drop tables and recreate these directories remain. Also when we
> dropped the table one node was down, when it came back, we tried to do
> Nodetool repair and repair kept failing referring to CFID error listed
> below
>
>
> drwxr-xr-x. 16 cass cass 4096 May 24 06:49 ../
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> application_by_user-e0eec95019a211e58b954ffc8e9bfaa6/
> drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15 application_info-
> 4dba2bf0054f11e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> application_info-a0ee65d019a311e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> configproperties-228ea2e0c13811e4aa1d4ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> user_activation-95d005f019a311e58b954ffc8e9bfaa6/
> drwxr-xr-x. 3 cass cass 4096 Jun 25 10:16
> user_app_permission-9fddcd62ffbe11e4a25a45259f96ec68/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> user_credential-86cfff1019a311e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jul 2 11:09
> user_info-2fa076221b1011e58b954ffc8e9bfaa6/
> drwxr-xr-x. 2 cass cass 4096 Jun 25 10:15
> user_info-36028c00054f11e58b954ffc8e9bfaa6/
> drwxr-xr-x. 3 cass cass 4096 Jun 25 10:15
> user_info-fe1d7b101a5711e58b954ffc8e9bfaa6/
> drwxr-xr-x. 4 cass cass 4096 Jun 25 10:16
> user_role-9ed0ca30ffbe11e4b71d09335ad2d5a9/
>
>
> WARN [Thread-2579] 2015-07-02 16:02:27,523 IncomingTcpConnection.java:91
> - UnknownColumnFamilyException reading from socket; closing
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
> cfId=218e3c90-1b0e-11e5-a34b-d7c17b3e318a
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeCfId(ColumnFamilySerializer.java:164)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:97)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:168)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:150)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:82)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
>
>
> Naidu Saladi
>