You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@cassandra.apache.org by Eunsu Kim <eu...@gmail.com> on 2018/01/12 07:14:51 UTC

Even after the drop table, the data actually was not erased.

hi everyone

On the development server, I dropped all the tables and even keyspace dropped to change the table schema.
Then I created the keyspace and the table.

However, the actual size of the data directory did not decrease at all. Disk Load monitored by JMX has been decreased.




After that, Cassandra does not work normally.

The client fails to establish a connection and I see the following exceptions in the Cassandra logs.

org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId…….org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch exception……


After the data is forcibly deleted, Cassandra is restarted in a clean state and works well.

Can anyone guess why this is happening?

Thank you in advance.

RE: [EXTERNAL] Re: Even after the drop table, the data actually was not erased.

Posted by "Durity, Sean R" <SE...@homedepot.com>.

We have found it very useful to set up an infrastructure where we can execute a nodetool command (or any other arbitrary command) from a single (non-Cassandra) host that will get executed on each node across the cluster (or a list of nodes).

Sean Durity

From: Alain RODRIGUEZ [mailto:arodrime@gmail.com]
Sent: Monday, January 15, 2018 1:19 PM
To: user cassandra.apache.org <us...@cassandra.apache.org>
Subject: [EXTERNAL] Re: Even after the drop table, the data actually was not erased.

As you said, the auto_bootstrap setting was turned on.

Well I was talking about the 'auto_snapshot' ;-). I understand that's what you meant to say.

This command seems to apply only to one node. Can it be applied cluster-wide? Or should I run this command on each node?

Indeed, 'nodetool clearsnapshot' is only for the node where you run the command, like most of the nodetool commands (repair is a bit specific).

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com<ma...@thelastpickle.com>
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.thelastpickle.com&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=4-1G2nWjQZ3g8ZYnEhVV_NcG9TtjHJQOvqGGS1yS2KY&s=6pnDPB6ROBIPCE3CTHw-S8py99MKIG_kVcap49KKkP0&e=>

2018-01-15 1:56 GMT+00:00 Eunsu Kim <eu...@gmail.com>>:
Thank you for your response.  As you said, the auto_bootstrap setting was turned on.
The actual data was deleted with the 'nodetool clearsnapshot' command.
This command seems to apply only to one node. Can it be applied cluster-wide? Or should I run this command on each node?

On 12 Jan 2018, at 8:10 PM, Alain RODRIGUEZ <ar...@gmail.com>> wrote:

Hello,

However, the actual size of the data directory did not decrease at all. Disk Load monitored by JMX has been decreased.

This sounds like 'auto_snapshot' is enabled. This option will trigger a snapshot before any table drop / truncate to prevent user mistakes mostly. Then the data is removed but as it is still referenced by the snapshot (hard link), space cannot be freed.

Running 'nodetool clearsnapshot' should help reducing the dataset size in this situation.

The client fails to establish a connection and I see the following exceptions in the Cassandra logs.
org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId…

This does not look like a failed connection to me but rather a try to query some inexistent data. If that's the data you just deleted (keyspace / table), this is expected. If not there is an other issue, I hope not related to the delete in this case...

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com<ma...@thelastpickle.com>
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.thelastpickle.com_&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=4-1G2nWjQZ3g8ZYnEhVV_NcG9TtjHJQOvqGGS1yS2KY&s=N4OglLwD-qj1p9otMAQ08CD4i88-uksigSEFHKrqmc4&e=>

2018-01-12 7:14 GMT+00:00 Eunsu Kim <eu...@gmail.com>>:
hi everyone

On the development server, I dropped all the tables and even keyspace dropped to change the table schema.
Then I created the keyspace and the table.

However, the actual size of the data directory did not decrease at all. Disk Load monitored by JMX has been decreased.

<Screen Shot 2018-01-12 at 3.50.22 PM.png>

After that, Cassandra does not work normally.

The client fails to establish a connection and I see the following exceptions in the Cassandra logs.

org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId…….org.apache.cassandra.io<https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.cassandra.io&d=DwMFaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=4-1G2nWjQZ3g8ZYnEhVV_NcG9TtjHJQOvqGGS1yS2KY&s=_R6V8rSsGURn7qbw7xM5Eo4P_mfxvXN2IcDa31iIQGU&e=>.FSReadError: java.io.IOException: Digest mismatch exception……

After the data is forcibly deleted, Cassandra is restarted in a clean state and works well.

Can anyone guess why this is happening?

Thank you in advance.

________________________________

The information in this Internet Email is confidential and may be legally privileged. It is intended solely for the addressee. Access to this Email by anyone else is unauthorized. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. When addressed to our clients any opinions or advice contained in this Email are subject to the terms and conditions expressed in any applicable governing The Home Depot terms of business or client engagement letter. The Home Depot disclaims all responsibility and liability for the accuracy and content of this attachment and for any damages or losses arising from any inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other items of a destructive nature, which may be contained in this attachment and shall not be liable for direct, indirect, consequential or special damages in connection with this e-mail message or its attachment.

Re: Even after the drop table, the data actually was not erased.

Posted by Alain RODRIGUEZ <ar...@gmail.com>.

>
> As you said, the auto_bootstrap setting was turned on.


Well I was talking about the 'auto_snapshot' ;-). I understand that's what
you meant to say.

This command seems to apply only to one node. Can it be applied
> cluster-wide? Or should I run this command on each node?


Indeed, 'nodetool clearsnapshot' is only for the node where you run the
command, like most of the nodetool commands (repair is a bit specific).

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-01-15 1:56 GMT+00:00 Eunsu Kim <eu...@gmail.com>:

> Thank you for your response.  As you said, the auto_bootstrap setting was
> turned on.
> The actual data was deleted with the 'nodetool clearsnapshot' command.
> This command seems to apply only to one node. Can it be applied
> cluster-wide? Or should I run this command on each node?
>
>
>
> On 12 Jan 2018, at 8:10 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>
> Hello,
>
> However, the actual size of the data directory did not decrease at all.
>> Disk Load monitored by JMX has been decreased.
>
>
> This sounds like 'auto_snapshot' is enabled. This option will trigger a
> snapshot before any table drop / truncate to prevent user mistakes mostly.
> Then the data is removed but as it is still referenced by the snapshot
> (hard link), space cannot be freed.
>
> Running 'nodetool clearsnapshot' should help reducing the dataset size in
> this situation.
>
>
> The client fails to establish a connection and I see the following
>> exceptions in the Cassandra logs.
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table
>> for cfId…
>
>
> This does not look like a failed connection to me but rather a try to
> query some inexistent data. If that's the data you just deleted (keyspace /
> table), this is expected. If not there is an other issue, I hope not
> related to the delete in this case...
>
> C*heers,
> -----------------------
> Alain Rodriguez - @arodream - alain@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
>
> 2018-01-12 7:14 GMT+00:00 Eunsu Kim <eu...@gmail.com>:
>
>> hi everyone
>>
>> On the development server, I dropped all the tables and even keyspace
>> dropped to change the table schema.
>> Then I created the keyspace and the table.
>>
>> However, the actual size of the data directory did not decrease at all.
>> Disk Load monitored by JMX has been decreased.
>>
>> <Screen Shot 2018-01-12 at 3.50.22 PM.png>
>>
>>
>> After that, Cassandra does not work normally.
>>
>> The client fails to establish a connection and I see the following
>> exceptions in the Cassandra logs.
>>
>> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find
>> table for cfId…….org.apache.cassandra.io.FSReadError:
>> java.io.IOException: Digest mismatch exception……
>>
>>
>> After the data is forcibly deleted, Cassandra is restarted in a clean
>> state and works well.
>>
>> Can anyone guess why this is happening?
>>
>> Thank you in advance.
>>
>
>
>

Re: Even after the drop table, the data actually was not erased.

Posted by Eunsu Kim <eu...@gmail.com>.

Thank you for your response.  As you said, the auto_bootstrap setting was turned on. 
The actual data was deleted with the 'nodetool clearsnapshot' command.
This command seems to apply only to one node. Can it be applied cluster-wide? Or should I run this command on each node?



> On 12 Jan 2018, at 8:10 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> 
> Hello,
> 
> However, the actual size of the data directory did not decrease at all. Disk Load monitored by JMX has been decreased.
> 
> This sounds like 'auto_snapshot' is enabled. This option will trigger a snapshot before any table drop / truncate to prevent user mistakes mostly. Then the data is removed but as it is still referenced by the snapshot (hard link), space cannot be freed.
> 
> Running 'nodetool clearsnapshot' should help reducing the dataset size in this situation.
> 
> 
> The client fails to establish a connection and I see the following exceptions in the Cassandra logs. 
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId…
> 
> This does not look like a failed connection to me but rather a try to query some inexistent data. If that's the data you just deleted (keyspace / table), this is expected. If not there is an other issue, I hope not related to the delete in this case...
> 
> C*heers,
> -----------------------
> Alain Rodriguez - @arodream - alain@thelastpickle.com <ma...@thelastpickle.com>
> France / Spain
> 
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com <http://www.thelastpickle.com/>
> 
> 
> 
> 2018-01-12 7:14 GMT+00:00 Eunsu Kim <eunsu.bill23@gmail.com <ma...@gmail.com>>:
> hi everyone
> 
> On the development server, I dropped all the tables and even keyspace dropped to change the table schema.
> Then I created the keyspace and the table.
> 
> However, the actual size of the data directory did not decrease at all. Disk Load monitored by JMX has been decreased.
> 
> <Screen Shot 2018-01-12 at 3.50.22 PM.png>
> 
> 
> After that, Cassandra does not work normally.
> 
> The client fails to establish a connection and I see the following exceptions in the Cassandra logs.
> 
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId…….org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch exception……
> 
> 
> After the data is forcibly deleted, Cassandra is restarted in a clean state and works well.
> 
> Can anyone guess why this is happening?
> 
> Thank you in advance.
>

Re: Even after the drop table, the data actually was not erased.

Posted by Alain RODRIGUEZ <ar...@gmail.com>.

Hello,

However, the actual size of the data directory did not decrease at all.
> Disk Load monitored by JMX has been decreased.


This sounds like 'auto_snapshot' is enabled. This option will trigger a
snapshot before any table drop / truncate to prevent user mistakes mostly.
Then the data is removed but as it is still referenced by the snapshot
(hard link), space cannot be freed.

Running 'nodetool clearsnapshot' should help reducing the dataset size in
this situation.


The client fails to establish a connection and I see the following
> exceptions in the Cassandra logs.

org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table
> for cfId…


This does not look like a failed connection to me but rather a try to query
some inexistent data. If that's the data you just deleted (keyspace /
table), this is expected. If not there is an other issue, I hope not
related to the delete in this case...

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com



2018-01-12 7:14 GMT+00:00 Eunsu Kim <eu...@gmail.com>:

> hi everyone
>
> On the development server, I dropped all the tables and even keyspace
> dropped to change the table schema.
> Then I created the keyspace and the table.
>
> However, the actual size of the data directory did not decrease at all.
> Disk Load monitored by JMX has been decreased.
>
>
>
> After that, Cassandra does not work normally.
>
> The client fails to establish a connection and I see the following
> exceptions in the Cassandra logs.
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table
> for cfId…….org.apache.cassandra.io.FSReadError: java.io.IOException:
> Digest mismatch exception……
>
>
> After the data is forcibly deleted, Cassandra is restarted in a clean
> state and works well.
>
> Can anyone guess why this is happening?
>
> Thank you in advance.
>