You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Gábor Auth <au...@gmail.com> on 2017/10/01 13:23:16 UTC
Alter table gc_grace_seconds
Hi,
The `alter table number_item with gc_grace_seconds = 3600;` is sets the
grace seconds of tombstones of the future modification of number_item
column family or affects all existing data?
Bye,
Gábor Auth
Re: Alter table gc_grace_seconds
Posted by Gábor Auth <au...@gmail.com>.
Hi,
On Sun, Oct 1, 2017 at 6:53 PM Jonathan Haddad <jo...@jonhaddad.com> wrote:
> The TTL is applied to the cells on insert. Changing it doesn't change the
> TTL on data that was inserted previously.
>
Oh! So that the tombstoned cell's TTL is equals with the CF's
gc_grace_seconds value and the repair will be remove it. Am I right?
Bye,
Gábor Auth
Re: Alter table gc_grace_seconds
Posted by Gábor Auth <au...@gmail.com>.
Hi,
On Mon, Oct 2, 2017 at 8:41 AM Varun Barala <va...@gmail.com> wrote:
> Might be possible C* is not compacting the sstables [
> https://stackoverflow.com/questions/28437301/cassandra-not-compacting-sstables
> ]
>
Oh, the other CF-s in the same keyspace are compacted, but the
`number_item` not.
[cassandra@dc02-rack01-cass01
number_item-524bf49001d911e798503511c5f98764]$ ls -l
total 172704
drwxr-xr-x 4 cassandra cassandra 4096 Oct 1 10:43 backups
-rw-r--r-- 1 cassandra cassandra 15227 Oct 2 01:15
mc-48278-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 46562318 Oct 2 01:15 mc-48278-big-Data.db
-rw-r--r-- 1 cassandra cassandra 10 Oct 2 01:15
mc-48278-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra 176 Oct 2 01:15
mc-48278-big-Filter.db
-rw-r--r-- 1 cassandra cassandra 119665 Oct 2 01:15
mc-48278-big-Index.db
-rw-r--r-- 1 cassandra cassandra 6368 Oct 2 01:15
mc-48278-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra 92 Oct 2 01:15
mc-48278-big-Summary.db
-rw-r--r-- 1 cassandra cassandra 92 Oct 2 01:15 mc-48278-big-TOC.txt
-rw-r--r-- 1 cassandra cassandra 20643 Oct 2 01:16
mc-48279-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 62681705 Oct 2 01:16 mc-48279-big-Data.db
-rw-r--r-- 1 cassandra cassandra 10 Oct 2 01:16
mc-48279-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra 176 Oct 2 01:16
mc-48279-big-Filter.db
-rw-r--r-- 1 cassandra cassandra 162571 Oct 2 01:16
mc-48279-big-Index.db
-rw-r--r-- 1 cassandra cassandra 6375 Oct 2 01:16
mc-48279-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra 92 Oct 2 01:16
mc-48279-big-Summary.db
-rw-r--r-- 1 cassandra cassandra 92 Oct 2 01:16 mc-48279-big-TOC.txt
-rw-r--r-- 1 cassandra cassandra 20099 Oct 2 01:16
mc-48280-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 62587865 Oct 2 01:16 mc-48280-big-Data.db
-rw-r--r-- 1 cassandra cassandra 10 Oct 2 01:16
mc-48280-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra 176 Oct 2 01:16
mc-48280-big-Filter.db
-rw-r--r-- 1 cassandra cassandra 158379 Oct 2 01:16
mc-48280-big-Index.db
-rw-r--r-- 1 cassandra cassandra 6375 Oct 2 01:16
mc-48280-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra 92 Oct 2 01:16
mc-48280-big-Summary.db
-rw-r--r-- 1 cassandra cassandra 92 Oct 2 01:16 mc-48280-big-TOC.txt
-rw-r--r-- 1 cassandra cassandra 51 Oct 2 01:16
mc-48281-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 50 Oct 2 01:16 mc-48281-big-Data.db
-rw-r--r-- 1 cassandra cassandra 10 Oct 2 01:16
mc-48281-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra 176 Oct 2 01:16
mc-48281-big-Filter.db
-rw-r--r-- 1 cassandra cassandra 20 Oct 2 01:16
mc-48281-big-Index.db
-rw-r--r-- 1 cassandra cassandra 4640 Oct 2 01:16
mc-48281-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra 92 Oct 2 01:16
mc-48281-big-Summary.db
-rw-r--r-- 1 cassandra cassandra 92 Oct 2 01:16 mc-48281-big-TOC.txt
Now check both the list results. If they have some common sstables then we
> can say that C* is not compacting sstables.
>
Yes, exactly. How can I fix it?
Bye,
Gábor Auth
Re: Alter table gc_grace_seconds
Posted by Varun Barala <va...@gmail.com>.
+1 Justing,
Might be possible C* is not compacting the sstables [
https://stackoverflow.com/questions/28437301/cassandra-not-compacting-sstables
].
You can this fact by doing below procedure:-
*Run this before compaction:-*
ls /var/lib/cassandra/data/mat/number_item-*/
Store result to some file [just to keep track of sstables]
*Run compaction:-*nodetool compact mat number_item
*List directories again:-*
ls /var/lib/cassandra/data/mat/number_item-*/
Now check both the list results. If they have some common sstables then we
can say that C* is not compacting sstables.
Thanks!!
On Mon, Oct 2, 2017 at 2:32 PM, Justin Cameron <ju...@instaclustr.com>
wrote:
> >> * You should not try on real clusters directly.
> >Why not? :)
>
> It's highly recommended that you complete a full repair before the GC
> grace period expires, otherwise it's possible you could experience zombie
> data (i.e. data that was previously deleted coming back to life)
>
> See http://thelastpickle.com/blog/2016/07/27/about-deletes-
> and-tombstones.html for a good overview of the problem
>
>
> On Mon, 2 Oct 2017 at 16:51 Gábor Auth <au...@gmail.com> wrote:
>
>> Hi,
>>
>> On Mon, Oct 2, 2017 at 5:55 AM Varun Barala <va...@gmail.com>
>> wrote:
>>
>>> *select gc_grace_seconds from system_schema.tables where keyspace_name =
>>> 'keyspace' and table_name = 'number_item;*
>>>
>>
>> cassandra@cqlsh:mat> DESCRIBE TABLE mat.number_item;
>>
>>
>> CREATE TABLE mat.number_item (
>> nodeid uuid,
>> type text,
>> created timeuuid,
>> value float,
>> PRIMARY KEY (nodeid, type, created)
>> ) WITH CLUSTERING ORDER BY (type ASC, created ASC)
>> AND bloom_filter_fp_chance = 0.01
>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
>> AND cdc = false
>> AND comment = ''
>> AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
>> 'max_threshold': '32', 'min_threshold': '4'}
>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>> org.apache.cassandra.io.compress.LZ4Compressor'}
>> AND crc_check_chance = 1.0
>> AND dclocal_read_repair_chance = 0.1
>> AND default_time_to_live = 0
>> AND gc_grace_seconds = 3600
>> AND max_index_interval = 2048
>> AND memtable_flush_period_in_ms = 0
>> AND min_index_interval = 128
>> AND read_repair_chance = 0.0
>> AND speculative_retry = '99PERCENTILE';
>>
>> cassandra@cqlsh:mat> select gc_grace_seconds from system_schema.tables
>> where keyspace_name = 'mat' and table_name = 'number_item';
>>
>> gc_grace_seconds
>> ------------------
>> 3600
>>
>> (1 rows)
>>
>> Bye,
>> Gábor Auth
>>
> --
>
>
> *Justin Cameron*Senior Software Engineer
>
>
> <https://www.instaclustr.com/>
>
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information. If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
Re: Alter table gc_grace_seconds
Posted by Gábor Auth <au...@gmail.com>.
Hi,
On Mon, Oct 2, 2017 at 8:32 AM Justin Cameron <ju...@instaclustr.com>
wrote:
> >> * You should not try on real clusters directly.
> >Why not? :)
>
> It's highly recommended that you complete a full repair before the GC
> grace period expires, otherwise it's possible you could experience zombie
> data (i.e. data that was previously deleted coming back to life)
>
It is a test cluster with test keyspaces. :)
Bye,
Gábor Auth
>
Re: Alter table gc_grace_seconds
Posted by Gábor Auth <au...@gmail.com>.
Hi,
On Mon, Oct 2, 2017 at 1:43 PM Varun Barala <va...@gmail.com> wrote:
> Either you can change min_threshold to three in your case or you can
> change compaction strategy for this table.
>
I've changed:
alter table number_item with compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '2'};
The list of database files of `number_item` on one node:
-rw-r--r-- 1 cassandra cassandra 71353717 Oct 2 11:53 mc-48399-big-Data.db
-rw-r--r-- 1 cassandra cassandra 32843468 Oct 2 19:15 mc-48435-big-Data.db
-rw-r--r-- 1 cassandra cassandra 24734857 Oct 2 19:49 mc-48439-big-Data.db
I've initiated a compaction on the `number_item` CF. After the compaction:
-rw-r--r-- 1 cassandra cassandra 71353717 Oct 2 11:53 mc-48399-big-Data.db
-rw-r--r-- 1 cassandra cassandra 32843468 Oct 2 19:15 mc-48435-big-Data.db
-rw-r--r-- 1 cassandra cassandra 24734857 Oct 2 19:53 mc-48440-big-Data.db
Two of them untouched and one rewritten with the same content. :/
Bye,
Gábor Auth
Re: Alter table gc_grace_seconds
Posted by Varun Barala <va...@gmail.com>.
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsConfigureCompaction.html
As pe STCS:
*This strategy triggers a minor compaction when there are a number of
similar sized SSTables on disk as configured by the table subproperty,
min_threshold.*
In your case you have *min_threshold=4 and 3 Big sstable and one small
sstable.*
Either you can change min_threshold to three in your case or you can change
compaction strategy for this table.
Thanks!!
On Mon, Oct 2, 2017 at 4:48 PM, Steinmaurer, Thomas <
thomas.steinmaurer@dynatrace.com> wrote:
> Hello Justin,
>
>
>
> yes, but in real-world, hard to accomplish for high volume column families
> >= 3-digit GB. Even with the default of 10 days grace period, this may get
> a real challenge, to complete a full repair. J
>
>
>
> Possibly back again to the discussion that incremental repair has some
> flaws, full repair (-full option) in 3.0+ (2.2+?) isn’t behaving the same
> way as in 2.1 anymore, due to troubles when kicking off with –pr on several
> nodes at once.
>
>
>
> Regards,
>
> Thomas
>
>
>
> *From:* Justin Cameron [mailto:justin@instaclustr.com]
> *Sent:* Montag, 02. Oktober 2017 08:32
> *To:* user@cassandra.apache.org
> *Subject:* Re: Alter table gc_grace_seconds
>
>
>
> >> * You should not try on real clusters directly.
>
> >Why not? :)
>
> It's highly recommended that you complete a full repair before the GC
> grace period expires, otherwise it's possible you could experience zombie
> data (i.e. data that was previously deleted coming back to life)
>
> See http://thelastpickle.com/blog/2016/07/27/about-deletes-
> and-tombstones.html for a good overview of the problem
>
>
>
>
>
> On Mon, 2 Oct 2017 at 16:51 Gábor Auth <au...@gmail.com> wrote:
>
> Hi,
>
> On Mon, Oct 2, 2017 at 5:55 AM Varun Barala <va...@gmail.com>
> wrote:
>
> *select gc_grace_seconds from system_schema.tables where keyspace_name =
> 'keyspace' and table_name = 'number_item;*
>
>
>
> cassandra@cqlsh:mat> DESCRIBE TABLE mat.number_item;
>
>
> CREATE TABLE mat.number_item (
> nodeid uuid,
> type text,
> created timeuuid,
> value float,
> PRIMARY KEY (nodeid, type, created)
> ) WITH CLUSTERING ORDER BY (type ASC, created ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND cdc = false
> AND comment = ''
> AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class': '
> org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 3600
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
>
> cassandra@cqlsh:mat> select gc_grace_seconds from system_schema.tables
> where keyspace_name = 'mat' and table_name = 'number_item';
>
> *gc_grace_seconds*
> ------------------
> *3600*
>
> (1 rows)
>
>
> Bye,
> Gábor Auth
>
> --
>
> *Justin Cameron*
> Senior Software Engineer
>
>
>
> [image: Image removed by sender.] <https://www.instaclustr.com/>
>
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information. If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
> The contents of this e-mail are intended for the named addressee only. It
> contains information that may be confidential. Unless you are the named
> addressee or an authorized designee, you may not copy or use it, or
> disclose it to anyone else. If you received it in error please notify us
> immediately and then destroy it. Dynatrace Austria GmbH (registration
> number FN 91482h) is a company registered in Linz whose registered office
> is at 4040 Linz, Austria, Freistädterstraße 313
> <https://maps.google.com/?q=4040+Linz,+Austria,+Freist%C3%A4dterstra%C3%9Fe+313&entry=gmail&source=g>
>
RE: Alter table gc_grace_seconds
Posted by "Steinmaurer, Thomas" <th...@dynatrace.com>.
Hello Justin,
yes, but in real-world, hard to accomplish for high volume column families >= 3-digit GB. Even with the default of 10 days grace period, this may get a real challenge, to complete a full repair. ☺
Possibly back again to the discussion that incremental repair has some flaws, full repair (-full option) in 3.0+ (2.2+?) isn’t behaving the same way as in 2.1 anymore, due to troubles when kicking off with –pr on several nodes at once.
Regards,
Thomas
From: Justin Cameron [mailto:justin@instaclustr.com]
Sent: Montag, 02. Oktober 2017 08:32
To: user@cassandra.apache.org
Subject: Re: Alter table gc_grace_seconds
>> * You should not try on real clusters directly.
>Why not? :)
It's highly recommended that you complete a full repair before the GC grace period expires, otherwise it's possible you could experience zombie data (i.e. data that was previously deleted coming back to life)
See http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html for a good overview of the problem
On Mon, 2 Oct 2017 at 16:51 Gábor Auth <au...@gmail.com>> wrote:
Hi,
On Mon, Oct 2, 2017 at 5:55 AM Varun Barala <va...@gmail.com>> wrote:
select gc_grace_seconds from system_schema.tables where keyspace_name = 'keyspace' and table_name = 'number_item;
cassandra@cqlsh:mat> DESCRIBE TABLE mat.number_item;
CREATE TABLE mat.number_item (
nodeid uuid,
type text,
created timeuuid,
value float,
PRIMARY KEY (nodeid, type, created)
) WITH CLUSTERING ORDER BY (type ASC, created ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND cdc = false
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 3600
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
cassandra@cqlsh:mat> select gc_grace_seconds from system_schema.tables where keyspace_name = 'mat' and table_name = 'number_item';
gc_grace_seconds
------------------
3600
(1 rows)
Bye,
Gábor Auth
--
Justin Cameron
Senior Software Engineer
[Image removed by sender.]<https://www.instaclustr.com/>
This email has been sent on behalf of Instaclustr Pty. Limited (Australia) and Instaclustr Inc (USA).
This email and any attachments may contain confidential and legally privileged information. If you are not the intended recipient, do not copy or disclose its content, but please reply to this email immediately and highlight the error to the sender and then immediately delete the message.
The contents of this e-mail are intended for the named addressee only. It contains information that may be confidential. Unless you are the named addressee or an authorized designee, you may not copy or use it, or disclose it to anyone else. If you received it in error please notify us immediately and then destroy it. Dynatrace Austria GmbH (registration number FN 91482h) is a company registered in Linz whose registered office is at 4040 Linz, Austria, Freistädterstraße 313
Re: Alter table gc_grace_seconds
Posted by Justin Cameron <ju...@instaclustr.com>.
>> * You should not try on real clusters directly.
>Why not? :)
It's highly recommended that you complete a full repair before the GC grace
period expires, otherwise it's possible you could experience zombie data
(i.e. data that was previously deleted coming back to life)
See
http://thelastpickle.com/blog/2016/07/27/about-deletes-and-tombstones.html for
a good overview of the problem
On Mon, 2 Oct 2017 at 16:51 Gábor Auth <au...@gmail.com> wrote:
> Hi,
>
> On Mon, Oct 2, 2017 at 5:55 AM Varun Barala <va...@gmail.com>
> wrote:
>
>> *select gc_grace_seconds from system_schema.tables where keyspace_name =
>> 'keyspace' and table_name = 'number_item;*
>>
>
> cassandra@cqlsh:mat> DESCRIBE TABLE mat.number_item;
>
>
> CREATE TABLE mat.number_item (
> nodeid uuid,
> type text,
> created timeuuid,
> value float,
> PRIMARY KEY (nodeid, type, created)
> ) WITH CLUSTERING ORDER BY (type ASC, created ASC)
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND cdc = false
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '64', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.1
> AND default_time_to_live = 0
> AND gc_grace_seconds = 3600
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99PERCENTILE';
>
> cassandra@cqlsh:mat> select gc_grace_seconds from system_schema.tables
> where keyspace_name = 'mat' and table_name = 'number_item';
>
> gc_grace_seconds
> ------------------
> 3600
>
> (1 rows)
>
> Bye,
> Gábor Auth
>
--
*Justin Cameron*Senior Software Engineer
<https://www.instaclustr.com/>
This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).
This email and any attachments may contain confidential and legally
privileged information. If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.
Re: Alter table gc_grace_seconds
Posted by Gábor Auth <au...@gmail.com>.
Hi,
On Mon, Oct 2, 2017 at 5:55 AM Varun Barala <va...@gmail.com> wrote:
> *select gc_grace_seconds from system_schema.tables where keyspace_name =
> 'keyspace' and table_name = 'number_item;*
>
cassandra@cqlsh:mat> DESCRIBE TABLE mat.number_item;
CREATE TABLE mat.number_item (
nodeid uuid,
type text,
created timeuuid,
value float,
PRIMARY KEY (nodeid, type, created)
) WITH CLUSTERING ORDER BY (type ASC, created ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND cdc = false
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 3600
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
cassandra@cqlsh:mat> select gc_grace_seconds from system_schema.tables
where keyspace_name = 'mat' and table_name = 'number_item';
gc_grace_seconds
------------------
3600
(1 rows)
Bye,
Gábor Auth
Re: Alter table gc_grace_seconds
Posted by Varun Barala <va...@gmail.com>.
Can you share result of query:-
*select gc_grace_seconds from system_schema.tables where keyspace_name =
'keyspace' and table_name = 'number_item;*
Thanks!!
On Mon, Oct 2, 2017 at 3:42 AM, Gábor Auth <au...@gmail.com> wrote:
> Hi,
>
> On Sun, Oct 1, 2017 at 9:36 PM Varun Barala <va...@gmail.com>
> wrote:
>
>> * You should not try on real clusters directly.
>>
>
> Why not? :)
>
> Did you change gc_grace for all column families?
>>
>
> Not, only on the `number_item` CF.
>
> > But not in the `number_item` CF... :(
>> Could you please explain?
>>
>
> I've tried the test case that you described and it is works (the compact
> removed the marked_deleted rows) on a newly created CF. But the same
> gc_grace_seconds settings has no effect in the `number_item` CF (millions
> of rows has been deleted during a last week migration).
>
> Bye,
> Gábor Auth
>
>
Re: Alter table gc_grace_seconds
Posted by Gábor Auth <au...@gmail.com>.
Hi,
On Sun, Oct 1, 2017 at 9:36 PM Varun Barala <va...@gmail.com> wrote:
> * You should not try on real clusters directly.
>
Why not? :)
Did you change gc_grace for all column families?
>
Not, only on the `number_item` CF.
> But not in the `number_item` CF... :(
> Could you please explain?
>
I've tried the test case that you described and it is works (the compact
removed the marked_deleted rows) on a newly created CF. But the same
gc_grace_seconds settings has no effect in the `number_item` CF (millions
of rows has been deleted during a last week migration).
Bye,
Gábor Auth
Re: Alter table gc_grace_seconds
Posted by Varun Barala <va...@gmail.com>.
* You should not try on real clusters directly.
Did you change gc_grace for all column families?
> But not in the `number_item` CF... :(
Could you please explain?
Thanks!!
On Mon, Oct 2, 2017 at 2:24 AM, Gábor Auth <au...@gmail.com> wrote:
> Hi,
>
> On Sun, Oct 1, 2017 at 7:44 PM Varun Barala <va...@gmail.com>
> wrote:
>
>> Sorry If I misunderstood the situation.
>>
>
> Ok, I'm confused... :/
>
> I've just tested it on the same cluster and the compact removed the
> marked_deleted rows. But not in the `number_item` CF... :(
>
> Cassandra 3.11.0, two DC (with 4-4 nodes).
>
> Bye,
> Gábor Auth
>
Re: Alter table gc_grace_seconds
Posted by Gábor Auth <au...@gmail.com>.
Hi,
On Sun, Oct 1, 2017 at 7:44 PM Varun Barala <va...@gmail.com> wrote:
> Sorry If I misunderstood the situation.
>
Ok, I'm confused... :/
I've just tested it on the same cluster and the compact removed the
marked_deleted rows. But not in the `number_item` CF... :(
Cassandra 3.11.0, two DC (with 4-4 nodes).
Bye,
Gábor Auth
Re: Alter table gc_grace_seconds
Posted by Varun Barala <va...@gmail.com>.
Sorry If I misunderstood the situation.
{
"type" : "row",
"position" : 146160,
"clustering" : [ "humidity", "97781fd0-9dab-11e7-a3d5-7f6ef9a844c7" ],
"deletion_info" : { "marked_deleted" : "2017-09-25T11:51:19.165276Z",
"local_delete_time" : "2017-09-25T11:51:19Z" },
"cells" : [ ]
}
How did we come to know that TTL is applied by looking at this data?
I performed a basic test in my local:-
*# C* version* 3.0.14,
*# Table schema:-*
CREATE TABLE test.table1 (
id int,
ck1 text,
ck2 text,
nk1 text,
PRIMARY KEY (id,ck1, ck2)
);
*# insert statement:-*
insert into test.table1 (id, ck1, ck2, nk1) VALUES (1,'1','1','1');
*# performed flush to generate the sstable(json format):-*
{
"partition" : {
"key" : [ "1" ],
"position" : 0
},
"rows" : [
{
"type" : "row",
"position" : 30,
"clustering" : [ "1", "1" ],
"liveness_info" : { "tstamp" : "2017-10-01T17:28:56.889Z" },
"cells" : [
{ "name" : "nk1", "value" : "1" }
]
}
]
}
*# delete statement:-*
delete from test.table1 where id=1 and ck1='1' and ck2='1';
*# performed flush to generate new sstable(json format):-*
{
"partition" : {
"key" : [ "1" ],
"position" : 0
},
"rows" : [
{
"type" : "row",
"position" : 32,
"clustering" : [ "1", "1" ],
"deletion_info" : { "marked_deleted" : "2017-10-01T17:30:15.397Z",
"local_delete_time" : "2017-10-01T17:30:15Z" },
"cells" : [ ]
}
]
}
*# performed compaction to compact above two sstables (latest won,
tombstone has not been purged):-* {
"partition" : {
"key" : [ "1" ],
"position" : 0
},
"rows" : [
{
"type" : "row",
"position" : 32,
"clustering" : [ "1", "1" ],
"deletion_info" : { "marked_deleted" : "2017-10-01T17:30:15.397Z",
"local_delete_time" : "2017-10-01T17:30:15Z" },
"cells" : [ ]
}
]
}
*# changed gc_grace:-*
alter TABLE test.table1 WITH gc_grace_seconds = 0;
*# performed compaction again(no sstable exists, since it has purged the
tombstones older than gc_grace):-*
no sstable exist in data dir
Please let me know If I'm missing some fact.
Thanks!!
On Mon, Oct 2, 2017 at 1:11 AM, Gábor Auth <au...@gmail.com> wrote:
> Hi,
>
> On Sun, Oct 1, 2017 at 6:53 PM Jonathan Haddad <jo...@jonhaddad.com> wrote:
>
>> The TTL is applied to the cells on insert. Changing it doesn't change the
>> TTL on data that was inserted previously.
>>
>
> Is there any way to purge out these tombstoned data?
>
> Bye,
> Gábor Auth
>
Re: Alter table gc_grace_seconds
Posted by Gábor Auth <au...@gmail.com>.
Hi,
On Sun, Oct 1, 2017 at 6:53 PM Jonathan Haddad <jo...@jonhaddad.com> wrote:
> The TTL is applied to the cells on insert. Changing it doesn't change the
> TTL on data that was inserted previously.
>
Is there any way to purge out these tombstoned data?
Bye,
Gábor Auth
Re: Alter table gc_grace_seconds
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
The TTL is applied to the cells on insert. Changing it doesn't change the
TTL on data that was inserted previously.
On Sun, Oct 1, 2017 at 6:23 AM Gábor Auth <au...@gmail.com> wrote:
> Hi,
>
> The `alter table number_item with gc_grace_seconds = 3600;` is sets the
> grace seconds of tombstones of the future modification of number_item
> column family or affects all existing data?
>
> Bye,
> Gábor Auth
>
>
Re: Alter table gc_grace_seconds
Posted by Varun Barala <va...@gmail.com>.
* Which C* version are you using?
* How many nodes are there in this cluster?
These tombstones will not be deleted if they are not older than
gc_grace_seconds.
On Sun, Oct 1, 2017 at 10:14 PM, Gábor Auth <au...@gmail.com> wrote:
> Hi,
>
> On Sun, Oct 1, 2017 at 3:44 PM Varun Barala <va...@gmail.com>
> wrote:
>
>> This is the property of table and It's not written in sstables. If you
>> change gc_grace, It'll get applied for all the data.
>>
>
> Hm... I've migrated lot of data from `number_item` to `measurement` CF
> because of scheme upgrade. During the migration, the script created rows in
> the `measurement` CF and deleted the migrated rows in the `number_item` CF,
> one-by-one.
>
> I've just take a look on the sstables of `number_item` and it is full of
> deleted rows:
> {
> "type" : "row",
> "position" : 146160,
> "clustering" : [ "humidity", "97781fd0-9dab-11e7-a3d5-7f6ef9a844c7" ],
> "deletion_info" : { "marked_deleted" : "2017-09-25T11:51:19.165276Z",
> "local_delete_time" : "2017-09-25T11:51:19Z" },
> "cells" : [ ]
> }
>
> How can I purge these old rows? :)
>
> I've tried: compact, scrub, cleanup, clearsnapshot, flush and full repair.
>
> Bye,
> Gábor Auth
>
>
Re: Alter table gc_grace_seconds
Posted by Gábor Auth <au...@gmail.com>.
Hi,
On Wed, Oct 4, 2017 at 8:39 AM Oleksandr Shulgin <
oleksandr.shulgin@zalando.de> wrote:
> If you have migrated ALL the data from the old CF, you could just use
> TRUNCATE or DROP TABLE, followed by "nodetool clearsnapshot" to reclaim the
> disk space (this step has to be done per-node).
>
Unfortunately not all the data migrated, but to CFs came from one CF:
"only" the 80% for data migrated to another CF.
At the moment, the compaction works with:
compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '2'} AND gc_grace_seconds = 172800
But I don't know why it did not work and why it works now... :)
Bye,
Auth Gábor
Re: Alter table gc_grace_seconds
Posted by Oleksandr Shulgin <ol...@zalando.de>.
On Sun, Oct 1, 2017 at 4:14 PM, Gábor Auth <au...@gmail.com> wrote:
> Hi,
>
> On Sun, Oct 1, 2017 at 3:44 PM Varun Barala <va...@gmail.com>
> wrote:
>
>> This is the property of table and It's not written in sstables. If you
>> change gc_grace, It'll get applied for all the data.
>>
>
> Hm... I've migrated lot of data from `number_item` to `measurement` CF
> because of scheme upgrade. During the migration, the script created rows in
> the `measurement` CF and deleted the migrated rows in the `number_item` CF,
> one-by-one.
>
> I've just take a look on the sstables of `number_item` and it is full of
> deleted rows:
> {
> "type" : "row",
> "position" : 146160,
> "clustering" : [ "humidity", "97781fd0-9dab-11e7-a3d5-7f6ef9a844c7" ],
> "deletion_info" : { "marked_deleted" : "2017-09-25T11:51:19.165276Z",
> "local_delete_time" : "2017-09-25T11:51:19Z" },
> "cells" : [ ]
> }
>
> How can I purge these old rows? :)
>
> I've tried: compact, scrub, cleanup, clearsnapshot, flush and full repair.
>
If you have migrated ALL the data from the old CF, you could just use
TRUNCATE or DROP TABLE, followed by "nodetool clearsnapshot" to reclaim the
disk space (this step has to be done per-node).
--
Alex
Re: Alter table gc_grace_seconds
Posted by Gábor Auth <au...@gmail.com>.
Hi,
On Sun, Oct 1, 2017 at 3:44 PM Varun Barala <va...@gmail.com> wrote:
> This is the property of table and It's not written in sstables. If you
> change gc_grace, It'll get applied for all the data.
>
Hm... I've migrated lot of data from `number_item` to `measurement` CF
because of scheme upgrade. During the migration, the script created rows in
the `measurement` CF and deleted the migrated rows in the `number_item` CF,
one-by-one.
I've just take a look on the sstables of `number_item` and it is full of
deleted rows:
{
"type" : "row",
"position" : 146160,
"clustering" : [ "humidity", "97781fd0-9dab-11e7-a3d5-7f6ef9a844c7" ],
"deletion_info" : { "marked_deleted" : "2017-09-25T11:51:19.165276Z",
"local_delete_time" : "2017-09-25T11:51:19Z" },
"cells" : [ ]
}
How can I purge these old rows? :)
I've tried: compact, scrub, cleanup, clearsnapshot, flush and full repair.
Bye,
Gábor Auth
Re: Alter table gc_grace_seconds
Posted by Varun Barala <va...@gmail.com>.
Hi,
This is the property of table and It's not written in sstables. If you
change gc_grace, It'll get applied for all the data. Thanks!!
C* stores this info inside schema_columnfamilies
Regards,
Varun Barala
On Sun, Oct 1, 2017 at 9:23 PM, Gábor Auth <au...@gmail.com> wrote:
> Hi,
>
> The `alter table number_item with gc_grace_seconds = 3600;` is sets the
> grace seconds of tombstones of the future modification of number_item
> column family or affects all existing data?
>
> Bye,
> Gábor Auth
>
>