You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Eric Wong <Er...@solvians.com> on 2021/06/29 09:57:49 UTC
TWCS repair and compact help
Hi:
We need some help on cassandra repair and compact for a table that uses TWCS. We are running cassandra 4.0-rc1. A database called test_db, biggest table "minute_rate", storing time-series data. It has the following configuration:
CREATE TABLE test_db.minute_rate (
market smallint,
sin bigint,
field smallint,
slot timestamp,
close frozen<pricerecord>,
high frozen<pricerecord>,
low frozen<pricerecord>,
open frozen<pricerecord>,
PRIMARY KEY ((market, sin, field), slot)
) WITH CLUSTERING ORDER BY (slot ASC)
AND additional_write_policy = '99p'
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND cdc = false
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy', 'compaction_window_size': '4', 'compaction_window_unit': 'HOURS', 'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND default_time_to_live = 604800
AND extensions = {}
AND gc_grace_seconds = 86400
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair = 'BLOCKING'
AND speculative_retry = '99p';
minute_rate is configured to use TWCS. We can see that it creates about 8 data files per day. Each data file is relatively small size around 2GB each. so far so good.
However, when "nodetool compact test_db minute_rate" ran on the weekend, it seems to consolidate all small data files into one big files. nodetool compact after 2 or 3 weeks we ended up having bigger and bigger data files. eating up all disk space on the machine.
From what I understand about TWCS, cassandra will simply drop the data files when the records inside the data file is older than default_time_to_live of 604800 (7 days). But somehow this is not what we are seeing. When I sstablemetadata the oldest data file (sample below), I can see tombstone drop time all got updated. Resulting in the data file never get removed.
This lead me to think I am configuring things the wrong way. So I want to know when using TWCS, do we need to repair and compact?
I saw in cassandra-reaper (we use reaper for repair) it is configured to skip TWCS.
Should I stop running "nodetool compact test_db minute_rate"? If without "nodetool compact", will cassandra clean up the tombstoned data file?
Thanks,
Eric
# sstablemetadata na-6681303-big-Data.db
SSTable: /var/lib/cassandra/data/test_db/minute_rate-d7955270f31d11ea88fabb8dcc37b800/na-6681303-big
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.01
Minimum timestamp: 1622528285761740 (06/01/2021 06:18:05)
Maximum timestamp: 1624699757821614 (06/26/2021 09:29:17)
SSTable min local deletion time: 1624767191 (06/27/2021 04:13:11)
SSTable max local deletion time: 2147483647 (no tombstones)
Compressor: org.apache.cassandra.io.compress.LZ4Compressor
Compression ratio: 0.2753232939720765
TTL min: 0
TTL max: 604800 (7 days)
First token: -9223371506914883753 (3870:16432974:7)
Last token: 9223358359954725918 (505:23187788:6)
minClusteringValues: [2021-04-12T16:54:00.000Z]
maxClusteringValues: [2021-06-26T09:28:00.000Z]
Estimated droppable tombstones: 0.3683520383378445
SSTable Level: 0
Repaired at: 0
Pending repair: --
Replay positions covered: {}
totalColumnsSet: 17179881008
totalRows: 4413102532
Estimated tombstone drop times:
Drop Time | Count (%) Histogram
1624768080 (06/27/2021 04:28:00) | 2 ( 0)
1624771440 (06/27/2021 05:24:00) | 98 ( 0)
1624777020 (06/27/2021 06:57:00) | 85 ( 0)
1624781640 (06/27/2021 08:14:00) | 74 ( 0)
1624786080 (06/27/2021 09:28:00) | 74 ( 0)
1624790280 (06/27/2021 10:38:00) | 66 ( 0)
1624794900 (06/27/2021 11:55:00) | 87 ( 0)
1624800060 (06/27/2021 13:21:00) | 83 ( 0)
1624804680 (06/27/2021 14:38:00) | 108064 ( 0)
1624809180 (06/27/2021 15:53:00) | 304148 ( 0)
1624812540 (06/27/2021 16:49:00) | 133188 ( 0)
1624819440 (06/27/2021 18:44:00) | 88 ( 0)
1624824060 (06/27/2021 20:01:00) | 73 ( 0)
1624828080 (06/27/2021 21:08:00) | 66 ( 0)
1624832520 (06/27/2021 22:22:00) | 1 ( 0)
1624835880 (06/27/2021 23:18:00) | 23578916 ( 0) o
1624839720 (06/28/2021 00:22:00) | 21783899 ( 0) o
1624843740 (06/28/2021 01:29:00) | 22758204 ( 0) o
1624848120 (06/28/2021 02:42:00) | 25237306 ( 0) o
1624853520 (06/28/2021 04:12:00) | 44003185 ( 0) O.
1624858080 (06/28/2021 05:28:00) | 145977595 ( 0) OOOOO
1624862460 (06/28/2021 06:41:00) | 331875915 ( 1) OOOOOOOOOOOo
1624866060 (06/28/2021 07:41:00) | 463284230 ( 2) OOOOOOOOOOOOOOOO.
1624869540 (06/28/2021 08:39:00) | 455732185 ( 2) OOOOOOOOOOOOOOOO
1624873140 (06/28/2021 09:39:00) | 492533095 ( 2) OOOOOOOOOOOOOOOOO.
1624876620 (06/28/2021 10:37:00) | 481999210 ( 2) OOOOOOOOOOOOOOOOO
1624880280 (06/28/2021 11:38:00) | 498443030 ( 2) OOOOOOOOOOOOOOOOOo
1624883940 (06/28/2021 12:39:00) | 520404180 ( 2) OOOOOOOOOOOOOOOOOO.
1624887840 (06/28/2021 13:44:00) | 581806225 ( 2) OOOOOOOOOOOOOOOOOOOO.
1624891680 (06/28/2021 14:48:00) | 581503090 ( 2) OOOOOOOOOOOOOOOOOOOO.
1624895220 (06/28/2021 15:47:00) | 465873220 ( 2) OOOOOOOOOOOOOOOO.
1624898820 (06/28/2021 16:47:00) | 406239880 ( 1) OOOOOOOOOOOOOO.
1624902600 (06/28/2021 17:50:00) | 429656505 ( 1) OOOOOOOOOOOOOOO.
1624906500 (06/28/2021 18:55:00) | 410067155 ( 1) OOOOOOOOOOOOOO.
1624910460 (06/28/2021 20:01:00) | 365053925 ( 1) OOOOOOOOOOOOO
1624914480 (06/28/2021 21:08:00) | 58380400 ( 0) OO
1624919700 (06/28/2021 22:35:00) | 23698865 ( 0) o
1624956900 (06/29/2021 08:55:00) | 3894845 ( 0)
1624960380 (06/29/2021 09:53:00) | 842061700 ( 3) OOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
1624963860 (06/29/2021 10:51:00) | 1423860 ( 0)
1624967820 (06/29/2021 11:57:00) | 698030 ( 0)
1624971900 (06/29/2021 13:05:00) | 240193520 ( 1) OOOOOOOO.
1624980000 (06/29/2021 15:20:00) | 78693575 ( 0) OOo
1624984680 (06/29/2021 16:38:00) | 8843580 ( 0) .
1624994220 (06/29/2021 19:17:00) | 101922005 ( 0) OOO.
1624999860 (06/29/2021 20:51:00) | 89190335 ( 0) OOO
1625005380 (06/29/2021 22:23:00) | 14607250 ( 0) .
1625021760 (06/30/2021 02:56:00) | 390529310 ( 1) OOOOOOOOOOOOOo
1625026020 (06/30/2021 04:07:00) | 26338835 ( 0) o
1625030760 (06/30/2021 05:26:00) | 17315030 ( 0) .
1625034180 (06/30/2021 06:23:00) | 17466955 ( 0) .
1625038260 (06/30/2021 07:31:00) | 476323610 ( 2) OOOOOOOOOOOOOOOOo
1625043240 (06/30/2021 08:54:00) | 360663480 ( 1) OOOOOOOOOOOOo
1625047020 (06/30/2021 09:57:00) | 162371390 ( 0) OOOOOo
1625053380 (06/30/2021 11:43:00) | 131161180 ( 0) OOOO.
1625058780 (06/30/2021 13:13:00) | 8394630 ( 0)
1625062260 (06/30/2021 14:11:00) | 105939220 ( 0) OOOo
1625065680 (06/30/2021 15:08:00) | 76280800 ( 0) OO.
1625075940 (06/30/2021 17:59:00) | 501265 ( 0)
1625135880 (07/01/2021 10:38:00) | 761088820 ( 3) OOOOOOOOOOOOOOOOOOOOOOOOOOO
1625140140 (07/01/2021 11:49:00) | 511159205 ( 2) OOOOOOOOOOOOOOOOOO
1625143980 (07/01/2021 12:53:00) | 492520020 ( 2) OOOOOOOOOOOOOOOOO.
1625147580 (07/01/2021 13:53:00) | 520719095 ( 2) OOOOOOOOOOOOOOOOOO.
1625151180 (07/01/2021 14:53:00) | 504806855 ( 2) OOOOOOOOOOOOOOOOOo
1625154720 (07/01/2021 15:52:00) | 446265015 ( 2) OOOOOOOOOOOOOOOo
1625158380 (07/01/2021 16:53:00) | 408591530 ( 1) OOOOOOOOOOOOOO.
1625161980 (07/01/2021 17:53:00) | 407185215 ( 1) OOOOOOOOOOOOOO.
1625165700 (07/01/2021 18:55:00) | 387552375 ( 1) OOOOOOOOOOOOOo
1625169840 (07/01/2021 20:04:00) | 352752965 ( 1) OOOOOOOOOOOO.
1625173800 (07/01/2021 21:10:00) | 61856610 ( 0) OO
1625178600 (07/01/2021 22:30:00) | 660 ( 0)
1625187720 (07/02/2021 01:02:00) | 127432425 ( 0) OOOO.
1625192100 (07/02/2021 02:15:00) | 110082460 ( 0) OOOo
1625196180 (07/02/2021 03:23:00) | 99408705 ( 0) OOO.
1625200080 (07/02/2021 04:28:00) | 96594150 ( 0) OOO.
1625204280 (07/02/2021 05:38:00) | 122591330 ( 0) OOOO.
1625208120 (07/02/2021 06:42:00) | 304278735 ( 1) OOOOOOOOOOo
1625211660 (07/02/2021 07:41:00) | 480851410 ( 2) OOOOOOOOOOOOOOOOO
1625215140 (07/02/2021 08:39:00) | 477311285 ( 2) OOOOOOOOOOOOOOOOO
1625218680 (07/02/2021 09:38:00) | 454566725 ( 2) OOOOOOOOOOOOOOOO
1625222400 (07/02/2021 10:40:00) | 462280685 ( 2) OOOOOOOOOOOOOOOO.
1625225940 (07/02/2021 11:39:00) | 448421865 ( 2) OOOOOOOOOOOOOOOo
1625229780 (07/02/2021 12:43:00) | 492199905 ( 2) OOOOOOOOOOOOOOOOO.
1625233680 (07/02/2021 13:48:00) | 545296705 ( 2) OOOOOOOOOOOOOOOOOOO.
1625237520 (07/02/2021 14:52:00) | 542032490 ( 2) OOOOOOOOOOOOOOOOOOO.
1625241120 (07/02/2021 15:52:00) | 439191210 ( 2) OOOOOOOOOOOOOOO.
1625244720 (07/02/2021 16:52:00) | 384025420 ( 1) OOOOOOOOOOOOO.
1625248380 (07/02/2021 17:53:00) | 386264375 ( 1) OOOOOOOOOOOOOo
1625252280 (07/02/2021 18:58:00) | 378855715 ( 1) OOOOOOOOOOOOO.
1625256660 (07/02/2021 20:11:00) | 380212785 ( 1) OOOOOOOOOOOOO.
1625261280 (07/02/2021 21:28:00) | 12215 ( 0)
1625266380 (07/02/2021 22:53:00) | 455 ( 0)
1625271120 (07/03/2021 00:12:00) | 365 ( 0)
1625275200 (07/03/2021 01:20:00) | 330 ( 0)
1625279100 (07/03/2021 02:25:00) | 325 ( 0)
1625283360 (07/03/2021 03:36:00) | 355 ( 0)
1625289180 (07/03/2021 05:13:00) | 2080 ( 0)
1625294400 (07/03/2021 06:40:00) | 5 ( 0)
1625298000 (07/03/2021 07:40:00) | 697840 ( 0)
1625303040 (07/03/2021 09:04:00) | 621800 ( 0)
Percentiles
50th 1663415872 (09/17/2022 11:57:52)
75th 1663415872 (09/17/2022 11:57:52)
95th 1663415872 (09/17/2022 11:57:52)
98th 1663415872 (09/17/2022 11:57:52)
99th 1663415872 (09/17/2022 11:57:52)
Min 1386179894 (12/04/2013 17:58:14)
Max 1663415872 (09/17/2022 11:57:52)
Partition Size:
Size (bytes) | Count (%) Histogram
72 (72 B) | 3 ( 0)
103 (103 B) | 10 ( 0)
215 (215 B) | 4047 ( 0)
258 (258 B) | 4 ( 0)
372 (372 B) | 2803 ( 0)
535 (535 B) | 2645 ( 0)
642 (642 B) | 6 ( 0)
770 (770 B) | 4157 ( 0)
924 (924 B) | 3935 ( 0)
1109 (1.1 kB) | 11326 ( 0) .
1331 (1.3 kB) | 15721 ( 0) o
1597 (1.6 kB) | 3963 ( 0)
1916 (1.9 kB) | 4821 ( 0)
2299 (2.2 kB) | 1526 ( 0)
2759 (2.7 kB) | 2563 ( 0)
3311 (3.2 kB) | 3520 ( 0)
3973 (3.9 kB) | 2920 ( 0)
4768 (4.7 kB) | 5226 ( 0) .
5722 (5.6 kB) | 7591 ( 0) .
6866 (6.7 kB) | 9223 ( 0) .
8239 (8.0 kB) | 8735 ( 0) .
9887 (9.7 kB) | 8812 ( 0) .
11864 (11.6 kB) | 10849 ( 0) .
14237 (13.9 kB) | 12796 ( 0) o
17084 (16.7 kB) | 18246 ( 0) O
20501 (20.0 kB) | 23489 ( 0) O.
24601 (24.0 kB) | 31568 ( 1) Oo
29521 (28.8 kB) | 37923 ( 1) OO
35425 (34.6 kB) | 37482 ( 1) OO
42510 (41.5 kB) | 45582 ( 1) OO.
51012 (49.8 kB) | 51966 ( 1) OOO
61214 (59.8 kB) | 61596 ( 2) OOO.
73457 (71.7 kB) | 76006 ( 2) OOOO.
88148 (86.1 kB) | 85373 ( 3) OOOOO
105778 (103.3 kB) | 95825 ( 3) OOOOOo
126934 (124.0 kB) | 102830 ( 3) OOOOOO
152321 (148.8 kB) | 113760 ( 4) OOOOOOo
182785 (178.5 kB) | 133287 ( 4) OOOOOOOO
219342 (214.2 kB) | 162030 ( 5) OOOOOOOOO.
263210 (257.0 kB) | 255776 ( 9) OOOOOOOOOOOOOOO.
315852 (308.4 kB) | 267785 ( 9) OOOOOOOOOOOOOOOO
379022 (370.1 kB) | 326228 ( 11) OOOOOOOOOOOOOOOOOOO.
454826 (444.2 kB) | 499363 ( 17) OOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
545791 (533.0 kB) | 154907 ( 5) OOOOOOOOO.
654949 (639.6 kB) | 79305 ( 2) OOOOo
785939 (767.5 kB) | 16 ( 0)
Percentiles
50th 263210 (257.0 kB)
75th 454826 (444.2 kB)
95th 545791 (533.0 kB)
98th 654949 (639.6 kB)
99th 654949 (639.6 kB)
Min 61 (61 B)
Max 785939 (767.5 kB)
Column Count:
Columns | Count (%) Histogram
1 | 1575 ( 0)
4 | 4051 ( 0)
8 | 2805 ( 0)
12 | 2667 ( 0)
17 | 4151 ( 0)
20 | 3915 ( 0)
24 | 11321 ( 0) o
29 | 9564 ( 0) .
35 | 6160 ( 0) .
42 | 6606 ( 0) .
50 | 2437 ( 0)
60 | 2046 ( 0)
72 | 3431 ( 0)
86 | 3468 ( 0)
103 | 4385 ( 0) .
124 | 7725 ( 0) .
149 | 6543 ( 0) .
179 | 6714 ( 0) .
215 | 7848 ( 0) .
258 | 9551 ( 0) .
310 | 12224 ( 0) o
372 | 18113 ( 0) O.
446 | 22412 ( 0) O.
535 | 25057 ( 0) Oo
642 | 30530 ( 1) OO
770 | 33483 ( 1) OO.
924 | 41045 ( 1) OOo
1109 | 46979 ( 1) OOO.
1331 | 54195 ( 1) OOOo
1597 | 68071 ( 2) OOOOo
1916 | 77555 ( 2) OOOOO.
2299 | 87523 ( 3) OOOOOO.
2759 | 98115 ( 3) OOOOOOO
3311 | 105961 ( 3) OOOOOOO.
3973 | 119527 ( 4) OOOOOOOO.
4768 | 137830 ( 4) OOOOOOOOOo
5722 | 193041 ( 6) OOOOOOOOOOOOOo
6866 | 262273 ( 9) OOOOOOOOOOOOOOOOOOo
8239 | 267143 ( 9) OOOOOOOOOOOOOOOOOOO
9887 | 403882 ( 14) OOOOOOOOOOOOOOOOOOOOOOOOOOOOO
11864 | 415674 ( 14) OOOOOOOOOOOOOOOOOOOOOOOOOOOOOO
14237 | 126880 ( 4) OOOOOOOOO
17084 | 33068 ( 1) OO.
20501 | 1 ( 0)
Percentiles
50th 6866
75th 9887
95th 14237
98th 14237
99th 17084
Min 0
Max 20501
Estimated cardinality: 2772003
EncodingStats minTTL: 0
EncodingStats minLocalDeletionTime: 1624674132 (06/26/2021 02:22:12)
EncodingStats minTimestamp: 1622528285761740 (06/01/2021 06:18:05)
KeyType: org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.ShortType,org.apache.cassandra.db.marshal.LongType,org.apache.cassandra.db.marshal.ShortType)
ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
StaticColumns:
RegularColumns: high:org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.UserType(storage_system,70726963657265636f7264,73697a65:org.apache.cassandra.db.marshal.Int32Type,7473:org.apache.cassandra.db.marshal.TimestampType,76616c7565:org.apache.cassandra.db.marshal.DoubleType)), low:org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.UserType(storage_system,70726963657265636f7264,73697a65:org.apache.cassandra.db.marshal.Int32Type,7473:org.apache.cassandra.db.marshal.TimestampType,76616c7565:org.apache.cassandra.db.marshal.DoubleType)), close:org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.UserType(storage_system,70726963657265636f7264,73697a65:org.apache.cassandra.db.marshal.Int32Type,7473:org.apache.cassandra.db.marshal.TimestampType,76616c7565:org.apache.cassandra.db.marshal.DoubleType)), open:org.apache.cassandra.db.marshal.FrozenType(org.apache.cassandra.db.marshal.UserType(storage_system,70726963657265636f7264,73697a65:org.apache.cassandra.db.marshal.Int32Type,7473:org.apache.cassandra.db.marshal.TimestampType,76616c7565:org.apache.cassandra.db.marshal.DoubleType))
IsTransient: false
[09:22:38 root@storageserver-cassandra-d1-1.fra1.quotehub:/var/lib/cassandra/data/storage_system/min-d7955270f31d11ea88fabb8dcc37b800<mailto:root@storageserver-cassandra-d1-1.fra1.quotehub:/var/lib/cassandra/data/storage_system/min-d7955270f31d11ea88fabb8dcc37b800>]
#
Re: TWCS repair and compact help
Posted by Kane Wilson <k...@raft.so>.
>
> Oh. So our data is all messed up now because of the “nodetool compact” I
> ran.
>
>
>
> Hi Erick. Thanks for the quick reply.
>
>
>
> I just want to be sure about compact. I saw Cassandra will do compaction
> by itself even when I do not run “nodetool compact” manually (nodetool
> compactionstats always has some compaction running). So this automatic
> compact by Cassandra will clean up the tombstoned data files?
>
They won't "compact", but will rather just be deleted once all the data in
the file passes its expiration time.
> Another question I have is, is there a way to un-mess my messed up data
> now?
>
Not really. The easiest way would be to re-insert all your data. If you're
not having any read performance issues you might be better just waiting the
7 days until the large SSTable is dropped.
--
raft.so - Cassandra consulting, support, and managed services
RE: TWCS repair and compact help
Posted by Eric Wong <Er...@solvians.com>.
Oh. So our data is all messed up now because of the "nodetool compact" I ran.
Hi Erick. Thanks for the quick reply.
I just want to be sure about compact. I saw Cassandra will do compaction by itself even when I do not run "nodetool compact" manually (nodetool compactionstats always has some compaction running). So this automatic compact by Cassandra will clean up the tombstoned data files?
Another question I have is, is there a way to un-mess my messed up data now?
Thanks,
Eric
From: Erick Ramirez <er...@datastax.com>
Sent: Tuesday, June 29, 2021 6:34 PM
To: user@cassandra.apache.org
Subject: Re: TWCS repair and compact help
You definitely shouldn't perform manual compactions -- you should let the normal compaction tasks take care of it. It is unnecessary to manually run compactions since it creates more problems than it solves as I've explained in this post -- https://community.datastax.com/questions/6396/<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.datastax.com%2Fquestions%2F6396%2F&data=04%7C01%7C%7C752550b0150740d8738008d93ae97621%7C621e62193ff94100b0e6fbe83e87c529%7C1%7C0%7C637605596658021039%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pGXhTowWOcVC9Ag7zAkT85RuRzUdQFdner747sj737s%3D&reserved=0>. Cheers!
Re: TWCS repair and compact help
Posted by Gábor Auth <au...@gmail.com>.
Hi,
On Tue, Jun 29, 2021 at 12:34 PM Erick Ramirez <er...@datastax.com>
wrote:
> You definitely shouldn't perform manual compactions -- you should let the
> normal compaction tasks take care of it. It is unnecessary to manually run
> compactions since it creates more problems than it solves as I've explained
> in this post -- https://community.datastax.com/questions/6396/. Cheers!
>
Same issue here... Iwant to replace SizeTieredCompactionStrategy to
TimeWindowCompactionStrategy but I cannot achieve to split of the existing
SSTables to daily SSTables. Any idea about it? :)
--
Bye,
Auth Gábor (https://iotguru.cloud)
Re: TWCS repair and compact help
Posted by Erick Ramirez <er...@datastax.com>.
You definitely shouldn't perform manual compactions -- you should let the
normal compaction tasks take care of it. It is unnecessary to manually run
compactions since it creates more problems than it solves as I've explained
in this post -- https://community.datastax.com/questions/6396/. Cheers!