You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Roy Burstein <bu...@gmail.com> on 2018/12/14 06:26:31 UTC
TWCS sstables gets merged following node removal
Hi all ,
My colleague opened Jira ticket for the issue but we are struggling with
this issue for a while and we have space issues :
https://issues.apache.org/jira/browse/CASSANDRA-14929
After removing a node from the cluster, a table that is defined as TWCS,
has sstables from different time windows merged together, making old and
new data sit in the same sstable(the full example is in the ticket).
Any idea about this?workaround?Known issues?
Thanks!
Roy
Re: TWCS sstables gets merged following node removal
Posted by Jeff Jirsa <jj...@gmail.com>.
The read repair you have disabled is the probabilistic background repair - foreground repair due to mismatch still happens
Streaming should respect windows. Streaming doesn’t write to the memtable, only the write path puts data into the memtable.
--
Jeff Jirsa
> On Dec 18, 2018, at 1:49 AM, Roy Burstein <bu...@gmail.com> wrote:
>
> read repair is disabled in this table :
>
> CREATE TABLE gil_test.my_test (
> id int,
> creation_time timestamp,
> ...
> AND crc_check_chance = 1.0
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 3600
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = 'NONE';
>
> 1228 is the result of compaction from 873 and 1196, so it makes sense that they would have the same max timestamp
> 1196 is an sstable that was created during streaming (added that part of the log before), since it was compacted right away when the node removal was done i don't have its metadata.
>
> when data is being streamed from other nodes during node removal, do i have data mixed from different time window in one stream?
> also even if it is being separated, what would stop old data being stored in the same memtable as data just being written from the application?
>
>
>> On Mon, Dec 17, 2018 at 7:18 PM Jeff Jirsa <jj...@gmail.com> wrote:
>>
>> The min timestamps vary (likely due to read repairing old values into the memtable and flushing into these sstables), but the max timestamps for both are in the same second (same microsecond, even, so probably the same write):
>>
>> Maximum timestamp: 1544903882074190
>> Maximum timestamp: 1544903882074190
>>
>> jjirsa:~ jjirsa$ date -r 1544903882
>> Sat Dec 15 11:58:02 PST 2018
>>
>> TWCS buckets based on max timestamp per file, so they belong together:
>>
>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java#L247
>>
>>
>>
>>> On Sun, Dec 16, 2018 at 11:39 PM Roy Burstein <bu...@gmail.com> wrote:
>>> hey jeff, attaching more information.
>>> so this the situation before - 3 nodes in the cluster (3.11.3 in this case but i saw same thing in 2.1 and 3.0), there is a script writing one row every minute and another script doing nodetool flush every 10 minute.
>>> window is defined as two hours, so after a few days this is how the directory listing looks :
>>>
>>> drwxr-xr-x 2 cassandra cassandra 4096 Dec 11 10:38 backups
>>> -rw-r--r-- 1 cassandra cassandra 646 Dec 12 05:25 mc-171-big-Index.db
>>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-171-big-Filter.db
>>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-171-big-Summary.db
>>> -rw-r--r-- 1 cassandra cassandra 3561 Dec 12 05:25 mc-171-big-Data.db
>>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-171-big-Digest.crc32
>>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25 mc-171-big-CompressionInfo.db
>>> -rw-r--r-- 1 cassandra cassandra 4893 Dec 12 05:25 mc-171-big-Statistics.db
>>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-171-big-TOC.txt
>>> -rw-r--r-- 1 cassandra cassandra 565 Dec 12 05:25 mc-172-big-Index.db
>>> -rw-r--r-- 1 cassandra cassandra 96 Dec 12 05:25 mc-172-big-Filter.db
>>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-172-big-Summary.db
>>> -rw-r--r-- 1 cassandra cassandra 3475 Dec 12 05:25 mc-172-big-Data.db
>>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-172-big-Digest.crc32
>>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25 mc-172-big-CompressionInfo.db
>>> -rw-r--r-- 1 cassandra cassandra 4865 Dec 12 05:25 mc-172-big-Statistics.db
>>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-172-big-TOC.txt
>>> -rw-r--r-- 1 cassandra cassandra 637 Dec 12 05:25 mc-173-big-Index.db
>>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-173-big-Filter.db
>>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-173-big-Summary.db
>>> -rw-r--r-- 1 cassandra cassandra 3678 Dec 12 05:25 mc-173-big-Data.db
>>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-173-big-Digest.crc32
>>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25 mc-173-big-CompressionInfo.db
>>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-173-big-TOC.txt
>>> -rw-r--r-- 1 cassandra cassandra 4888 Dec 12 05:25 mc-173-big-Statistics.db
>>> .
>>> .
>>> -rw-r--r-- 1 cassandra cassandra 340 Dec 15 20:10 mc-873-big-Index.db
>>> -rw-r--r-- 1 cassandra cassandra 64 Dec 15 20:10 mc-873-big-Filter.db
>>> -rw-r--r-- 1 cassandra cassandra 56 Dec 15 20:10 mc-873-big-Summary.db
>>> -rw-r--r-- 1 cassandra cassandra 1910 Dec 15 20:10 mc-873-big-Data.db
>>> -rw-r--r-- 1 cassandra cassandra 10 Dec 15 20:10 mc-873-big-Digest.crc32
>>> -rw-r--r-- 1 cassandra cassandra 51 Dec 15 20:10 mc-873-big-CompressionInfo.db
>>> -rw-r--r-- 1 cassandra cassandra 4793 Dec 15 20:10 mc-873-big-Statistics.db
>>> -rw-r--r-- 1 cassandra cassandra 92 Dec 15 20:10 mc-873-big-TOC.txt
>>> .
>>> .
>>> .
>>> -rw-r--r-- 1 cassandra cassandra 24 Dec 17 06:50 mc-1150-big-Filter.db
>>> -rw-r--r-- 1 cassandra cassandra 51 Dec 17 06:50 mc-1150-big-Index.db
>>> -rw-r--r-- 1 cassandra cassandra 56 Dec 17 06:50 mc-1150-big-Summary.db
>>> -rw-r--r-- 1 cassandra cassandra 10 Dec 17 06:50 mc-1150-big-Digest.crc32
>>> -rw-r--r-- 1 cassandra cassandra 226 Dec 17 06:50 mc-1150-big-Data.db
>>> -rw-r--r-- 1 cassandra cassandra 43 Dec 17 06:50 mc-1150-big-CompressionInfo.db
>>> -rw-r--r-- 1 cassandra cassandra 92 Dec 17 06:50 mc-1150-big-TOC.txt
>>> -rw-r--r-- 1 cassandra cassandra 4691 Dec 17 06:50 mc-1150-big-Statistics.db
>>>
>>>
>>>
>>> check out sstable 873 is from dec 15. at this point i remove node no 3.
>>> and the log shows streamed data is creating sstables 1150-1200. it also shows one of these sstables being compacted with sstable 873, resulting in sstable number 1228, an operation which from what i understand should not happen.
>>>
>>> DEBUG [StreamReceiveTask:1] 2018-12-17 06:59:54,199 StreamReceiveTask.java:213 - [Stream #5ae370a0-01c9-11e9-ba69-d743ff231136] Received 58 sstables from /10.106.12.34 ([BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1151-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1152-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1153-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1154-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1155-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1156-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1157-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1158-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1159-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1160-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1161-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1162-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1163-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1164-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1165-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1166-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1167-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1168-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1169-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1170-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1171-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1172-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1173-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1174-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1175-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1176-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1177-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1178-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1179-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1180-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1181-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1182-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1183-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1184-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1185-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1186-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1187-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1188-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1189-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1190-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1191-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1192-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1193-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1194-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1195-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1197-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1198-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1199-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1200-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1201-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0
>>>
>>>
>>>
>>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564 TimeWindowCompactionStrategy.java:308 - bucket size 2 >= 2 and not in current bucket, compacting what's here: [BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db')]
>>>
>>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564 CompactionTask.java:155 - Compacting (5b702040-01c9-11e9-ba69-d743ff231136) [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db:level=0, /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db:level=0, ]
>>>
>>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,578 CompactionTask.java:255 - Compacted (5b702040-01c9-11e9-ba69-d743ff231136) 2 sstables to [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big,] to level=0. 3.397KiB to 3.383KiB (~99% of original) in 13ms. Read Throughput = 244.120KiB/s, Write Throughput = 243.067KiB/s, Row Throughput = ~184/s. 71 total partitions merged to 71. Partition merge counts were {1:71, }
>>>
>>> see sstablemetadata output for sstable 873 and 1228. as you can see the 1228 min timstamp is old.
>>>
>>> 873:
>>> WARN 07:11:30,845 Only 45.238GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
>>> SSTable: /root/gil_sstables_before/mc-873-big
>>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>> Bloom Filter FP chance: 0.010000
>>> Minimum timestamp: 1544896864561707
>>> Maximum timestamp: 1544903882074190
>>> SSTable min local deletion time: 2147483647
>>> SSTable max local deletion time: 2147483647
>>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
>>> Compression ratio: 0.6954296160877513
>>> TTL min: 0
>>> TTL max: 0
>>> First token: -7160136740246525330 (key=42)
>>> Last token: 8209703677368372043 (key=34)
>>> minClustringValues: [2018-12-15 18:01Z]
>>> maxClustringValues: [2018-12-15 19:58Z]
>>> Estimated droppable tombstones: 0.0
>>> SSTable Level: 0
>>> Repaired at: 0
>>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001, position=19800215)=CommitLogPosition(segmentId=1544706489001, position=20586499)}
>>> totalColumnsSet: 62
>>> totalRows: 62
>>> Estimated tombstone drop times:
>>> Count Row Size Cell Count
>>> 1 0 24
>>> 2 0 7
>>> 3 0 5
>>> 4 0 1
>>> 5 0 1
>>> 6 0 0
>>> 7 0 0
>>> 8 0 0
>>> 10 0 0
>>> 12 0 0
>>> 14 0 0
>>> 17 0 0
>>> 20 0 0
>>> 24 0 0
>>> 29 0 0
>>> 35 0 0
>>> 42 0 0
>>> 50 7 0
>>> 60 17 0
>>> 72 0 0
>>> 86 5 0
>>> 103 2 0
>>> 124 5 0
>>> 149 1 0
>>> 179 0 0
>>> 215 1 0
>>> 258 0 0
>>> 310 0 0
>>> 372 0 0
>>> 446 0 0
>>> 535 0 0
>>> 642 0 0
>>> 770 0 0
>>> 924 0 0
>>> 1109 0 0
>>> 1331 0 0
>>> 1597 0 0
>>> 1916 0 0
>>> 2299 0 0
>>> 2759 0 0
>>> 3311 0 0
>>> 3973 0 0
>>> 4768 0 0
>>> 5722 0 0
>>> 6866 0 0
>>> 8239 0 0
>>> 9887 0 0
>>> 11864 0 0
>>> 14237 0 0
>>> 17084 0 0
>>> 20501 0 0
>>> 24601 0 0
>>> 29521 0 0
>>> 35425 0 0
>>> 42510 0 0
>>> 51012 0 0
>>> 61214 0 0
>>> 73457 0 0
>>> 88148 0 0
>>> 105778 0 0
>>> 126934 0 0
>>> 152321 0 0
>>> 182785 0 0
>>> 219342 0 0
>>> 263210 0 0
>>> 315852 0 0
>>> 379022 0 0
>>> 454826 0 0
>>> 545791 0 0
>>> 654949 0 0
>>> 785939 0 0
>>> 943127 0 0
>>> 1131752 0 0
>>> 1358102 0 0
>>> 1629722 0 0
>>> 1955666 0 0
>>> 2346799 0 0
>>> 2816159 0 0
>>> 3379391 0 0
>>> 4055269 0 0
>>> 4866323 0 0
>>> 5839588 0 0
>>> 7007506 0 0
>>> 8409007 0 0
>>> 10090808 0 0
>>> 12108970 0 0
>>> 14530764 0 0
>>> 17436917 0 0
>>> 20924300 0 0
>>> 25109160 0 0
>>> 30130992 0 0
>>> 36157190 0 0
>>> 43388628 0 0
>>> 52066354 0 0
>>> 62479625 0 0
>>> 74975550 0 0
>>> 89970660 0 0
>>> 107964792 0 0
>>> 129557750 0 0
>>> 155469300 0 0
>>> 186563160 0 0
>>> 223875792 0 0
>>> 268650950 0 0
>>> 322381140 0 0
>>> 386857368 0 0
>>> 464228842 0 0
>>> 557074610 0 0
>>> 668489532 0 0
>>> 802187438 0 0
>>> 962624926 0 0
>>> 1155149911 0 0
>>> 1386179893 0 0
>>> 1663415872 0 0
>>> 1996099046 0 0
>>> 2395318855 0 0
>>> 2874382626 0
>>> 3449259151 0
>>> 4139110981 0
>>> 4966933177 0
>>> 5960319812 0
>>> 7152383774 0
>>> 8582860529 0
>>> 10299432635 0
>>> 12359319162 0
>>> 14831182994 0
>>> 17797419593 0
>>> 21356903512 0
>>> 25628284214 0
>>> 30753941057 0
>>> 36904729268 0
>>> 44285675122 0
>>> 53142810146 0
>>> 63771372175 0
>>> 76525646610 0
>>> 91830775932 0
>>> 110196931118 0
>>> 132236317342 0
>>> 158683580810 0
>>> 190420296972 0
>>> 228504356366 0
>>> 274205227639 0
>>> 329046273167 0
>>> 394855527800 0
>>> 473826633360 0
>>> 568591960032 0
>>> 682310352038 0
>>> 818772422446 0
>>> 982526906935 0
>>> 1179032288322 0
>>> 1414838745986 0
>>> Estimated cardinality: 38
>>> EncodingStats minTTL: 0
>>> EncodingStats minLocalDeletionTime: 1442880000
>>> EncodingStats minTimestamp: 1544896864561707
>>> KeyType: org.apache.cassandra.db.marshal.Int32Type
>>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
>>> StaticColumns: {}
>>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>>> You have new mail in /var/spool/mail/root
>>>
>>>
>>>
>>> 1228 :
>>>
>>> WARN 07:10:51,187 Only 45.238GiB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
>>> SSTable: /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big
>>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>> Bloom Filter FP chance: 0.010000
>>> Minimum timestamp: 1544896805142916
>>> Maximum timestamp: 1544903882074190
>>> SSTable min local deletion time: 2147483647
>>> SSTable max local deletion time: 2147483647
>>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
>>> Compression ratio: 0.6942828485456369
>>> TTL min: 0
>>> TTL max: 0
>>> First token: -9108684050423740263 (key=53)
>>> Last token: 8669466869824630438 (key=77)
>>> minClustringValues: [2018-12-15 18:00Z]
>>> maxClustringValues: [2018-12-15 19:58Z]
>>> Estimated droppable tombstones: 0.0
>>> SSTable Level: 0
>>> Repaired at: 0
>>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001, position=19800215)=CommitLogPosition(segmentId=1544706489001, position=20586499)}
>>> totalColumnsSet: 113
>>> totalRows: 113
>>> Estimated tombstone drop times:
>>> Count Row Size Cell Count
>>> 1 0 41
>>> 2 0 21
>>> 3 0 7
>>> 4 0 1
>>> 5 0 1
>>> 6 0 0
>>> 7 0 0
>>> 8 0 0
>>> 10 0 0
>>> 12 0 0
>>> 14 0 0
>>> 17 0 0
>>> 20 0 0
>>> 24 0 0
>>> 29 0 0
>>> 35 0 0
>>> 42 0 0
>>> 50 16 0
>>> 60 25 0
>>> 72 0 0
>>> 86 17 0
>>> 103 4 0
>>> 124 7 0
>>> 149 1 0
>>> 179 0 0
>>> 215 1 0
>>> 258 0 0
>>> 310 0 0
>>> 372 0 0
>>> 446 0 0
>>> 535 0 0
>>> 642 0 0
>>> 770 0 0
>>> 924 0 0
>>> 1109 0 0
>>> 1331 0 0
>>> 1597 0 0
>>> 1916 0 0
>>> 2299 0 0
>>> 2759 0 0
>>> 3311 0 0
>>> 3973 0 0
>>> 4768 0 0
>>> 5722 0 0
>>> 6866 0 0
>>> 8239 0 0
>>> 9887 0 0
>>> 11864 0 0
>>> 14237 0 0
>>> 17084 0 0
>>> 20501 0 0
>>> 24601 0 0
>>> 29521 0 0
>>> 35425 0 0
>>> 42510 0 0
>>> 51012 0 0
>>> 61214 0 0
>>> 73457 0 0
>>> 88148 0 0
>>> 105778 0 0
>>> 126934 0 0
>>> 152321 0 0
>>> 182785 0 0
>>> 219342 0 0
>>> 263210 0 0
>>> 315852 0 0
>>> 379022 0 0
>>> 454826 0 0
>>> 545791 0 0
>>> 654949 0 0
>>> 785939 0 0
>>> 943127 0 0
>>> 1131752 0 0
>>> 1358102 0 0
>>> 1629722 0 0
>>> 1955666 0 0
>>> 2346799 0 0
>>> 2816159 0 0
>>> 3379391 0 0
>>> 4055269 0 0
>>> 4866323 0 0
>>> 5839588 0 0
>>> 7007506 0 0
>>> 8409007 0 0
>>> 10090808 0 0
>>> 12108970 0 0
>>> 14530764 0 0
>>> 17436917 0 0
>>> 20924300 0 0
>>> 25109160 0 0
>>> 30130992 0 0
>>> 36157190 0 0
>>> 43388628 0 0
>>> 52066354 0 0
>>> 62479625 0 0
>>> 74975550 0 0
>>> 89970660 0 0
>>> 107964792 0 0
>>> 129557750 0 0
>>> 155469300 0 0
>>> 186563160 0 0
>>> 223875792 0 0
>>> 268650950 0 0
>>> 322381140 0 0
>>> 386857368 0 0
>>> 464228842 0 0
>>> 557074610 0 0
>>> 668489532 0 0
>>> 802187438 0 0
>>> 962624926 0 0
>>> 1155149911 0 0
>>> 1386179893 0 0
>>> 1663415872 0 0
>>> 1996099046 0 0
>>> 2395318855 0 0
>>> 2874382626 0
>>> 3449259151 0
>>> 4139110981 0
>>> 4966933177 0
>>> 5960319812 0
>>> 7152383774 0
>>> 8582860529 0
>>> 10299432635 0
>>> 12359319162 0
>>> 14831182994 0
>>> 17797419593 0
>>> 21356903512 0
>>> 25628284214 0
>>> 30753941057 0
>>> 36904729268 0
>>> 44285675122 0
>>> 53142810146 0
>>> 63771372175 0
>>> 76525646610 0
>>> 91830775932 0
>>> 110196931118 0
>>> 132236317342 0
>>> 158683580810 0
>>> 190420296972 0
>>> 228504356366 0
>>> 274205227639 0
>>> 329046273167 0
>>> 394855527800 0
>>> 473826633360 0
>>> 568591960032 0
>>> 682310352038 0
>>> 818772422446 0
>>> 982526906935 0
>>> 1179032288322 0
>>> 1414838745986 0
>>> Estimated cardinality: 71
>>> EncodingStats minTTL: 0
>>> EncodingStats minLocalDeletionTime: 1442880000
>>> EncodingStats minTimestamp: 1544896805142916
>>> KeyType: org.apache.cassandra.db.marshal.Int32Type
>>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
>>> StaticColumns: {}
>>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>>>
>>>
>>>
>>>> On Fri, Dec 14, 2018 at 8:56 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>>> Remove node will stream data from all windows to remote nodes , so some compaction is expected
>>>>
>>>> Would need to see the sstablemetadata to understand what’s happening there.
>>>>
>>>> --
>>>> Jeff Jirsa
>>>>
>>>>
>>>>> On Dec 13, 2018, at 10:26 PM, Roy Burstein <bu...@gmail.com> wrote:
>>>>>
>>>>> Hi all ,
>>>>> My colleague opened Jira ticket for the issue but we are struggling with this issue for a while and we have space issues :
>>>>>
>>>>> https://issues.apache.org/jira/browse/CASSANDRA-14929
>>>>>
>>>>> After removing a node from the cluster, a table that is defined as TWCS, has sstables from different time windows merged together, making old and new data sit in the same sstable(the full example is in the ticket).
>>>>> Any idea about this?workaround?Known issues?
>>>>> Thanks!
>>>>> Roy
Re: TWCS sstables gets merged following node removal
Posted by Roy Burstein <bu...@gmail.com>.
read repair is disabled in this table :
CREATE TABLE gil_test.my_test (
id int,
creation_time timestamp,
...
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 3600
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = 'NONE';
1228 is the result of compaction from 873 and 1196, so it makes sense that
they would have the same max timestamp
1196 is an sstable that was created during streaming (added that part of
the log before), since it was compacted right away when the node removal
was done i don't have its metadata.
when data is being streamed from other nodes during node removal, do i have
data mixed from different time window in one stream?
also even if it is being separated, what would stop old data being stored
in the same memtable as data just being written from the application?
On Mon, Dec 17, 2018 at 7:18 PM Jeff Jirsa <jj...@gmail.com> wrote:
>
> The min timestamps vary (likely due to read repairing old values into the
> memtable and flushing into these sstables), but the max timestamps for both
> are in the same second (same microsecond, even, so probably the same write):
>
> Maximum timestamp: 1544903882074190
> Maximum timestamp: 1544903882074190
>
> jjirsa:~ jjirsa$ date -r 1544903882
> Sat Dec 15 11:58:02 PST 2018
>
> TWCS buckets based on max timestamp per file, so they belong together:
>
>
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java#L247
>
>
>
> On Sun, Dec 16, 2018 at 11:39 PM Roy Burstein <bu...@gmail.com>
> wrote:
>
>> hey jeff, attaching more information.
>> so this the situation before - 3 nodes in the cluster (3.11.3 in this
>> case but i saw same thing in 2.1 and 3.0), there is a script writing one
>> row every minute and another script doing nodetool flush every 10 minute.
>> window is defined as two hours, so after a few days this is how the
>> directory listing looks :
>>
>> drwxr-xr-x 2 cassandra cassandra 4096 Dec 11 10:38 backups
>> -rw-r--r-- 1 cassandra cassandra 646 Dec 12 05:25 mc-171-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-171-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-171-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 3561 Dec 12 05:25 mc-171-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-171-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>> mc-171-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 4893 Dec 12 05:25
>> mc-171-big-Statistics.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-171-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 565 Dec 12 05:25 mc-172-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 96 Dec 12 05:25 mc-172-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-172-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 3475 Dec 12 05:25 mc-172-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-172-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>> mc-172-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 4865 Dec 12 05:25
>> mc-172-big-Statistics.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-172-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 637 Dec 12 05:25 mc-173-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-173-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-173-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 3678 Dec 12 05:25 mc-173-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-173-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>> mc-173-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-173-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 4888 Dec 12 05:25
>> mc-173-big-Statistics.db
>> .
>> .
>> -rw-r--r-- 1 cassandra cassandra 340 Dec 15 20:10 mc-873-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 64 Dec 15 20:10 mc-873-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 15 20:10 mc-873-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 1910 Dec 15 20:10 mc-873-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 15 20:10 mc-873-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 51 Dec 15 20:10
>> mc-873-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 4793 Dec 15 20:10
>> mc-873-big-Statistics.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 15 20:10 mc-873-big-TOC.txt
>> .
>> .
>> .
>> -rw-r--r-- 1 cassandra cassandra 24 Dec 17 06:50 mc-1150-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 51 Dec 17 06:50 mc-1150-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 17 06:50 mc-1150-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 17 06:50
>> mc-1150-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 226 Dec 17 06:50 mc-1150-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 43 Dec 17 06:50
>> mc-1150-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 17 06:50 mc-1150-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 4691 Dec 17 06:50
>> mc-1150-big-Statistics.db
>>
>>
>>
>> check out sstable 873 is from dec 15. at this point i remove node no 3.
>> and the log shows streamed data is creating sstables 1150-1200. it also
>> shows one of these sstables being compacted with sstable 873, resulting in
>> sstable number 1228, an operation which from what i understand should not
>> happen.
>>
>> DEBUG [StreamReceiveTask:1] 2018-12-17 06:59:54,199
>> StreamReceiveTask.java:213 - [Stream #5ae370a0-01c9-11e9-ba69-d743ff231136]
>> Received 58 sstables from /10.106.12.34 ([BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1151-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1152-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1153-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1154-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1155-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1156-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1157-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1158-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1159-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1160-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1161-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1162-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1163-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1164-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1165-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1166-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1167-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1168-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1169-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1170-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1171-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1172-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1173-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1174-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1175-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1176-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1177-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1178-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1179-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1180-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1181-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1182-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1183-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1184-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1185-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1186-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1187-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1188-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1189-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1190-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1191-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1192-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1193-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1194-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1195-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1197-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1198-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1199-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1200-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1201-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0
>>
>>
>>
>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
>> TimeWindowCompactionStrategy.java:308 - bucket size 2 >= 2 and not in
>> current bucket, compacting what's here:
>> [BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db')]
>>
>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
>> CompactionTask.java:155 - Compacting (5b702040-01c9-11e9-ba69-d743ff231136)
>> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db:level=0,
>> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db:level=0,
>> ]
>>
>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,578
>> CompactionTask.java:255 - Compacted (5b702040-01c9-11e9-ba69-d743ff231136)
>> 2 sstables to
>> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big,]
>> to level=0. 3.397KiB to 3.383KiB (~99% of original) in 13ms. Read
>> Throughput = 244.120KiB/s, Write Throughput = 243.067KiB/s, Row Throughput
>> = ~184/s. 71 total partitions merged to 71. Partition merge counts were
>> {1:71, }
>>
>> see sstablemetadata output for sstable 873 and 1228. as you can see the
>> 1228 min timstamp is old.
>>
>> 873:
>> WARN 07:11:30,845 Only 45.238GiB free across all data volumes. Consider
>> adding more capacity to your cluster or removing obsolete snapshots
>> SSTable: /root/gil_sstables_before/mc-873-big
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Bloom Filter FP chance: 0.010000
>> Minimum timestamp: 1544896864561707
>> Maximum timestamp: 1544903882074190
>> SSTable min local deletion time: 2147483647
>> SSTable max local deletion time: 2147483647
>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
>> Compression ratio: 0.6954296160877513
>> TTL min: 0
>> TTL max: 0
>> First token: -7160136740246525330 (key=42)
>> Last token: 8209703677368372043 (key=34)
>> minClustringValues: [2018-12-15 18:01Z]
>> maxClustringValues: [2018-12-15 19:58Z]
>> Estimated droppable tombstones: 0.0
>> SSTable Level: 0
>> Repaired at: 0
>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
>> position=19800215)=CommitLogPosition(segmentId=1544706489001,
>> position=20586499)}
>> totalColumnsSet: 62
>> totalRows: 62
>> Estimated tombstone drop times:
>> Count Row Size Cell Count
>> 1 0 24
>> 2 0 7
>> 3 0 5
>> 4 0 1
>> 5 0 1
>> 6 0 0
>> 7 0 0
>> 8 0 0
>> 10 0 0
>> 12 0 0
>> 14 0 0
>> 17 0 0
>> 20 0 0
>> 24 0 0
>> 29 0 0
>> 35 0 0
>> 42 0 0
>> 50 7 0
>> 60 17 0
>> 72 0 0
>> 86 5 0
>> 103 2 0
>> 124 5 0
>> 149 1 0
>> 179 0 0
>> 215 1 0
>> 258 0 0
>> 310 0 0
>> 372 0 0
>> 446 0 0
>> 535 0 0
>> 642 0 0
>> 770 0 0
>> 924 0 0
>> 1109 0 0
>> 1331 0 0
>> 1597 0 0
>> 1916 0 0
>> 2299 0 0
>> 2759 0 0
>> 3311 0 0
>> 3973 0 0
>> 4768 0 0
>> 5722 0 0
>> 6866 0 0
>> 8239 0 0
>> 9887 0 0
>> 11864 0 0
>> 14237 0 0
>> 17084 0 0
>> 20501 0 0
>> 24601 0 0
>> 29521 0 0
>> 35425 0 0
>> 42510 0 0
>> 51012 0 0
>> 61214 0 0
>> 73457 0 0
>> 88148 0 0
>> 105778 0 0
>> 126934 0 0
>> 152321 0 0
>> 182785 0 0
>> 219342 0 0
>> 263210 0 0
>> 315852 0 0
>> 379022 0 0
>> 454826 0 0
>> 545791 0 0
>> 654949 0 0
>> 785939 0 0
>> 943127 0 0
>> 1131752 0 0
>> 1358102 0 0
>> 1629722 0 0
>> 1955666 0 0
>> 2346799 0 0
>> 2816159 0 0
>> 3379391 0 0
>> 4055269 0 0
>> 4866323 0 0
>> 5839588 0 0
>> 7007506 0 0
>> 8409007 0 0
>> 10090808 0 0
>> 12108970 0 0
>> 14530764 0 0
>> 17436917 0 0
>> 20924300 0 0
>> 25109160 0 0
>> 30130992 0 0
>> 36157190 0 0
>> 43388628 0 0
>> 52066354 0 0
>> 62479625 0 0
>> 74975550 0 0
>> 89970660 0 0
>> 107964792 0 0
>> 129557750 0 0
>> 155469300 0 0
>> 186563160 0 0
>> 223875792 0 0
>> 268650950 0 0
>> 322381140 0 0
>> 386857368 0 0
>> 464228842 0 0
>> 557074610 0 0
>> 668489532 0 0
>> 802187438 0 0
>> 962624926 0 0
>> 1155149911 0 0
>> 1386179893 0 0
>> 1663415872 0 0
>> 1996099046 0 0
>> 2395318855 0 0
>> 2874382626 0
>> 3449259151 0
>> 4139110981 0
>> 4966933177 0
>> 5960319812 0
>> 7152383774 0
>> 8582860529 0
>> 10299432635 0
>> 12359319162 0
>> 14831182994 0
>> 17797419593 0
>> 21356903512 0
>> 25628284214 0
>> 30753941057 0
>> 36904729268 0
>> 44285675122 0
>> 53142810146 0
>> 63771372175 0
>> 76525646610 0
>> 91830775932 0
>> 110196931118 0
>> 132236317342 0
>> 158683580810 0
>> 190420296972 0
>> 228504356366 0
>> 274205227639 0
>> 329046273167 0
>> 394855527800 0
>> 473826633360 0
>> 568591960032 0
>> 682310352038 0
>> 818772422446 0
>> 982526906935 0
>> 1179032288322 0
>> 1414838745986 0
>> Estimated cardinality: 38
>> EncodingStats minTTL: 0
>> EncodingStats minLocalDeletionTime: 1442880000
>> EncodingStats minTimestamp: 1544896864561707
>> KeyType: org.apache.cassandra.db.marshal.Int32Type
>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
>> StaticColumns: {}
>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>> You have new mail in /var/spool/mail/root
>>
>>
>>
>> 1228 :
>>
>> WARN 07:10:51,187 Only 45.238GiB free across all data volumes. Consider
>> adding more capacity to your cluster or removing obsolete snapshots
>> SSTable:
>> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Bloom Filter FP chance: 0.010000
>> Minimum timestamp: 1544896805142916
>> Maximum timestamp: 1544903882074190
>> SSTable min local deletion time: 2147483647
>> SSTable max local deletion time: 2147483647
>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
>> Compression ratio: 0.6942828485456369
>> TTL min: 0
>> TTL max: 0
>> First token: -9108684050423740263 (key=53)
>> Last token: 8669466869824630438 (key=77)
>> minClustringValues: [2018-12-15 18:00Z]
>> maxClustringValues: [2018-12-15 19:58Z]
>> Estimated droppable tombstones: 0.0
>> SSTable Level: 0
>> Repaired at: 0
>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
>> position=19800215)=CommitLogPosition(segmentId=1544706489001,
>> position=20586499)}
>> totalColumnsSet: 113
>> totalRows: 113
>> Estimated tombstone drop times:
>> Count Row Size Cell Count
>> 1 0 41
>> 2 0 21
>> 3 0 7
>> 4 0 1
>> 5 0 1
>> 6 0 0
>> 7 0 0
>> 8 0 0
>> 10 0 0
>> 12 0 0
>> 14 0 0
>> 17 0 0
>> 20 0 0
>> 24 0 0
>> 29 0 0
>> 35 0 0
>> 42 0 0
>> 50 16 0
>> 60 25 0
>> 72 0 0
>> 86 17 0
>> 103 4 0
>> 124 7 0
>> 149 1 0
>> 179 0 0
>> 215 1 0
>> 258 0 0
>> 310 0 0
>> 372 0 0
>> 446 0 0
>> 535 0 0
>> 642 0 0
>> 770 0 0
>> 924 0 0
>> 1109 0 0
>> 1331 0 0
>> 1597 0 0
>> 1916 0 0
>> 2299 0 0
>> 2759 0 0
>> 3311 0 0
>> 3973 0 0
>> 4768 0 0
>> 5722 0 0
>> 6866 0 0
>> 8239 0 0
>> 9887 0 0
>> 11864 0 0
>> 14237 0 0
>> 17084 0 0
>> 20501 0 0
>> 24601 0 0
>> 29521 0 0
>> 35425 0 0
>> 42510 0 0
>> 51012 0 0
>> 61214 0 0
>> 73457 0 0
>> 88148 0 0
>> 105778 0 0
>> 126934 0 0
>> 152321 0 0
>> 182785 0 0
>> 219342 0 0
>> 263210 0 0
>> 315852 0 0
>> 379022 0 0
>> 454826 0 0
>> 545791 0 0
>> 654949 0 0
>> 785939 0 0
>> 943127 0 0
>> 1131752 0 0
>> 1358102 0 0
>> 1629722 0 0
>> 1955666 0 0
>> 2346799 0 0
>> 2816159 0 0
>> 3379391 0 0
>> 4055269 0 0
>> 4866323 0 0
>> 5839588 0 0
>> 7007506 0 0
>> 8409007 0 0
>> 10090808 0 0
>> 12108970 0 0
>> 14530764 0 0
>> 17436917 0 0
>> 20924300 0 0
>> 25109160 0 0
>> 30130992 0 0
>> 36157190 0 0
>> 43388628 0 0
>> 52066354 0 0
>> 62479625 0 0
>> 74975550 0 0
>> 89970660 0 0
>> 107964792 0 0
>> 129557750 0 0
>> 155469300 0 0
>> 186563160 0 0
>> 223875792 0 0
>> 268650950 0 0
>> 322381140 0 0
>> 386857368 0 0
>> 464228842 0 0
>> 557074610 0 0
>> 668489532 0 0
>> 802187438 0 0
>> 962624926 0 0
>> 1155149911 0 0
>> 1386179893 0 0
>> 1663415872 0 0
>> 1996099046 0 0
>> 2395318855 0 0
>> 2874382626 0
>> 3449259151 0
>> 4139110981 0
>> 4966933177 0
>> 5960319812 0
>> 7152383774 0
>> 8582860529 0
>> 10299432635 0
>> 12359319162 0
>> 14831182994 0
>> 17797419593 0
>> 21356903512 0
>> 25628284214 0
>> 30753941057 0
>> 36904729268 0
>> 44285675122 0
>> 53142810146 0
>> 63771372175 0
>> 76525646610 0
>> 91830775932 0
>> 110196931118 0
>> 132236317342 0
>> 158683580810 0
>> 190420296972 0
>> 228504356366 0
>> 274205227639 0
>> 329046273167 0
>> 394855527800 0
>> 473826633360 0
>> 568591960032 0
>> 682310352038 0
>> 818772422446 0
>> 982526906935 0
>> 1179032288322 0
>> 1414838745986 0
>> Estimated cardinality: 71
>> EncodingStats minTTL: 0
>> EncodingStats minLocalDeletionTime: 1442880000
>> EncodingStats minTimestamp: 1544896805142916
>> KeyType: org.apache.cassandra.db.marshal.Int32Type
>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
>> StaticColumns: {}
>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>>
>>
>>
>> On Fri, Dec 14, 2018 at 8:56 AM Jeff Jirsa <jj...@gmail.com> wrote:
>>
>>> Remove node will stream data from all windows to remote nodes , so some
>>> compaction is expected
>>>
>>> Would need to see the sstablemetadata to understand what’s happening
>>> there.
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Dec 13, 2018, at 10:26 PM, Roy Burstein <bu...@gmail.com>
>>> wrote:
>>>
>>> Hi all ,
>>>
>>> My colleague opened Jira ticket for the issue but we are struggling with
>>> this issue for a while and we have space issues :
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-14929
>>>
>>>
>>> After removing a node from the cluster, a table that is defined as TWCS,
>>> has sstables from different time windows merged together, making old and
>>> new data sit in the same sstable(the full example is in the ticket).
>>>
>>> Any idea about this?workaround?Known issues?
>>>
>>> Thanks!
>>>
>>> Roy
>>>
>>>
Re: TWCS sstables gets merged following node removal
Posted by Jeff Jirsa <jj...@gmail.com>.
The min timestamps vary (likely due to read repairing old values into the
memtable and flushing into these sstables), but the max timestamps for both
are in the same second (same microsecond, even, so probably the same write):
Maximum timestamp: 1544903882074190
Maximum timestamp: 1544903882074190
jjirsa:~ jjirsa$ date -r 1544903882
Sat Dec 15 11:58:02 PST 2018
TWCS buckets based on max timestamp per file, so they belong together:
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/compaction/TimeWindowCompactionStrategy.java#L247
On Sun, Dec 16, 2018 at 11:39 PM Roy Burstein <bu...@gmail.com>
wrote:
> hey jeff, attaching more information.
> so this the situation before - 3 nodes in the cluster (3.11.3 in this case
> but i saw same thing in 2.1 and 3.0), there is a script writing one row
> every minute and another script doing nodetool flush every 10 minute.
> window is defined as two hours, so after a few days this is how the
> directory listing looks :
>
> drwxr-xr-x 2 cassandra cassandra 4096 Dec 11 10:38 backups
> -rw-r--r-- 1 cassandra cassandra 646 Dec 12 05:25 mc-171-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-171-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-171-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 3561 Dec 12 05:25 mc-171-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-171-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
> mc-171-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 4893 Dec 12 05:25 mc-171-big-Statistics.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-171-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 565 Dec 12 05:25 mc-172-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 96 Dec 12 05:25 mc-172-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-172-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 3475 Dec 12 05:25 mc-172-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-172-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
> mc-172-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 4865 Dec 12 05:25 mc-172-big-Statistics.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-172-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 637 Dec 12 05:25 mc-173-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-173-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-173-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 3678 Dec 12 05:25 mc-173-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-173-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
> mc-173-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-173-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 4888 Dec 12 05:25 mc-173-big-Statistics.db
> .
> .
> -rw-r--r-- 1 cassandra cassandra 340 Dec 15 20:10 mc-873-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 64 Dec 15 20:10 mc-873-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 15 20:10 mc-873-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 1910 Dec 15 20:10 mc-873-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 15 20:10 mc-873-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 51 Dec 15 20:10
> mc-873-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 4793 Dec 15 20:10 mc-873-big-Statistics.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 15 20:10 mc-873-big-TOC.txt
> .
> .
> .
> -rw-r--r-- 1 cassandra cassandra 24 Dec 17 06:50 mc-1150-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 51 Dec 17 06:50 mc-1150-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 17 06:50 mc-1150-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 17 06:50 mc-1150-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 226 Dec 17 06:50 mc-1150-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 43 Dec 17 06:50
> mc-1150-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 17 06:50 mc-1150-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 4691 Dec 17 06:50
> mc-1150-big-Statistics.db
>
>
>
> check out sstable 873 is from dec 15. at this point i remove node no 3.
> and the log shows streamed data is creating sstables 1150-1200. it also
> shows one of these sstables being compacted with sstable 873, resulting in
> sstable number 1228, an operation which from what i understand should not
> happen.
>
> DEBUG [StreamReceiveTask:1] 2018-12-17 06:59:54,199
> StreamReceiveTask.java:213 - [Stream #5ae370a0-01c9-11e9-ba69-d743ff231136]
> Received 58 sstables from /10.106.12.34 ([BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1151-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1152-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1153-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1154-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1155-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1156-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1157-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1158-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1159-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1160-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1161-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1162-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1163-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1164-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1165-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1166-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1167-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1168-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1169-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1170-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1171-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1172-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1173-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1174-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1175-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1176-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1177-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1178-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1179-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1180-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1181-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1182-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1183-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1184-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1185-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1186-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1187-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1188-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1189-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1190-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1191-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1192-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1193-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1194-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1195-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1197-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1198-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1199-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1200-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1201-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0
>
>
>
> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
> TimeWindowCompactionStrategy.java:308 - bucket size 2 >= 2 and not in
> current bucket, compacting what's here:
> [BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db')]
>
> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
> CompactionTask.java:155 - Compacting (5b702040-01c9-11e9-ba69-d743ff231136)
> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db:level=0,
> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db:level=0,
> ]
>
> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,578
> CompactionTask.java:255 - Compacted (5b702040-01c9-11e9-ba69-d743ff231136)
> 2 sstables to
> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big,]
> to level=0. 3.397KiB to 3.383KiB (~99% of original) in 13ms. Read
> Throughput = 244.120KiB/s, Write Throughput = 243.067KiB/s, Row Throughput
> = ~184/s. 71 total partitions merged to 71. Partition merge counts were
> {1:71, }
>
> see sstablemetadata output for sstable 873 and 1228. as you can see the
> 1228 min timstamp is old.
>
> 873:
> WARN 07:11:30,845 Only 45.238GiB free across all data volumes. Consider
> adding more capacity to your cluster or removing obsolete snapshots
> SSTable: /root/gil_sstables_before/mc-873-big
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Bloom Filter FP chance: 0.010000
> Minimum timestamp: 1544896864561707
> Maximum timestamp: 1544903882074190
> SSTable min local deletion time: 2147483647
> SSTable max local deletion time: 2147483647
> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
> Compression ratio: 0.6954296160877513
> TTL min: 0
> TTL max: 0
> First token: -7160136740246525330 (key=42)
> Last token: 8209703677368372043 (key=34)
> minClustringValues: [2018-12-15 18:01Z]
> maxClustringValues: [2018-12-15 19:58Z]
> Estimated droppable tombstones: 0.0
> SSTable Level: 0
> Repaired at: 0
> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
> position=19800215)=CommitLogPosition(segmentId=1544706489001,
> position=20586499)}
> totalColumnsSet: 62
> totalRows: 62
> Estimated tombstone drop times:
> Count Row Size Cell Count
> 1 0 24
> 2 0 7
> 3 0 5
> 4 0 1
> 5 0 1
> 6 0 0
> 7 0 0
> 8 0 0
> 10 0 0
> 12 0 0
> 14 0 0
> 17 0 0
> 20 0 0
> 24 0 0
> 29 0 0
> 35 0 0
> 42 0 0
> 50 7 0
> 60 17 0
> 72 0 0
> 86 5 0
> 103 2 0
> 124 5 0
> 149 1 0
> 179 0 0
> 215 1 0
> 258 0 0
> 310 0 0
> 372 0 0
> 446 0 0
> 535 0 0
> 642 0 0
> 770 0 0
> 924 0 0
> 1109 0 0
> 1331 0 0
> 1597 0 0
> 1916 0 0
> 2299 0 0
> 2759 0 0
> 3311 0 0
> 3973 0 0
> 4768 0 0
> 5722 0 0
> 6866 0 0
> 8239 0 0
> 9887 0 0
> 11864 0 0
> 14237 0 0
> 17084 0 0
> 20501 0 0
> 24601 0 0
> 29521 0 0
> 35425 0 0
> 42510 0 0
> 51012 0 0
> 61214 0 0
> 73457 0 0
> 88148 0 0
> 105778 0 0
> 126934 0 0
> 152321 0 0
> 182785 0 0
> 219342 0 0
> 263210 0 0
> 315852 0 0
> 379022 0 0
> 454826 0 0
> 545791 0 0
> 654949 0 0
> 785939 0 0
> 943127 0 0
> 1131752 0 0
> 1358102 0 0
> 1629722 0 0
> 1955666 0 0
> 2346799 0 0
> 2816159 0 0
> 3379391 0 0
> 4055269 0 0
> 4866323 0 0
> 5839588 0 0
> 7007506 0 0
> 8409007 0 0
> 10090808 0 0
> 12108970 0 0
> 14530764 0 0
> 17436917 0 0
> 20924300 0 0
> 25109160 0 0
> 30130992 0 0
> 36157190 0 0
> 43388628 0 0
> 52066354 0 0
> 62479625 0 0
> 74975550 0 0
> 89970660 0 0
> 107964792 0 0
> 129557750 0 0
> 155469300 0 0
> 186563160 0 0
> 223875792 0 0
> 268650950 0 0
> 322381140 0 0
> 386857368 0 0
> 464228842 0 0
> 557074610 0 0
> 668489532 0 0
> 802187438 0 0
> 962624926 0 0
> 1155149911 0 0
> 1386179893 0 0
> 1663415872 0 0
> 1996099046 0 0
> 2395318855 0 0
> 2874382626 0
> 3449259151 0
> 4139110981 0
> 4966933177 0
> 5960319812 0
> 7152383774 0
> 8582860529 0
> 10299432635 0
> 12359319162 0
> 14831182994 0
> 17797419593 0
> 21356903512 0
> 25628284214 0
> 30753941057 0
> 36904729268 0
> 44285675122 0
> 53142810146 0
> 63771372175 0
> 76525646610 0
> 91830775932 0
> 110196931118 0
> 132236317342 0
> 158683580810 0
> 190420296972 0
> 228504356366 0
> 274205227639 0
> 329046273167 0
> 394855527800 0
> 473826633360 0
> 568591960032 0
> 682310352038 0
> 818772422446 0
> 982526906935 0
> 1179032288322 0
> 1414838745986 0
> Estimated cardinality: 38
> EncodingStats minTTL: 0
> EncodingStats minLocalDeletionTime: 1442880000
> EncodingStats minTimestamp: 1544896864561707
> KeyType: org.apache.cassandra.db.marshal.Int32Type
> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
> StaticColumns: {}
> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
> You have new mail in /var/spool/mail/root
>
>
>
> 1228 :
>
> WARN 07:10:51,187 Only 45.238GiB free across all data volumes. Consider
> adding more capacity to your cluster or removing obsolete snapshots
> SSTable:
> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Bloom Filter FP chance: 0.010000
> Minimum timestamp: 1544896805142916
> Maximum timestamp: 1544903882074190
> SSTable min local deletion time: 2147483647
> SSTable max local deletion time: 2147483647
> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
> Compression ratio: 0.6942828485456369
> TTL min: 0
> TTL max: 0
> First token: -9108684050423740263 (key=53)
> Last token: 8669466869824630438 (key=77)
> minClustringValues: [2018-12-15 18:00Z]
> maxClustringValues: [2018-12-15 19:58Z]
> Estimated droppable tombstones: 0.0
> SSTable Level: 0
> Repaired at: 0
> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
> position=19800215)=CommitLogPosition(segmentId=1544706489001,
> position=20586499)}
> totalColumnsSet: 113
> totalRows: 113
> Estimated tombstone drop times:
> Count Row Size Cell Count
> 1 0 41
> 2 0 21
> 3 0 7
> 4 0 1
> 5 0 1
> 6 0 0
> 7 0 0
> 8 0 0
> 10 0 0
> 12 0 0
> 14 0 0
> 17 0 0
> 20 0 0
> 24 0 0
> 29 0 0
> 35 0 0
> 42 0 0
> 50 16 0
> 60 25 0
> 72 0 0
> 86 17 0
> 103 4 0
> 124 7 0
> 149 1 0
> 179 0 0
> 215 1 0
> 258 0 0
> 310 0 0
> 372 0 0
> 446 0 0
> 535 0 0
> 642 0 0
> 770 0 0
> 924 0 0
> 1109 0 0
> 1331 0 0
> 1597 0 0
> 1916 0 0
> 2299 0 0
> 2759 0 0
> 3311 0 0
> 3973 0 0
> 4768 0 0
> 5722 0 0
> 6866 0 0
> 8239 0 0
> 9887 0 0
> 11864 0 0
> 14237 0 0
> 17084 0 0
> 20501 0 0
> 24601 0 0
> 29521 0 0
> 35425 0 0
> 42510 0 0
> 51012 0 0
> 61214 0 0
> 73457 0 0
> 88148 0 0
> 105778 0 0
> 126934 0 0
> 152321 0 0
> 182785 0 0
> 219342 0 0
> 263210 0 0
> 315852 0 0
> 379022 0 0
> 454826 0 0
> 545791 0 0
> 654949 0 0
> 785939 0 0
> 943127 0 0
> 1131752 0 0
> 1358102 0 0
> 1629722 0 0
> 1955666 0 0
> 2346799 0 0
> 2816159 0 0
> 3379391 0 0
> 4055269 0 0
> 4866323 0 0
> 5839588 0 0
> 7007506 0 0
> 8409007 0 0
> 10090808 0 0
> 12108970 0 0
> 14530764 0 0
> 17436917 0 0
> 20924300 0 0
> 25109160 0 0
> 30130992 0 0
> 36157190 0 0
> 43388628 0 0
> 52066354 0 0
> 62479625 0 0
> 74975550 0 0
> 89970660 0 0
> 107964792 0 0
> 129557750 0 0
> 155469300 0 0
> 186563160 0 0
> 223875792 0 0
> 268650950 0 0
> 322381140 0 0
> 386857368 0 0
> 464228842 0 0
> 557074610 0 0
> 668489532 0 0
> 802187438 0 0
> 962624926 0 0
> 1155149911 0 0
> 1386179893 0 0
> 1663415872 0 0
> 1996099046 0 0
> 2395318855 0 0
> 2874382626 0
> 3449259151 0
> 4139110981 0
> 4966933177 0
> 5960319812 0
> 7152383774 0
> 8582860529 0
> 10299432635 0
> 12359319162 0
> 14831182994 0
> 17797419593 0
> 21356903512 0
> 25628284214 0
> 30753941057 0
> 36904729268 0
> 44285675122 0
> 53142810146 0
> 63771372175 0
> 76525646610 0
> 91830775932 0
> 110196931118 0
> 132236317342 0
> 158683580810 0
> 190420296972 0
> 228504356366 0
> 274205227639 0
> 329046273167 0
> 394855527800 0
> 473826633360 0
> 568591960032 0
> 682310352038 0
> 818772422446 0
> 982526906935 0
> 1179032288322 0
> 1414838745986 0
> Estimated cardinality: 71
> EncodingStats minTTL: 0
> EncodingStats minLocalDeletionTime: 1442880000
> EncodingStats minTimestamp: 1544896805142916
> KeyType: org.apache.cassandra.db.marshal.Int32Type
> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
> StaticColumns: {}
> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>
>
>
> On Fri, Dec 14, 2018 at 8:56 AM Jeff Jirsa <jj...@gmail.com> wrote:
>
>> Remove node will stream data from all windows to remote nodes , so some
>> compaction is expected
>>
>> Would need to see the sstablemetadata to understand what’s happening
>> there.
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Dec 13, 2018, at 10:26 PM, Roy Burstein <bu...@gmail.com>
>> wrote:
>>
>> Hi all ,
>>
>> My colleague opened Jira ticket for the issue but we are struggling with
>> this issue for a while and we have space issues :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-14929
>>
>>
>> After removing a node from the cluster, a table that is defined as TWCS,
>> has sstables from different time windows merged together, making old and
>> new data sit in the same sstable(the full example is in the ticket).
>>
>> Any idea about this?workaround?Known issues?
>>
>> Thanks!
>>
>> Roy
>>
>>
Re: TWCS sstables gets merged following node removal
Posted by Roy Burstein <bu...@gmail.com>.
hey jeff, attaching more information.
so this the situation before - 3 nodes in the cluster (3.11.3 in this case
but i saw same thing in 2.1 and 3.0), there is a script writing one row
every minute and another script doing nodetool flush every 10 minute.
window is defined as two hours, so after a few days this is how the
directory listing looks :
drwxr-xr-x 2 cassandra cassandra 4096 Dec 11 10:38 backups
-rw-r--r-- 1 cassandra cassandra 646 Dec 12 05:25 mc-171-big-Index.db
-rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-171-big-Filter.db
-rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-171-big-Summary.db
-rw-r--r-- 1 cassandra cassandra 3561 Dec 12 05:25 mc-171-big-Data.db
-rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-171-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
mc-171-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 4893 Dec 12 05:25 mc-171-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-171-big-TOC.txt
-rw-r--r-- 1 cassandra cassandra 565 Dec 12 05:25 mc-172-big-Index.db
-rw-r--r-- 1 cassandra cassandra 96 Dec 12 05:25 mc-172-big-Filter.db
-rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-172-big-Summary.db
-rw-r--r-- 1 cassandra cassandra 3475 Dec 12 05:25 mc-172-big-Data.db
-rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-172-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
mc-172-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 4865 Dec 12 05:25 mc-172-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-172-big-TOC.txt
-rw-r--r-- 1 cassandra cassandra 637 Dec 12 05:25 mc-173-big-Index.db
-rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-173-big-Filter.db
-rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-173-big-Summary.db
-rw-r--r-- 1 cassandra cassandra 3678 Dec 12 05:25 mc-173-big-Data.db
-rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-173-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
mc-173-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-173-big-TOC.txt
-rw-r--r-- 1 cassandra cassandra 4888 Dec 12 05:25 mc-173-big-Statistics.db
.
.
-rw-r--r-- 1 cassandra cassandra 340 Dec 15 20:10 mc-873-big-Index.db
-rw-r--r-- 1 cassandra cassandra 64 Dec 15 20:10 mc-873-big-Filter.db
-rw-r--r-- 1 cassandra cassandra 56 Dec 15 20:10 mc-873-big-Summary.db
-rw-r--r-- 1 cassandra cassandra 1910 Dec 15 20:10 mc-873-big-Data.db
-rw-r--r-- 1 cassandra cassandra 10 Dec 15 20:10 mc-873-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra 51 Dec 15 20:10
mc-873-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 4793 Dec 15 20:10 mc-873-big-Statistics.db
-rw-r--r-- 1 cassandra cassandra 92 Dec 15 20:10 mc-873-big-TOC.txt
.
.
.
-rw-r--r-- 1 cassandra cassandra 24 Dec 17 06:50 mc-1150-big-Filter.db
-rw-r--r-- 1 cassandra cassandra 51 Dec 17 06:50 mc-1150-big-Index.db
-rw-r--r-- 1 cassandra cassandra 56 Dec 17 06:50 mc-1150-big-Summary.db
-rw-r--r-- 1 cassandra cassandra 10 Dec 17 06:50 mc-1150-big-Digest.crc32
-rw-r--r-- 1 cassandra cassandra 226 Dec 17 06:50 mc-1150-big-Data.db
-rw-r--r-- 1 cassandra cassandra 43 Dec 17 06:50
mc-1150-big-CompressionInfo.db
-rw-r--r-- 1 cassandra cassandra 92 Dec 17 06:50 mc-1150-big-TOC.txt
-rw-r--r-- 1 cassandra cassandra 4691 Dec 17 06:50 mc-1150-big-Statistics.db
check out sstable 873 is from dec 15. at this point i remove node no 3.
and the log shows streamed data is creating sstables 1150-1200. it also
shows one of these sstables being compacted with sstable 873, resulting in
sstable number 1228, an operation which from what i understand should not
happen.
DEBUG [StreamReceiveTask:1] 2018-12-17 06:59:54,199
StreamReceiveTask.java:213 - [Stream #5ae370a0-01c9-11e9-ba69-d743ff231136]
Received 58 sstables from /10.106.12.34
([BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1151-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1152-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1153-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1154-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1155-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1156-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1157-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1158-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1159-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1160-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1161-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1162-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1163-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1164-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1165-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1166-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1167-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1168-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1169-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1170-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1171-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1172-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1173-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1174-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1175-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1176-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1177-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1178-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1179-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1180-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1181-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1182-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1183-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1184-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1185-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1186-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1187-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1188-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1189-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1190-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1191-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1192-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1193-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1194-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1195-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1197-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1198-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1199-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1200-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1201-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0
DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
TimeWindowCompactionStrategy.java:308 - bucket size 2 >= 2 and not in
current bucket, compacting what's here:
[BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db'),
BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db')]
DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
CompactionTask.java:155 - Compacting (5b702040-01c9-11e9-ba69-d743ff231136)
[/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db:level=0,
/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,578
CompactionTask.java:255 - Compacted (5b702040-01c9-11e9-ba69-d743ff231136)
2 sstables to
[/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big,]
to level=0. 3.397KiB to 3.383KiB (~99% of original) in 13ms. Read
Throughput = 244.120KiB/s, Write Throughput = 243.067KiB/s, Row Throughput
= ~184/s. 71 total partitions merged to 71. Partition merge counts were
{1:71, }
see sstablemetadata output for sstable 873 and 1228. as you can see the
1228 min timstamp is old.
873:
WARN 07:11:30,845 Only 45.238GiB free across all data volumes. Consider
adding more capacity to your cluster or removing obsolete snapshots
SSTable: /root/gil_sstables_before/mc-873-big
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.010000
Minimum timestamp: 1544896864561707
Maximum timestamp: 1544903882074190
SSTable min local deletion time: 2147483647
SSTable max local deletion time: 2147483647
Compressor: org.apache.cassandra.io.compress.LZ4Compressor
Compression ratio: 0.6954296160877513
TTL min: 0
TTL max: 0
First token: -7160136740246525330 (key=42)
Last token: 8209703677368372043 (key=34)
minClustringValues: [2018-12-15 18:01Z]
maxClustringValues: [2018-12-15 19:58Z]
Estimated droppable tombstones: 0.0
SSTable Level: 0
Repaired at: 0
Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
position=19800215)=CommitLogPosition(segmentId=1544706489001,
position=20586499)}
totalColumnsSet: 62
totalRows: 62
Estimated tombstone drop times:
Count Row Size Cell Count
1 0 24
2 0 7
3 0 5
4 0 1
5 0 1
6 0 0
7 0 0
8 0 0
10 0 0
12 0 0
14 0 0
17 0 0
20 0 0
24 0 0
29 0 0
35 0 0
42 0 0
50 7 0
60 17 0
72 0 0
86 5 0
103 2 0
124 5 0
149 1 0
179 0 0
215 1 0
258 0 0
310 0 0
372 0 0
446 0 0
535 0 0
642 0 0
770 0 0
924 0 0
1109 0 0
1331 0 0
1597 0 0
1916 0 0
2299 0 0
2759 0 0
3311 0 0
3973 0 0
4768 0 0
5722 0 0
6866 0 0
8239 0 0
9887 0 0
11864 0 0
14237 0 0
17084 0 0
20501 0 0
24601 0 0
29521 0 0
35425 0 0
42510 0 0
51012 0 0
61214 0 0
73457 0 0
88148 0 0
105778 0 0
126934 0 0
152321 0 0
182785 0 0
219342 0 0
263210 0 0
315852 0 0
379022 0 0
454826 0 0
545791 0 0
654949 0 0
785939 0 0
943127 0 0
1131752 0 0
1358102 0 0
1629722 0 0
1955666 0 0
2346799 0 0
2816159 0 0
3379391 0 0
4055269 0 0
4866323 0 0
5839588 0 0
7007506 0 0
8409007 0 0
10090808 0 0
12108970 0 0
14530764 0 0
17436917 0 0
20924300 0 0
25109160 0 0
30130992 0 0
36157190 0 0
43388628 0 0
52066354 0 0
62479625 0 0
74975550 0 0
89970660 0 0
107964792 0 0
129557750 0 0
155469300 0 0
186563160 0 0
223875792 0 0
268650950 0 0
322381140 0 0
386857368 0 0
464228842 0 0
557074610 0 0
668489532 0 0
802187438 0 0
962624926 0 0
1155149911 0 0
1386179893 0 0
1663415872 0 0
1996099046 0 0
2395318855 0 0
2874382626 0
3449259151 0
4139110981 0
4966933177 0
5960319812 0
7152383774 0
8582860529 0
10299432635 0
12359319162 0
14831182994 0
17797419593 0
21356903512 0
25628284214 0
30753941057 0
36904729268 0
44285675122 0
53142810146 0
63771372175 0
76525646610 0
91830775932 0
110196931118 0
132236317342 0
158683580810 0
190420296972 0
228504356366 0
274205227639 0
329046273167 0
394855527800 0
473826633360 0
568591960032 0
682310352038 0
818772422446 0
982526906935 0
1179032288322 0
1414838745986 0
Estimated cardinality: 38
EncodingStats minTTL: 0
EncodingStats minLocalDeletionTime: 1442880000
EncodingStats minTimestamp: 1544896864561707
KeyType: org.apache.cassandra.db.marshal.Int32Type
ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
StaticColumns: {}
RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
You have new mail in /var/spool/mail/root
1228 :
WARN 07:10:51,187 Only 45.238GiB free across all data volumes. Consider
adding more capacity to your cluster or removing obsolete snapshots
SSTable:
/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.010000
Minimum timestamp: 1544896805142916
Maximum timestamp: 1544903882074190
SSTable min local deletion time: 2147483647
SSTable max local deletion time: 2147483647
Compressor: org.apache.cassandra.io.compress.LZ4Compressor
Compression ratio: 0.6942828485456369
TTL min: 0
TTL max: 0
First token: -9108684050423740263 (key=53)
Last token: 8669466869824630438 (key=77)
minClustringValues: [2018-12-15 18:00Z]
maxClustringValues: [2018-12-15 19:58Z]
Estimated droppable tombstones: 0.0
SSTable Level: 0
Repaired at: 0
Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
position=19800215)=CommitLogPosition(segmentId=1544706489001,
position=20586499)}
totalColumnsSet: 113
totalRows: 113
Estimated tombstone drop times:
Count Row Size Cell Count
1 0 41
2 0 21
3 0 7
4 0 1
5 0 1
6 0 0
7 0 0
8 0 0
10 0 0
12 0 0
14 0 0
17 0 0
20 0 0
24 0 0
29 0 0
35 0 0
42 0 0
50 16 0
60 25 0
72 0 0
86 17 0
103 4 0
124 7 0
149 1 0
179 0 0
215 1 0
258 0 0
310 0 0
372 0 0
446 0 0
535 0 0
642 0 0
770 0 0
924 0 0
1109 0 0
1331 0 0
1597 0 0
1916 0 0
2299 0 0
2759 0 0
3311 0 0
3973 0 0
4768 0 0
5722 0 0
6866 0 0
8239 0 0
9887 0 0
11864 0 0
14237 0 0
17084 0 0
20501 0 0
24601 0 0
29521 0 0
35425 0 0
42510 0 0
51012 0 0
61214 0 0
73457 0 0
88148 0 0
105778 0 0
126934 0 0
152321 0 0
182785 0 0
219342 0 0
263210 0 0
315852 0 0
379022 0 0
454826 0 0
545791 0 0
654949 0 0
785939 0 0
943127 0 0
1131752 0 0
1358102 0 0
1629722 0 0
1955666 0 0
2346799 0 0
2816159 0 0
3379391 0 0
4055269 0 0
4866323 0 0
5839588 0 0
7007506 0 0
8409007 0 0
10090808 0 0
12108970 0 0
14530764 0 0
17436917 0 0
20924300 0 0
25109160 0 0
30130992 0 0
36157190 0 0
43388628 0 0
52066354 0 0
62479625 0 0
74975550 0 0
89970660 0 0
107964792 0 0
129557750 0 0
155469300 0 0
186563160 0 0
223875792 0 0
268650950 0 0
322381140 0 0
386857368 0 0
464228842 0 0
557074610 0 0
668489532 0 0
802187438 0 0
962624926 0 0
1155149911 0 0
1386179893 0 0
1663415872 0 0
1996099046 0 0
2395318855 0 0
2874382626 0
3449259151 0
4139110981 0
4966933177 0
5960319812 0
7152383774 0
8582860529 0
10299432635 0
12359319162 0
14831182994 0
17797419593 0
21356903512 0
25628284214 0
30753941057 0
36904729268 0
44285675122 0
53142810146 0
63771372175 0
76525646610 0
91830775932 0
110196931118 0
132236317342 0
158683580810 0
190420296972 0
228504356366 0
274205227639 0
329046273167 0
394855527800 0
473826633360 0
568591960032 0
682310352038 0
818772422446 0
982526906935 0
1179032288322 0
1414838745986 0
Estimated cardinality: 71
EncodingStats minTTL: 0
EncodingStats minLocalDeletionTime: 1442880000
EncodingStats minTimestamp: 1544896805142916
KeyType: org.apache.cassandra.db.marshal.Int32Type
ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
StaticColumns: {}
RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
On Fri, Dec 14, 2018 at 8:56 AM Jeff Jirsa <jj...@gmail.com> wrote:
> Remove node will stream data from all windows to remote nodes , so some
> compaction is expected
>
> Would need to see the sstablemetadata to understand what’s happening
> there.
>
> --
> Jeff Jirsa
>
>
> On Dec 13, 2018, at 10:26 PM, Roy Burstein <bu...@gmail.com> wrote:
>
> Hi all ,
>
> My colleague opened Jira ticket for the issue but we are struggling with
> this issue for a while and we have space issues :
>
> https://issues.apache.org/jira/browse/CASSANDRA-14929
>
>
> After removing a node from the cluster, a table that is defined as TWCS,
> has sstables from different time windows merged together, making old and
> new data sit in the same sstable(the full example is in the ticket).
>
> Any idea about this?workaround?Known issues?
>
> Thanks!
>
> Roy
>
>
Re: TWCS sstables gets merged following node removal
Posted by Gil Ganz <gi...@gmail.com>.
the reason for this all test i conducted i we have a prod cluster of 2.1
where twcs was defined on many tables a few months ago (all tables with
read repair setting disabled, also speculative retry disabled), and looked
very good. a month ago we noticed most of the servers had their data all in
newly created sstables (looking like plain size tiered), and we are not
sure what caused it (nobody ran major compaction on all the tables on all
those nodes). i was unable to find a bug in the cassandra jira that would
explain how this could happen. is there something you know that might cause
this (again, this is 2.1, with your jar, might be something specifically
there). thanks
gil
On Wed, Dec 19, 2018 at 1:39 PM Gil Ganz <gi...@gmail.com> wrote:
> sounds like the foreground read repair can cause issues to twcs (mix old
> and new data in same sstable), is there a way to disable the foreground
> read repair? is that indeed the case that it's problematic?
>
>
>
> On Mon, Dec 17, 2018 at 9:21 AM Gil Ganz <gi...@gmail.com> wrote:
>
>> hey jeff, attaching more information.
>> so this the situation before - 3 nodes in the cluster (3.11.3 in this
>> case but i saw same thing in 2.1 and 3.0), there is a script writing one
>> row every minute and another script doing nodetool flush every 10 minute.
>> window is defined as two hours, so after a few days this is how the
>> directory listing looks :
>>
>> drwxr-xr-x 2 cassandra cassandra 4096 Dec 11 10:38 backups
>> -rw-r--r-- 1 cassandra cassandra 646 Dec 12 05:25 mc-171-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-171-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-171-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 3561 Dec 12 05:25 mc-171-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-171-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>> mc-171-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 4893 Dec 12 05:25
>> mc-171-big-Statistics.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-171-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 565 Dec 12 05:25 mc-172-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 96 Dec 12 05:25 mc-172-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-172-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 3475 Dec 12 05:25 mc-172-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-172-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>> mc-172-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 4865 Dec 12 05:25
>> mc-172-big-Statistics.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-172-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 637 Dec 12 05:25 mc-173-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-173-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-173-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 3678 Dec 12 05:25 mc-173-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-173-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>> mc-173-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-173-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 4888 Dec 12 05:25
>> mc-173-big-Statistics.db
>> .
>> .
>> -rw-r--r-- 1 cassandra cassandra 340 Dec 15 20:10 mc-873-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 64 Dec 15 20:10 mc-873-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 15 20:10 mc-873-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 1910 Dec 15 20:10 mc-873-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 15 20:10 mc-873-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 51 Dec 15 20:10
>> mc-873-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 4793 Dec 15 20:10
>> mc-873-big-Statistics.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 15 20:10 mc-873-big-TOC.txt
>> .
>> .
>> .
>> -rw-r--r-- 1 cassandra cassandra 24 Dec 17 06:50 mc-1150-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 51 Dec 17 06:50 mc-1150-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 17 06:50 mc-1150-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 17 06:50
>> mc-1150-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 226 Dec 17 06:50 mc-1150-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 43 Dec 17 06:50
>> mc-1150-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 17 06:50 mc-1150-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 4691 Dec 17 06:50
>> mc-1150-big-Statistics.db
>>
>>
>>
>> check out sstable 873 is from dec 15. at this point i remove node no 3.
>> and the log shows streamed data is creating sstables 1150-1200. it also
>> shows one of these sstables being compacted with sstable 873, resulting in
>> sstable number 1228, an operation which from what i understand should not
>> happen.
>>
>> DEBUG [StreamReceiveTask:1] 2018-12-17 06:59:54,199
>> StreamReceiveTask.java:213 - [Stream #5ae370a0-01c9-11e9-ba69-d743ff231136]
>> Received 58 sstables from /10.106.12.34
>> ([BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1151-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1152-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1153-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1154-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1155-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1156-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1157-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1158-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1159-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1160-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1161-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1162-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1163-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1164-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1165-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1166-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1167-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1168-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1169-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1170-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1171-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1172-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1173-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1174-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1175-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1176-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1177-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1178-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1179-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1180-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1181-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1182-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1183-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1184-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1185-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1186-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1187-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1188-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1189-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1190-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1191-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1192-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1193-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1194-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1195-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1197-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1198-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1199-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1200-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1201-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0
>>
>>
>>
>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
>> TimeWindowCompactionStrategy.java:308 - bucket size 2 >= 2 and not in
>> current bucket, compacting what's here:
>> [BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db')]
>>
>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
>> CompactionTask.java:155 - Compacting (5b702040-01c9-11e9-ba69-d743ff231136)
>> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db:level=0,
>> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db:level=0,
>> ]
>>
>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,578
>> CompactionTask.java:255 - Compacted (5b702040-01c9-11e9-ba69-d743ff231136)
>> 2 sstables to
>> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big,]
>> to level=0. 3.397KiB to 3.383KiB (~99% of original) in 13ms. Read
>> Throughput = 244.120KiB/s, Write Throughput = 243.067KiB/s, Row Throughput
>> = ~184/s. 71 total partitions merged to 71. Partition merge counts were
>> {1:71, }
>>
>> see sstablemetadata output for sstable 873 and 1228. as you can see the
>> 1228 min timstamp is old.
>>
>> 873:
>> WARN 07:11:30,845 Only 45.238GiB free across all data volumes. Consider
>> adding more capacity to your cluster or removing obsolete snapshots
>> SSTable: /root/gil_sstables_before/mc-873-big
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Bloom Filter FP chance: 0.010000
>> Minimum timestamp: 1544896864561707
>> Maximum timestamp: 1544903882074190
>> SSTable min local deletion time: 2147483647
>> SSTable max local deletion time: 2147483647
>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
>> Compression ratio: 0.6954296160877513
>> TTL min: 0
>> TTL max: 0
>> First token: -7160136740246525330 (key=42)
>> Last token: 8209703677368372043 (key=34)
>> minClustringValues: [2018-12-15 18:01Z]
>> maxClustringValues: [2018-12-15 19:58Z]
>> Estimated droppable tombstones: 0.0
>> SSTable Level: 0
>> Repaired at: 0
>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
>> position=19800215)=CommitLogPosition(segmentId=1544706489001,
>> position=20586499)}
>> totalColumnsSet: 62
>> totalRows: 62
>> Estimated tombstone drop times:
>> Count Row Size Cell Count
>> 1 0 24
>> 2 0 7
>> 3 0 5
>> 4 0 1
>> 5 0 1
>> 6 0 0
>> 7 0 0
>> 8 0 0
>> 10 0 0
>> 12 0 0
>> 14 0 0
>> 17 0 0
>> 20 0 0
>> 24 0 0
>> 29 0 0
>> 35 0 0
>> 42 0 0
>> 50 7 0
>> 60 17 0
>> 72 0 0
>> 86 5 0
>> 103 2 0
>> 124 5 0
>> 149 1 0
>> 179 0 0
>> 215 1 0
>> 258 0 0
>> 310 0 0
>> 372 0 0
>> 446 0 0
>> 535 0 0
>> 642 0 0
>> 770 0 0
>> 924 0 0
>> 1109 0 0
>> 1331 0 0
>> 1597 0 0
>> 1916 0 0
>> 2299 0 0
>> 2759 0 0
>> 3311 0 0
>> 3973 0 0
>> 4768 0 0
>> 5722 0 0
>> 6866 0 0
>> 8239 0 0
>> 9887 0 0
>> 11864 0 0
>> 14237 0 0
>> 17084 0 0
>> 20501 0 0
>> 24601 0 0
>> 29521 0 0
>> 35425 0 0
>> 42510 0 0
>> 51012 0 0
>> 61214 0 0
>> 73457 0 0
>> 88148 0 0
>> 105778 0 0
>> 126934 0 0
>> 152321 0 0
>> 182785 0 0
>> 219342 0 0
>> 263210 0 0
>> 315852 0 0
>> 379022 0 0
>> 454826 0 0
>> 545791 0 0
>> 654949 0 0
>> 785939 0 0
>> 943127 0 0
>> 1131752 0 0
>> 1358102 0 0
>> 1629722 0 0
>> 1955666 0 0
>> 2346799 0 0
>> 2816159 0 0
>> 3379391 0 0
>> 4055269 0 0
>> 4866323 0 0
>> 5839588 0 0
>> 7007506 0 0
>> 8409007 0 0
>> 10090808 0 0
>> 12108970 0 0
>> 14530764 0 0
>> 17436917 0 0
>> 20924300 0 0
>> 25109160 0 0
>> 30130992 0 0
>> 36157190 0 0
>> 43388628 0 0
>> 52066354 0 0
>> 62479625 0 0
>> 74975550 0 0
>> 89970660 0 0
>> 107964792 0 0
>> 129557750 0 0
>> 155469300 0 0
>> 186563160 0 0
>> 223875792 0 0
>> 268650950 0 0
>> 322381140 0 0
>> 386857368 0 0
>> 464228842 0 0
>> 557074610 0 0
>> 668489532 0 0
>> 802187438 0 0
>> 962624926 0 0
>> 1155149911 0 0
>> 1386179893 0 0
>> 1663415872 0 0
>> 1996099046 0 0
>> 2395318855 0 0
>> 2874382626 0
>> 3449259151 0
>> 4139110981 0
>> 4966933177 0
>> 5960319812 0
>> 7152383774 0
>> 8582860529 0
>> 10299432635 0
>> 12359319162 0
>> 14831182994 0
>> 17797419593 0
>> 21356903512 0
>> 25628284214 0
>> 30753941057 0
>> 36904729268 0
>> 44285675122 0
>> 53142810146 0
>> 63771372175 0
>> 76525646610 0
>> 91830775932 0
>> 110196931118 0
>> 132236317342 0
>> 158683580810 0
>> 190420296972 0
>> 228504356366 0
>> 274205227639 0
>> 329046273167 0
>> 394855527800 0
>> 473826633360 0
>> 568591960032 0
>> 682310352038 0
>> 818772422446 0
>> 982526906935 0
>> 1179032288322 0
>> 1414838745986 0
>> Estimated cardinality: 38
>> EncodingStats minTTL: 0
>> EncodingStats minLocalDeletionTime: 1442880000
>> EncodingStats minTimestamp: 1544896864561707
>> KeyType: org.apache.cassandra.db.marshal.Int32Type
>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
>> StaticColumns: {}
>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>> You have new mail in /var/spool/mail/root
>>
>>
>>
>> 1228 :
>>
>> WARN 07:10:51,187 Only 45.238GiB free across all data volumes. Consider
>> adding more capacity to your cluster or removing obsolete snapshots
>> SSTable:
>> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Bloom Filter FP chance: 0.010000
>> Minimum timestamp: 1544896805142916
>> Maximum timestamp: 1544903882074190
>> SSTable min local deletion time: 2147483647
>> SSTable max local deletion time: 2147483647
>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
>> Compression ratio: 0.6942828485456369
>> TTL min: 0
>> TTL max: 0
>> First token: -9108684050423740263 (key=53)
>> Last token: 8669466869824630438 (key=77)
>> minClustringValues: [2018-12-15 18:00Z]
>> maxClustringValues: [2018-12-15 19:58Z]
>> Estimated droppable tombstones: 0.0
>> SSTable Level: 0
>> Repaired at: 0
>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
>> position=19800215)=CommitLogPosition(segmentId=1544706489001,
>> position=20586499)}
>> totalColumnsSet: 113
>> totalRows: 113
>> Estimated tombstone drop times:
>> Count Row Size Cell Count
>> 1 0 41
>> 2 0 21
>> 3 0 7
>> 4 0 1
>> 5 0 1
>> 6 0 0
>> 7 0 0
>> 8 0 0
>> 10 0 0
>> 12 0 0
>> 14 0 0
>> 17 0 0
>> 20 0 0
>> 24 0 0
>> 29 0 0
>> 35 0 0
>> 42 0 0
>> 50 16 0
>> 60 25 0
>> 72 0 0
>> 86 17 0
>> 103 4 0
>> 124 7 0
>> 149 1 0
>> 179 0 0
>> 215 1 0
>> 258 0 0
>> 310 0 0
>> 372 0 0
>> 446 0 0
>> 535 0 0
>> 642 0 0
>> 770 0 0
>> 924 0 0
>> 1109 0 0
>> 1331 0 0
>> 1597 0 0
>> 1916 0 0
>> 2299 0 0
>> 2759 0 0
>> 3311 0 0
>> 3973 0 0
>> 4768 0 0
>> 5722 0 0
>> 6866 0 0
>> 8239 0 0
>> 9887 0 0
>> 11864 0 0
>> 14237 0 0
>> 17084 0 0
>> 20501 0 0
>> 24601 0 0
>> 29521 0 0
>> 35425 0 0
>> 42510 0 0
>> 51012 0 0
>> 61214 0 0
>> 73457 0 0
>> 88148 0 0
>> 105778 0 0
>> 126934 0 0
>> 152321 0 0
>> 182785 0 0
>> 219342 0 0
>> 263210 0 0
>> 315852 0 0
>> 379022 0 0
>> 454826 0 0
>> 545791 0 0
>> 654949 0 0
>> 785939 0 0
>> 943127 0 0
>> 1131752 0 0
>> 1358102 0 0
>> 1629722 0 0
>> 1955666 0 0
>> 2346799 0 0
>> 2816159 0 0
>> 3379391 0 0
>> 4055269 0 0
>> 4866323 0 0
>> 5839588 0 0
>> 7007506 0 0
>> 8409007 0 0
>> 10090808 0 0
>> 12108970 0 0
>> 14530764 0 0
>> 17436917 0 0
>> 20924300 0 0
>> 25109160 0 0
>> 30130992 0 0
>> 36157190 0 0
>> 43388628 0 0
>> 52066354 0 0
>> 62479625 0 0
>> 74975550 0 0
>> 89970660 0 0
>> 107964792 0 0
>> 129557750 0 0
>> 155469300 0 0
>> 186563160 0 0
>> 223875792 0 0
>> 268650950 0 0
>> 322381140 0 0
>> 386857368 0 0
>> 464228842 0 0
>> 557074610 0 0
>> 668489532 0 0
>> 802187438 0 0
>> 962624926 0 0
>> 1155149911 0 0
>> 1386179893 0 0
>> 1663415872 0 0
>> 1996099046 0 0
>> 2395318855 0 0
>> 2874382626 0
>> 3449259151 0
>> 4139110981 0
>> 4966933177 0
>> 5960319812 0
>> 7152383774 0
>> 8582860529 0
>> 10299432635 0
>> 12359319162 0
>> 14831182994 0
>> 17797419593 0
>> 21356903512 0
>> 25628284214 0
>> 30753941057 0
>> 36904729268 0
>> 44285675122 0
>> 53142810146 0
>> 63771372175 0
>> 76525646610 0
>> 91830775932 0
>> 110196931118 0
>> 132236317342 0
>> 158683580810 0
>> 190420296972 0
>> 228504356366 0
>> 274205227639 0
>> 329046273167 0
>> 394855527800 0
>> 473826633360 0
>> 568591960032 0
>> 682310352038 0
>> 818772422446 0
>> 982526906935 0
>> 1179032288322 0
>> 1414838745986 0
>> Estimated cardinality: 71
>> EncodingStats minTTL: 0
>> EncodingStats minLocalDeletionTime: 1442880000
>> EncodingStats minTimestamp: 1544896805142916
>> KeyType: org.apache.cassandra.db.marshal.Int32Type
>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
>> StaticColumns: {}
>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>>
>> thanks
>> gil
>>
>>
>>>
>>> ---------- Forwarded message ---------
>>> From: Jeff Jirsa <jj...@gmail.com>
>>> Date: Fri, Dec 14, 2018 at 8:56 AM
>>> Subject: Re: TWCS sstables gets merged following node removal
>>> To: <us...@cassandra.apache.org>
>>>
>>>
>>> Remove node will stream data from all windows to remote nodes , so some
>>> compaction is expected
>>>
>>> Would need to see the sstablemetadata to understand what’s happening
>>> there.
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Dec 13, 2018, at 10:26 PM, Roy Burstein <bu...@gmail.com>
>>> wrote:
>>>
>>> Hi all ,
>>>
>>> My colleague opened Jira ticket for the issue but we are struggling with
>>> this issue for a while and we have space issues :
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-14929
>>>
>>>
>>> After removing a node from the cluster, a table that is defined as TWCS,
>>> has sstables from different time windows merged together, making old and
>>> new data sit in the same sstable(the full example is in the ticket).
>>>
>>> Any idea about this?workaround?Known issues?
>>>
>>> Thanks!
>>>
>>> Roy
>>>
>>>
Re: TWCS sstables gets merged following node removal
Posted by Gil Ganz <gi...@gmail.com>.
interesting jeff, thank you. ok so this is regarding new data merging with
old data, what about old sstables that were suddenly merged on many nodes
(as if i ran alter table to size tiered)? i do not have the sstables
themselfs now, but it is definitly something that happened, one day we had
sstables grouped by windows all working as planned, and a week later all
the sstables had timestamp of last couple of days, all this happened on
multiple tables configured with twcs. is there something you know that
might cause such a thing? if i understand correctly once i have an sstable
with a max timestamp tha is older then the defined window, it should never
be part of a compaction set.
dor - thanks, when is that version planned to be released?
gil
On Wed, Dec 19, 2018 at 8:38 PM Jeff Jirsa <jj...@gmail.com> wrote:
> Yes it can cause issues.
> No there's no way to disable it in any current release (I think it finally
> landed to disable it in 4.0, but dont have the JIRA handy)
>
> https://issues.apache.org/jira/browse/CASSANDRA-13418 was added to 3.11.1
> and higher to let people consciously say "ignore overlaps from read repair
> and just drop expired data when it's expired".
>
>
>
> On Wed, Dec 19, 2018 at 3:40 AM Gil Ganz <gi...@gmail.com> wrote:
>
>> sounds like the foreground read repair can cause issues to twcs (mix old
>> and new data in same sstable), is there a way to disable the foreground
>> read repair? is that indeed the case that it's problematic?
>>
>>
>>
>> On Mon, Dec 17, 2018 at 9:21 AM Gil Ganz <gi...@gmail.com> wrote:
>>
>>> hey jeff, attaching more information.
>>> so this the situation before - 3 nodes in the cluster (3.11.3 in this
>>> case but i saw same thing in 2.1 and 3.0), there is a script writing one
>>> row every minute and another script doing nodetool flush every 10 minute.
>>> window is defined as two hours, so after a few days this is how the
>>> directory listing looks :
>>>
>>> drwxr-xr-x 2 cassandra cassandra 4096 Dec 11 10:38 backups
>>> -rw-r--r-- 1 cassandra cassandra 646 Dec 12 05:25 mc-171-big-Index.db
>>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-171-big-Filter.db
>>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-171-big-Summary.db
>>> -rw-r--r-- 1 cassandra cassandra 3561 Dec 12 05:25 mc-171-big-Data.db
>>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25
>>> mc-171-big-Digest.crc32
>>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>>> mc-171-big-CompressionInfo.db
>>> -rw-r--r-- 1 cassandra cassandra 4893 Dec 12 05:25
>>> mc-171-big-Statistics.db
>>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-171-big-TOC.txt
>>> -rw-r--r-- 1 cassandra cassandra 565 Dec 12 05:25 mc-172-big-Index.db
>>> -rw-r--r-- 1 cassandra cassandra 96 Dec 12 05:25 mc-172-big-Filter.db
>>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-172-big-Summary.db
>>> -rw-r--r-- 1 cassandra cassandra 3475 Dec 12 05:25 mc-172-big-Data.db
>>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25
>>> mc-172-big-Digest.crc32
>>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>>> mc-172-big-CompressionInfo.db
>>> -rw-r--r-- 1 cassandra cassandra 4865 Dec 12 05:25
>>> mc-172-big-Statistics.db
>>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-172-big-TOC.txt
>>> -rw-r--r-- 1 cassandra cassandra 637 Dec 12 05:25 mc-173-big-Index.db
>>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-173-big-Filter.db
>>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-173-big-Summary.db
>>> -rw-r--r-- 1 cassandra cassandra 3678 Dec 12 05:25 mc-173-big-Data.db
>>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25
>>> mc-173-big-Digest.crc32
>>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>>> mc-173-big-CompressionInfo.db
>>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-173-big-TOC.txt
>>> -rw-r--r-- 1 cassandra cassandra 4888 Dec 12 05:25
>>> mc-173-big-Statistics.db
>>> .
>>> .
>>> -rw-r--r-- 1 cassandra cassandra 340 Dec 15 20:10 mc-873-big-Index.db
>>> -rw-r--r-- 1 cassandra cassandra 64 Dec 15 20:10 mc-873-big-Filter.db
>>> -rw-r--r-- 1 cassandra cassandra 56 Dec 15 20:10 mc-873-big-Summary.db
>>> -rw-r--r-- 1 cassandra cassandra 1910 Dec 15 20:10 mc-873-big-Data.db
>>> -rw-r--r-- 1 cassandra cassandra 10 Dec 15 20:10
>>> mc-873-big-Digest.crc32
>>> -rw-r--r-- 1 cassandra cassandra 51 Dec 15 20:10
>>> mc-873-big-CompressionInfo.db
>>> -rw-r--r-- 1 cassandra cassandra 4793 Dec 15 20:10
>>> mc-873-big-Statistics.db
>>> -rw-r--r-- 1 cassandra cassandra 92 Dec 15 20:10 mc-873-big-TOC.txt
>>> .
>>> .
>>> .
>>> -rw-r--r-- 1 cassandra cassandra 24 Dec 17 06:50 mc-1150-big-Filter.db
>>> -rw-r--r-- 1 cassandra cassandra 51 Dec 17 06:50 mc-1150-big-Index.db
>>> -rw-r--r-- 1 cassandra cassandra 56 Dec 17 06:50 mc-1150-big-Summary.db
>>> -rw-r--r-- 1 cassandra cassandra 10 Dec 17 06:50
>>> mc-1150-big-Digest.crc32
>>> -rw-r--r-- 1 cassandra cassandra 226 Dec 17 06:50 mc-1150-big-Data.db
>>> -rw-r--r-- 1 cassandra cassandra 43 Dec 17 06:50
>>> mc-1150-big-CompressionInfo.db
>>> -rw-r--r-- 1 cassandra cassandra 92 Dec 17 06:50 mc-1150-big-TOC.txt
>>> -rw-r--r-- 1 cassandra cassandra 4691 Dec 17 06:50
>>> mc-1150-big-Statistics.db
>>>
>>>
>>>
>>> check out sstable 873 is from dec 15. at this point i remove node no 3.
>>> and the log shows streamed data is creating sstables 1150-1200. it also
>>> shows one of these sstables being compacted with sstable 873, resulting in
>>> sstable number 1228, an operation which from what i understand should not
>>> happen.
>>>
>>> DEBUG [StreamReceiveTask:1] 2018-12-17 06:59:54,199
>>> StreamReceiveTask.java:213 - [Stream #5ae370a0-01c9-11e9-ba69-d743ff231136]
>>> Received 58 sstables from /10.106.12.34
>>> ([BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1151-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1152-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1153-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1154-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1155-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1156-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1157-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1158-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1159-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1160-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1161-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1162-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1163-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1164-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1165-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1166-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1167-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1168-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1169-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1170-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1171-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1172-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1173-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1174-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1175-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1176-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1177-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1178-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1179-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1180-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1181-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1182-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1183-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1184-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1185-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1186-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1187-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1188-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1189-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1190-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1191-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1192-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1193-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1194-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1195-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1197-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1198-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1199-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1200-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1201-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0
>>>
>>>
>>>
>>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
>>> TimeWindowCompactionStrategy.java:308 - bucket size 2 >= 2 and not in
>>> current bucket, compacting what's here:
>>> [BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db'),
>>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db')]
>>>
>>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
>>> CompactionTask.java:155 - Compacting (5b702040-01c9-11e9-ba69-d743ff231136)
>>> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db:level=0,
>>> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db:level=0,
>>> ]
>>>
>>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,578
>>> CompactionTask.java:255 - Compacted (5b702040-01c9-11e9-ba69-d743ff231136)
>>> 2 sstables to
>>> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big,]
>>> to level=0. 3.397KiB to 3.383KiB (~99% of original) in 13ms. Read
>>> Throughput = 244.120KiB/s, Write Throughput = 243.067KiB/s, Row Throughput
>>> = ~184/s. 71 total partitions merged to 71. Partition merge counts were
>>> {1:71, }
>>>
>>> see sstablemetadata output for sstable 873 and 1228. as you can see the
>>> 1228 min timstamp is old.
>>>
>>> 873:
>>> WARN 07:11:30,845 Only 45.238GiB free across all data volumes. Consider
>>> adding more capacity to your cluster or removing obsolete snapshots
>>> SSTable: /root/gil_sstables_before/mc-873-big
>>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>> Bloom Filter FP chance: 0.010000
>>> Minimum timestamp: 1544896864561707
>>> Maximum timestamp: 1544903882074190
>>> SSTable min local deletion time: 2147483647
>>> SSTable max local deletion time: 2147483647
>>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
>>> Compression ratio: 0.6954296160877513
>>> TTL min: 0
>>> TTL max: 0
>>> First token: -7160136740246525330 (key=42)
>>> Last token: 8209703677368372043 (key=34)
>>> minClustringValues: [2018-12-15 18:01Z]
>>> maxClustringValues: [2018-12-15 19:58Z]
>>> Estimated droppable tombstones: 0.0
>>> SSTable Level: 0
>>> Repaired at: 0
>>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
>>> position=19800215)=CommitLogPosition(segmentId=1544706489001,
>>> position=20586499)}
>>> totalColumnsSet: 62
>>> totalRows: 62
>>> Estimated tombstone drop times:
>>> Count Row Size Cell Count
>>> 1 0 24
>>> 2 0 7
>>> 3 0 5
>>> 4 0 1
>>> 5 0 1
>>> 6 0 0
>>> 7 0 0
>>> 8 0 0
>>> 10 0 0
>>> 12 0 0
>>> 14 0 0
>>> 17 0 0
>>> 20 0 0
>>> 24 0 0
>>> 29 0 0
>>> 35 0 0
>>> 42 0 0
>>> 50 7 0
>>> 60 17 0
>>> 72 0 0
>>> 86 5 0
>>> 103 2 0
>>> 124 5 0
>>> 149 1 0
>>> 179 0 0
>>> 215 1 0
>>> 258 0 0
>>> 310 0 0
>>> 372 0 0
>>> 446 0 0
>>> 535 0 0
>>> 642 0 0
>>> 770 0 0
>>> 924 0 0
>>> 1109 0 0
>>> 1331 0 0
>>> 1597 0 0
>>> 1916 0 0
>>> 2299 0 0
>>> 2759 0 0
>>> 3311 0 0
>>> 3973 0 0
>>> 4768 0 0
>>> 5722 0 0
>>> 6866 0 0
>>> 8239 0 0
>>> 9887 0 0
>>> 11864 0 0
>>> 14237 0 0
>>> 17084 0 0
>>> 20501 0 0
>>> 24601 0 0
>>> 29521 0 0
>>> 35425 0 0
>>> 42510 0 0
>>> 51012 0 0
>>> 61214 0 0
>>> 73457 0 0
>>> 88148 0 0
>>> 105778 0 0
>>> 126934 0 0
>>> 152321 0 0
>>> 182785 0 0
>>> 219342 0 0
>>> 263210 0 0
>>> 315852 0 0
>>> 379022 0 0
>>> 454826 0 0
>>> 545791 0 0
>>> 654949 0 0
>>> 785939 0 0
>>> 943127 0 0
>>> 1131752 0 0
>>> 1358102 0 0
>>> 1629722 0 0
>>> 1955666 0 0
>>> 2346799 0 0
>>> 2816159 0 0
>>> 3379391 0 0
>>> 4055269 0 0
>>> 4866323 0 0
>>> 5839588 0 0
>>> 7007506 0 0
>>> 8409007 0 0
>>> 10090808 0 0
>>> 12108970 0 0
>>> 14530764 0 0
>>> 17436917 0 0
>>> 20924300 0 0
>>> 25109160 0 0
>>> 30130992 0 0
>>> 36157190 0 0
>>> 43388628 0 0
>>> 52066354 0 0
>>> 62479625 0 0
>>> 74975550 0 0
>>> 89970660 0 0
>>> 107964792 0 0
>>> 129557750 0 0
>>> 155469300 0 0
>>> 186563160 0 0
>>> 223875792 0 0
>>> 268650950 0 0
>>> 322381140 0 0
>>> 386857368 0 0
>>> 464228842 0 0
>>> 557074610 0 0
>>> 668489532 0 0
>>> 802187438 0 0
>>> 962624926 0 0
>>> 1155149911 0 0
>>> 1386179893 0 0
>>> 1663415872 0 0
>>> 1996099046 0 0
>>> 2395318855 0 0
>>> 2874382626 0
>>> 3449259151 0
>>> 4139110981 0
>>> 4966933177 0
>>> 5960319812 0
>>> 7152383774 0
>>> 8582860529 0
>>> 10299432635 0
>>> 12359319162 0
>>> 14831182994 0
>>> 17797419593 0
>>> 21356903512 0
>>> 25628284214 0
>>> 30753941057 0
>>> 36904729268 0
>>> 44285675122 0
>>> 53142810146 0
>>> 63771372175 0
>>> 76525646610 0
>>> 91830775932 0
>>> 110196931118 0
>>> 132236317342 0
>>> 158683580810 0
>>> 190420296972 0
>>> 228504356366 0
>>> 274205227639 0
>>> 329046273167 0
>>> 394855527800 0
>>> 473826633360 0
>>> 568591960032 0
>>> 682310352038 0
>>> 818772422446 0
>>> 982526906935 0
>>> 1179032288322 0
>>> 1414838745986 0
>>> Estimated cardinality: 38
>>> EncodingStats minTTL: 0
>>> EncodingStats minLocalDeletionTime: 1442880000
>>> EncodingStats minTimestamp: 1544896864561707
>>> KeyType: org.apache.cassandra.db.marshal.Int32Type
>>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
>>> StaticColumns: {}
>>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>>> You have new mail in /var/spool/mail/root
>>>
>>>
>>>
>>> 1228 :
>>>
>>> WARN 07:10:51,187 Only 45.238GiB free across all data volumes. Consider
>>> adding more capacity to your cluster or removing obsolete snapshots
>>> SSTable:
>>> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big
>>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>> Bloom Filter FP chance: 0.010000
>>> Minimum timestamp: 1544896805142916
>>> Maximum timestamp: 1544903882074190
>>> SSTable min local deletion time: 2147483647
>>> SSTable max local deletion time: 2147483647
>>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
>>> Compression ratio: 0.6942828485456369
>>> TTL min: 0
>>> TTL max: 0
>>> First token: -9108684050423740263 (key=53)
>>> Last token: 8669466869824630438 (key=77)
>>> minClustringValues: [2018-12-15 18:00Z]
>>> maxClustringValues: [2018-12-15 19:58Z]
>>> Estimated droppable tombstones: 0.0
>>> SSTable Level: 0
>>> Repaired at: 0
>>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
>>> position=19800215)=CommitLogPosition(segmentId=1544706489001,
>>> position=20586499)}
>>> totalColumnsSet: 113
>>> totalRows: 113
>>> Estimated tombstone drop times:
>>> Count Row Size Cell Count
>>> 1 0 41
>>> 2 0 21
>>> 3 0 7
>>> 4 0 1
>>> 5 0 1
>>> 6 0 0
>>> 7 0 0
>>> 8 0 0
>>> 10 0 0
>>> 12 0 0
>>> 14 0 0
>>> 17 0 0
>>> 20 0 0
>>> 24 0 0
>>> 29 0 0
>>> 35 0 0
>>> 42 0 0
>>> 50 16 0
>>> 60 25 0
>>> 72 0 0
>>> 86 17 0
>>> 103 4 0
>>> 124 7 0
>>> 149 1 0
>>> 179 0 0
>>> 215 1 0
>>> 258 0 0
>>> 310 0 0
>>> 372 0 0
>>> 446 0 0
>>> 535 0 0
>>> 642 0 0
>>> 770 0 0
>>> 924 0 0
>>> 1109 0 0
>>> 1331 0 0
>>> 1597 0 0
>>> 1916 0 0
>>> 2299 0 0
>>> 2759 0 0
>>> 3311 0 0
>>> 3973 0 0
>>> 4768 0 0
>>> 5722 0 0
>>> 6866 0 0
>>> 8239 0 0
>>> 9887 0 0
>>> 11864 0 0
>>> 14237 0 0
>>> 17084 0 0
>>> 20501 0 0
>>> 24601 0 0
>>> 29521 0 0
>>> 35425 0 0
>>> 42510 0 0
>>> 51012 0 0
>>> 61214 0 0
>>> 73457 0 0
>>> 88148 0 0
>>> 105778 0 0
>>> 126934 0 0
>>> 152321 0 0
>>> 182785 0 0
>>> 219342 0 0
>>> 263210 0 0
>>> 315852 0 0
>>> 379022 0 0
>>> 454826 0 0
>>> 545791 0 0
>>> 654949 0 0
>>> 785939 0 0
>>> 943127 0 0
>>> 1131752 0 0
>>> 1358102 0 0
>>> 1629722 0 0
>>> 1955666 0 0
>>> 2346799 0 0
>>> 2816159 0 0
>>> 3379391 0 0
>>> 4055269 0 0
>>> 4866323 0 0
>>> 5839588 0 0
>>> 7007506 0 0
>>> 8409007 0 0
>>> 10090808 0 0
>>> 12108970 0 0
>>> 14530764 0 0
>>> 17436917 0 0
>>> 20924300 0 0
>>> 25109160 0 0
>>> 30130992 0 0
>>> 36157190 0 0
>>> 43388628 0 0
>>> 52066354 0 0
>>> 62479625 0 0
>>> 74975550 0 0
>>> 89970660 0 0
>>> 107964792 0 0
>>> 129557750 0 0
>>> 155469300 0 0
>>> 186563160 0 0
>>> 223875792 0 0
>>> 268650950 0 0
>>> 322381140 0 0
>>> 386857368 0 0
>>> 464228842 0 0
>>> 557074610 0 0
>>> 668489532 0 0
>>> 802187438 0 0
>>> 962624926 0 0
>>> 1155149911 0 0
>>> 1386179893 0 0
>>> 1663415872 0 0
>>> 1996099046 0 0
>>> 2395318855 0 0
>>> 2874382626 0
>>> 3449259151 0
>>> 4139110981 0
>>> 4966933177 0
>>> 5960319812 0
>>> 7152383774 0
>>> 8582860529 0
>>> 10299432635 0
>>> 12359319162 0
>>> 14831182994 0
>>> 17797419593 0
>>> 21356903512 0
>>> 25628284214 0
>>> 30753941057 0
>>> 36904729268 0
>>> 44285675122 0
>>> 53142810146 0
>>> 63771372175 0
>>> 76525646610 0
>>> 91830775932 0
>>> 110196931118 0
>>> 132236317342 0
>>> 158683580810 0
>>> 190420296972 0
>>> 228504356366 0
>>> 274205227639 0
>>> 329046273167 0
>>> 394855527800 0
>>> 473826633360 0
>>> 568591960032 0
>>> 682310352038 0
>>> 818772422446 0
>>> 982526906935 0
>>> 1179032288322 0
>>> 1414838745986 0
>>> Estimated cardinality: 71
>>> EncodingStats minTTL: 0
>>> EncodingStats minLocalDeletionTime: 1442880000
>>> EncodingStats minTimestamp: 1544896805142916
>>> KeyType: org.apache.cassandra.db.marshal.Int32Type
>>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
>>> StaticColumns: {}
>>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>>>
>>> thanks
>>> gil
>>>
>>>
>>>>
>>>> ---------- Forwarded message ---------
>>>> From: Jeff Jirsa <jj...@gmail.com>
>>>> Date: Fri, Dec 14, 2018 at 8:56 AM
>>>> Subject: Re: TWCS sstables gets merged following node removal
>>>> To: <us...@cassandra.apache.org>
>>>>
>>>>
>>>> Remove node will stream data from all windows to remote nodes , so some
>>>> compaction is expected
>>>>
>>>> Would need to see the sstablemetadata to understand what’s happening
>>>> there.
>>>>
>>>> --
>>>> Jeff Jirsa
>>>>
>>>>
>>>> On Dec 13, 2018, at 10:26 PM, Roy Burstein <bu...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi all ,
>>>>
>>>> My colleague opened Jira ticket for the issue but we are struggling
>>>> with this issue for a while and we have space issues :
>>>>
>>>> https://issues.apache.org/jira/browse/CASSANDRA-14929
>>>>
>>>>
>>>> After removing a node from the cluster, a table that is defined as
>>>> TWCS, has sstables from different time windows merged together, making old
>>>> and new data sit in the same sstable(the full example is in the ticket).
>>>>
>>>> Any idea about this?workaround?Known issues?
>>>>
>>>> Thanks!
>>>>
>>>> Roy
>>>>
>>>>
Re: TWCS sstables gets merged following node removal
Posted by Jeff Jirsa <jj...@gmail.com>.
Yes it can cause issues.
No there's no way to disable it in any current release (I think it finally
landed to disable it in 4.0, but dont have the JIRA handy)
https://issues.apache.org/jira/browse/CASSANDRA-13418 was added to 3.11.1
and higher to let people consciously say "ignore overlaps from read repair
and just drop expired data when it's expired".
On Wed, Dec 19, 2018 at 3:40 AM Gil Ganz <gi...@gmail.com> wrote:
> sounds like the foreground read repair can cause issues to twcs (mix old
> and new data in same sstable), is there a way to disable the foreground
> read repair? is that indeed the case that it's problematic?
>
>
>
> On Mon, Dec 17, 2018 at 9:21 AM Gil Ganz <gi...@gmail.com> wrote:
>
>> hey jeff, attaching more information.
>> so this the situation before - 3 nodes in the cluster (3.11.3 in this
>> case but i saw same thing in 2.1 and 3.0), there is a script writing one
>> row every minute and another script doing nodetool flush every 10 minute.
>> window is defined as two hours, so after a few days this is how the
>> directory listing looks :
>>
>> drwxr-xr-x 2 cassandra cassandra 4096 Dec 11 10:38 backups
>> -rw-r--r-- 1 cassandra cassandra 646 Dec 12 05:25 mc-171-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-171-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-171-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 3561 Dec 12 05:25 mc-171-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-171-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>> mc-171-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 4893 Dec 12 05:25
>> mc-171-big-Statistics.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-171-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 565 Dec 12 05:25 mc-172-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 96 Dec 12 05:25 mc-172-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-172-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 3475 Dec 12 05:25 mc-172-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-172-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>> mc-172-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 4865 Dec 12 05:25
>> mc-172-big-Statistics.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-172-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 637 Dec 12 05:25 mc-173-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-173-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-173-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 3678 Dec 12 05:25 mc-173-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-173-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
>> mc-173-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-173-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 4888 Dec 12 05:25
>> mc-173-big-Statistics.db
>> .
>> .
>> -rw-r--r-- 1 cassandra cassandra 340 Dec 15 20:10 mc-873-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 64 Dec 15 20:10 mc-873-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 15 20:10 mc-873-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 1910 Dec 15 20:10 mc-873-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 15 20:10 mc-873-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 51 Dec 15 20:10
>> mc-873-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 4793 Dec 15 20:10
>> mc-873-big-Statistics.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 15 20:10 mc-873-big-TOC.txt
>> .
>> .
>> .
>> -rw-r--r-- 1 cassandra cassandra 24 Dec 17 06:50 mc-1150-big-Filter.db
>> -rw-r--r-- 1 cassandra cassandra 51 Dec 17 06:50 mc-1150-big-Index.db
>> -rw-r--r-- 1 cassandra cassandra 56 Dec 17 06:50 mc-1150-big-Summary.db
>> -rw-r--r-- 1 cassandra cassandra 10 Dec 17 06:50
>> mc-1150-big-Digest.crc32
>> -rw-r--r-- 1 cassandra cassandra 226 Dec 17 06:50 mc-1150-big-Data.db
>> -rw-r--r-- 1 cassandra cassandra 43 Dec 17 06:50
>> mc-1150-big-CompressionInfo.db
>> -rw-r--r-- 1 cassandra cassandra 92 Dec 17 06:50 mc-1150-big-TOC.txt
>> -rw-r--r-- 1 cassandra cassandra 4691 Dec 17 06:50
>> mc-1150-big-Statistics.db
>>
>>
>>
>> check out sstable 873 is from dec 15. at this point i remove node no 3.
>> and the log shows streamed data is creating sstables 1150-1200. it also
>> shows one of these sstables being compacted with sstable 873, resulting in
>> sstable number 1228, an operation which from what i understand should not
>> happen.
>>
>> DEBUG [StreamReceiveTask:1] 2018-12-17 06:59:54,199
>> StreamReceiveTask.java:213 - [Stream #5ae370a0-01c9-11e9-ba69-d743ff231136]
>> Received 58 sstables from /10.106.12.34
>> ([BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1151-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1152-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1153-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1154-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1155-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1156-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1157-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1158-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1159-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1160-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1161-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1162-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1163-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1164-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1165-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1166-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1167-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1168-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1169-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1170-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1171-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1172-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1173-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1174-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1175-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1176-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1177-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1178-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1179-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1180-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1181-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1182-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1183-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1184-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1185-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1186-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1187-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1188-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1189-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1190-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1191-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1192-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1193-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1194-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1195-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1197-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1198-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1199-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1200-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1201-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0
>>
>>
>>
>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
>> TimeWindowCompactionStrategy.java:308 - bucket size 2 >= 2 and not in
>> current bucket, compacting what's here:
>> [BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db'),
>> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db')]
>>
>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
>> CompactionTask.java:155 - Compacting (5b702040-01c9-11e9-ba69-d743ff231136)
>> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db:level=0,
>> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db:level=0,
>> ]
>>
>> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,578
>> CompactionTask.java:255 - Compacted (5b702040-01c9-11e9-ba69-d743ff231136)
>> 2 sstables to
>> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big,]
>> to level=0. 3.397KiB to 3.383KiB (~99% of original) in 13ms. Read
>> Throughput = 244.120KiB/s, Write Throughput = 243.067KiB/s, Row Throughput
>> = ~184/s. 71 total partitions merged to 71. Partition merge counts were
>> {1:71, }
>>
>> see sstablemetadata output for sstable 873 and 1228. as you can see the
>> 1228 min timstamp is old.
>>
>> 873:
>> WARN 07:11:30,845 Only 45.238GiB free across all data volumes. Consider
>> adding more capacity to your cluster or removing obsolete snapshots
>> SSTable: /root/gil_sstables_before/mc-873-big
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Bloom Filter FP chance: 0.010000
>> Minimum timestamp: 1544896864561707
>> Maximum timestamp: 1544903882074190
>> SSTable min local deletion time: 2147483647
>> SSTable max local deletion time: 2147483647
>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
>> Compression ratio: 0.6954296160877513
>> TTL min: 0
>> TTL max: 0
>> First token: -7160136740246525330 (key=42)
>> Last token: 8209703677368372043 (key=34)
>> minClustringValues: [2018-12-15 18:01Z]
>> maxClustringValues: [2018-12-15 19:58Z]
>> Estimated droppable tombstones: 0.0
>> SSTable Level: 0
>> Repaired at: 0
>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
>> position=19800215)=CommitLogPosition(segmentId=1544706489001,
>> position=20586499)}
>> totalColumnsSet: 62
>> totalRows: 62
>> Estimated tombstone drop times:
>> Count Row Size Cell Count
>> 1 0 24
>> 2 0 7
>> 3 0 5
>> 4 0 1
>> 5 0 1
>> 6 0 0
>> 7 0 0
>> 8 0 0
>> 10 0 0
>> 12 0 0
>> 14 0 0
>> 17 0 0
>> 20 0 0
>> 24 0 0
>> 29 0 0
>> 35 0 0
>> 42 0 0
>> 50 7 0
>> 60 17 0
>> 72 0 0
>> 86 5 0
>> 103 2 0
>> 124 5 0
>> 149 1 0
>> 179 0 0
>> 215 1 0
>> 258 0 0
>> 310 0 0
>> 372 0 0
>> 446 0 0
>> 535 0 0
>> 642 0 0
>> 770 0 0
>> 924 0 0
>> 1109 0 0
>> 1331 0 0
>> 1597 0 0
>> 1916 0 0
>> 2299 0 0
>> 2759 0 0
>> 3311 0 0
>> 3973 0 0
>> 4768 0 0
>> 5722 0 0
>> 6866 0 0
>> 8239 0 0
>> 9887 0 0
>> 11864 0 0
>> 14237 0 0
>> 17084 0 0
>> 20501 0 0
>> 24601 0 0
>> 29521 0 0
>> 35425 0 0
>> 42510 0 0
>> 51012 0 0
>> 61214 0 0
>> 73457 0 0
>> 88148 0 0
>> 105778 0 0
>> 126934 0 0
>> 152321 0 0
>> 182785 0 0
>> 219342 0 0
>> 263210 0 0
>> 315852 0 0
>> 379022 0 0
>> 454826 0 0
>> 545791 0 0
>> 654949 0 0
>> 785939 0 0
>> 943127 0 0
>> 1131752 0 0
>> 1358102 0 0
>> 1629722 0 0
>> 1955666 0 0
>> 2346799 0 0
>> 2816159 0 0
>> 3379391 0 0
>> 4055269 0 0
>> 4866323 0 0
>> 5839588 0 0
>> 7007506 0 0
>> 8409007 0 0
>> 10090808 0 0
>> 12108970 0 0
>> 14530764 0 0
>> 17436917 0 0
>> 20924300 0 0
>> 25109160 0 0
>> 30130992 0 0
>> 36157190 0 0
>> 43388628 0 0
>> 52066354 0 0
>> 62479625 0 0
>> 74975550 0 0
>> 89970660 0 0
>> 107964792 0 0
>> 129557750 0 0
>> 155469300 0 0
>> 186563160 0 0
>> 223875792 0 0
>> 268650950 0 0
>> 322381140 0 0
>> 386857368 0 0
>> 464228842 0 0
>> 557074610 0 0
>> 668489532 0 0
>> 802187438 0 0
>> 962624926 0 0
>> 1155149911 0 0
>> 1386179893 0 0
>> 1663415872 0 0
>> 1996099046 0 0
>> 2395318855 0 0
>> 2874382626 0
>> 3449259151 0
>> 4139110981 0
>> 4966933177 0
>> 5960319812 0
>> 7152383774 0
>> 8582860529 0
>> 10299432635 0
>> 12359319162 0
>> 14831182994 0
>> 17797419593 0
>> 21356903512 0
>> 25628284214 0
>> 30753941057 0
>> 36904729268 0
>> 44285675122 0
>> 53142810146 0
>> 63771372175 0
>> 76525646610 0
>> 91830775932 0
>> 110196931118 0
>> 132236317342 0
>> 158683580810 0
>> 190420296972 0
>> 228504356366 0
>> 274205227639 0
>> 329046273167 0
>> 394855527800 0
>> 473826633360 0
>> 568591960032 0
>> 682310352038 0
>> 818772422446 0
>> 982526906935 0
>> 1179032288322 0
>> 1414838745986 0
>> Estimated cardinality: 38
>> EncodingStats minTTL: 0
>> EncodingStats minLocalDeletionTime: 1442880000
>> EncodingStats minTimestamp: 1544896864561707
>> KeyType: org.apache.cassandra.db.marshal.Int32Type
>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
>> StaticColumns: {}
>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>> You have new mail in /var/spool/mail/root
>>
>>
>>
>> 1228 :
>>
>> WARN 07:10:51,187 Only 45.238GiB free across all data volumes. Consider
>> adding more capacity to your cluster or removing obsolete snapshots
>> SSTable:
>> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Bloom Filter FP chance: 0.010000
>> Minimum timestamp: 1544896805142916
>> Maximum timestamp: 1544903882074190
>> SSTable min local deletion time: 2147483647
>> SSTable max local deletion time: 2147483647
>> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
>> Compression ratio: 0.6942828485456369
>> TTL min: 0
>> TTL max: 0
>> First token: -9108684050423740263 (key=53)
>> Last token: 8669466869824630438 (key=77)
>> minClustringValues: [2018-12-15 18:00Z]
>> maxClustringValues: [2018-12-15 19:58Z]
>> Estimated droppable tombstones: 0.0
>> SSTable Level: 0
>> Repaired at: 0
>> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
>> position=19800215)=CommitLogPosition(segmentId=1544706489001,
>> position=20586499)}
>> totalColumnsSet: 113
>> totalRows: 113
>> Estimated tombstone drop times:
>> Count Row Size Cell Count
>> 1 0 41
>> 2 0 21
>> 3 0 7
>> 4 0 1
>> 5 0 1
>> 6 0 0
>> 7 0 0
>> 8 0 0
>> 10 0 0
>> 12 0 0
>> 14 0 0
>> 17 0 0
>> 20 0 0
>> 24 0 0
>> 29 0 0
>> 35 0 0
>> 42 0 0
>> 50 16 0
>> 60 25 0
>> 72 0 0
>> 86 17 0
>> 103 4 0
>> 124 7 0
>> 149 1 0
>> 179 0 0
>> 215 1 0
>> 258 0 0
>> 310 0 0
>> 372 0 0
>> 446 0 0
>> 535 0 0
>> 642 0 0
>> 770 0 0
>> 924 0 0
>> 1109 0 0
>> 1331 0 0
>> 1597 0 0
>> 1916 0 0
>> 2299 0 0
>> 2759 0 0
>> 3311 0 0
>> 3973 0 0
>> 4768 0 0
>> 5722 0 0
>> 6866 0 0
>> 8239 0 0
>> 9887 0 0
>> 11864 0 0
>> 14237 0 0
>> 17084 0 0
>> 20501 0 0
>> 24601 0 0
>> 29521 0 0
>> 35425 0 0
>> 42510 0 0
>> 51012 0 0
>> 61214 0 0
>> 73457 0 0
>> 88148 0 0
>> 105778 0 0
>> 126934 0 0
>> 152321 0 0
>> 182785 0 0
>> 219342 0 0
>> 263210 0 0
>> 315852 0 0
>> 379022 0 0
>> 454826 0 0
>> 545791 0 0
>> 654949 0 0
>> 785939 0 0
>> 943127 0 0
>> 1131752 0 0
>> 1358102 0 0
>> 1629722 0 0
>> 1955666 0 0
>> 2346799 0 0
>> 2816159 0 0
>> 3379391 0 0
>> 4055269 0 0
>> 4866323 0 0
>> 5839588 0 0
>> 7007506 0 0
>> 8409007 0 0
>> 10090808 0 0
>> 12108970 0 0
>> 14530764 0 0
>> 17436917 0 0
>> 20924300 0 0
>> 25109160 0 0
>> 30130992 0 0
>> 36157190 0 0
>> 43388628 0 0
>> 52066354 0 0
>> 62479625 0 0
>> 74975550 0 0
>> 89970660 0 0
>> 107964792 0 0
>> 129557750 0 0
>> 155469300 0 0
>> 186563160 0 0
>> 223875792 0 0
>> 268650950 0 0
>> 322381140 0 0
>> 386857368 0 0
>> 464228842 0 0
>> 557074610 0 0
>> 668489532 0 0
>> 802187438 0 0
>> 962624926 0 0
>> 1155149911 0 0
>> 1386179893 0 0
>> 1663415872 0 0
>> 1996099046 0 0
>> 2395318855 0 0
>> 2874382626 0
>> 3449259151 0
>> 4139110981 0
>> 4966933177 0
>> 5960319812 0
>> 7152383774 0
>> 8582860529 0
>> 10299432635 0
>> 12359319162 0
>> 14831182994 0
>> 17797419593 0
>> 21356903512 0
>> 25628284214 0
>> 30753941057 0
>> 36904729268 0
>> 44285675122 0
>> 53142810146 0
>> 63771372175 0
>> 76525646610 0
>> 91830775932 0
>> 110196931118 0
>> 132236317342 0
>> 158683580810 0
>> 190420296972 0
>> 228504356366 0
>> 274205227639 0
>> 329046273167 0
>> 394855527800 0
>> 473826633360 0
>> 568591960032 0
>> 682310352038 0
>> 818772422446 0
>> 982526906935 0
>> 1179032288322 0
>> 1414838745986 0
>> Estimated cardinality: 71
>> EncodingStats minTTL: 0
>> EncodingStats minLocalDeletionTime: 1442880000
>> EncodingStats minTimestamp: 1544896805142916
>> KeyType: org.apache.cassandra.db.marshal.Int32Type
>> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
>> StaticColumns: {}
>> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>>
>> thanks
>> gil
>>
>>
>>>
>>> ---------- Forwarded message ---------
>>> From: Jeff Jirsa <jj...@gmail.com>
>>> Date: Fri, Dec 14, 2018 at 8:56 AM
>>> Subject: Re: TWCS sstables gets merged following node removal
>>> To: <us...@cassandra.apache.org>
>>>
>>>
>>> Remove node will stream data from all windows to remote nodes , so some
>>> compaction is expected
>>>
>>> Would need to see the sstablemetadata to understand what’s happening
>>> there.
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> On Dec 13, 2018, at 10:26 PM, Roy Burstein <bu...@gmail.com>
>>> wrote:
>>>
>>> Hi all ,
>>>
>>> My colleague opened Jira ticket for the issue but we are struggling with
>>> this issue for a while and we have space issues :
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-14929
>>>
>>>
>>> After removing a node from the cluster, a table that is defined as TWCS,
>>> has sstables from different time windows merged together, making old and
>>> new data sit in the same sstable(the full example is in the ticket).
>>>
>>> Any idea about this?workaround?Known issues?
>>>
>>> Thanks!
>>>
>>> Roy
>>>
>>>
Re: TWCS sstables gets merged following node removal
Posted by Gil Ganz <gi...@gmail.com>.
sounds like the foreground read repair can cause issues to twcs (mix old
and new data in same sstable), is there a way to disable the foreground
read repair? is that indeed the case that it's problematic?
On Mon, Dec 17, 2018 at 9:21 AM Gil Ganz <gi...@gmail.com> wrote:
> hey jeff, attaching more information.
> so this the situation before - 3 nodes in the cluster (3.11.3 in this case
> but i saw same thing in 2.1 and 3.0), there is a script writing one row
> every minute and another script doing nodetool flush every 10 minute.
> window is defined as two hours, so after a few days this is how the
> directory listing looks :
>
> drwxr-xr-x 2 cassandra cassandra 4096 Dec 11 10:38 backups
> -rw-r--r-- 1 cassandra cassandra 646 Dec 12 05:25 mc-171-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-171-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-171-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 3561 Dec 12 05:25 mc-171-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-171-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
> mc-171-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 4893 Dec 12 05:25 mc-171-big-Statistics.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-171-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 565 Dec 12 05:25 mc-172-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 96 Dec 12 05:25 mc-172-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-172-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 3475 Dec 12 05:25 mc-172-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-172-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
> mc-172-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 4865 Dec 12 05:25 mc-172-big-Statistics.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-172-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 637 Dec 12 05:25 mc-173-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 104 Dec 12 05:25 mc-173-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 12 05:25 mc-173-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 3678 Dec 12 05:25 mc-173-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 12 05:25 mc-173-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 59 Dec 12 05:25
> mc-173-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 12 05:25 mc-173-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 4888 Dec 12 05:25 mc-173-big-Statistics.db
> .
> .
> -rw-r--r-- 1 cassandra cassandra 340 Dec 15 20:10 mc-873-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 64 Dec 15 20:10 mc-873-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 15 20:10 mc-873-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 1910 Dec 15 20:10 mc-873-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 15 20:10 mc-873-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 51 Dec 15 20:10
> mc-873-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 4793 Dec 15 20:10 mc-873-big-Statistics.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 15 20:10 mc-873-big-TOC.txt
> .
> .
> .
> -rw-r--r-- 1 cassandra cassandra 24 Dec 17 06:50 mc-1150-big-Filter.db
> -rw-r--r-- 1 cassandra cassandra 51 Dec 17 06:50 mc-1150-big-Index.db
> -rw-r--r-- 1 cassandra cassandra 56 Dec 17 06:50 mc-1150-big-Summary.db
> -rw-r--r-- 1 cassandra cassandra 10 Dec 17 06:50 mc-1150-big-Digest.crc32
> -rw-r--r-- 1 cassandra cassandra 226 Dec 17 06:50 mc-1150-big-Data.db
> -rw-r--r-- 1 cassandra cassandra 43 Dec 17 06:50
> mc-1150-big-CompressionInfo.db
> -rw-r--r-- 1 cassandra cassandra 92 Dec 17 06:50 mc-1150-big-TOC.txt
> -rw-r--r-- 1 cassandra cassandra 4691 Dec 17 06:50
> mc-1150-big-Statistics.db
>
>
>
> check out sstable 873 is from dec 15. at this point i remove node no 3.
> and the log shows streamed data is creating sstables 1150-1200. it also
> shows one of these sstables being compacted with sstable 873, resulting in
> sstable number 1228, an operation which from what i understand should not
> happen.
>
> DEBUG [StreamReceiveTask:1] 2018-12-17 06:59:54,199
> StreamReceiveTask.java:213 - [Stream #5ae370a0-01c9-11e9-ba69-d743ff231136]
> Received 58 sstables from /10.106.12.34
> ([BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1151-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1152-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1153-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1154-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1155-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1156-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1157-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1158-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1159-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1160-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1161-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1162-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1163-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1164-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1165-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1166-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1167-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1168-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1169-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1170-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1171-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1172-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1173-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1174-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1175-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1176-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1177-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1178-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1179-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1180-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1181-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1182-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1183-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1184-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1185-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1186-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1187-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1188-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1189-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1190-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1191-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1192-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1193-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1194-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1195-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1197-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1198-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1199-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1200-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1201-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0
>
>
>
> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
> TimeWindowCompactionStrategy.java:308 - bucket size 2 >= 2 and not in
> current bucket, compacting what's here:
> [BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db'),
> BigTableReader(path='/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db')]
>
> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,564
> CompactionTask.java:155 - Compacting (5b702040-01c9-11e9-ba69-d743ff231136)
> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1196-big-Data.db:level=0,
> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-873-big-Data.db:level=0,
> ]
>
> DEBUG [CompactionExecutor:45] 2018-12-17 06:59:54,578
> CompactionTask.java:255 - Compacted (5b702040-01c9-11e9-ba69-d743ff231136)
> 2 sstables to
> [/var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big,]
> to level=0. 3.397KiB to 3.383KiB (~99% of original) in 13ms. Read
> Throughput = 244.120KiB/s, Write Throughput = 243.067KiB/s, Row Throughput
> = ~184/s. 71 total partitions merged to 71. Partition merge counts were
> {1:71, }
>
> see sstablemetadata output for sstable 873 and 1228. as you can see the
> 1228 min timstamp is old.
>
> 873:
> WARN 07:11:30,845 Only 45.238GiB free across all data volumes. Consider
> adding more capacity to your cluster or removing obsolete snapshots
> SSTable: /root/gil_sstables_before/mc-873-big
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Bloom Filter FP chance: 0.010000
> Minimum timestamp: 1544896864561707
> Maximum timestamp: 1544903882074190
> SSTable min local deletion time: 2147483647
> SSTable max local deletion time: 2147483647
> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
> Compression ratio: 0.6954296160877513
> TTL min: 0
> TTL max: 0
> First token: -7160136740246525330 (key=42)
> Last token: 8209703677368372043 (key=34)
> minClustringValues: [2018-12-15 18:01Z]
> maxClustringValues: [2018-12-15 19:58Z]
> Estimated droppable tombstones: 0.0
> SSTable Level: 0
> Repaired at: 0
> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
> position=19800215)=CommitLogPosition(segmentId=1544706489001,
> position=20586499)}
> totalColumnsSet: 62
> totalRows: 62
> Estimated tombstone drop times:
> Count Row Size Cell Count
> 1 0 24
> 2 0 7
> 3 0 5
> 4 0 1
> 5 0 1
> 6 0 0
> 7 0 0
> 8 0 0
> 10 0 0
> 12 0 0
> 14 0 0
> 17 0 0
> 20 0 0
> 24 0 0
> 29 0 0
> 35 0 0
> 42 0 0
> 50 7 0
> 60 17 0
> 72 0 0
> 86 5 0
> 103 2 0
> 124 5 0
> 149 1 0
> 179 0 0
> 215 1 0
> 258 0 0
> 310 0 0
> 372 0 0
> 446 0 0
> 535 0 0
> 642 0 0
> 770 0 0
> 924 0 0
> 1109 0 0
> 1331 0 0
> 1597 0 0
> 1916 0 0
> 2299 0 0
> 2759 0 0
> 3311 0 0
> 3973 0 0
> 4768 0 0
> 5722 0 0
> 6866 0 0
> 8239 0 0
> 9887 0 0
> 11864 0 0
> 14237 0 0
> 17084 0 0
> 20501 0 0
> 24601 0 0
> 29521 0 0
> 35425 0 0
> 42510 0 0
> 51012 0 0
> 61214 0 0
> 73457 0 0
> 88148 0 0
> 105778 0 0
> 126934 0 0
> 152321 0 0
> 182785 0 0
> 219342 0 0
> 263210 0 0
> 315852 0 0
> 379022 0 0
> 454826 0 0
> 545791 0 0
> 654949 0 0
> 785939 0 0
> 943127 0 0
> 1131752 0 0
> 1358102 0 0
> 1629722 0 0
> 1955666 0 0
> 2346799 0 0
> 2816159 0 0
> 3379391 0 0
> 4055269 0 0
> 4866323 0 0
> 5839588 0 0
> 7007506 0 0
> 8409007 0 0
> 10090808 0 0
> 12108970 0 0
> 14530764 0 0
> 17436917 0 0
> 20924300 0 0
> 25109160 0 0
> 30130992 0 0
> 36157190 0 0
> 43388628 0 0
> 52066354 0 0
> 62479625 0 0
> 74975550 0 0
> 89970660 0 0
> 107964792 0 0
> 129557750 0 0
> 155469300 0 0
> 186563160 0 0
> 223875792 0 0
> 268650950 0 0
> 322381140 0 0
> 386857368 0 0
> 464228842 0 0
> 557074610 0 0
> 668489532 0 0
> 802187438 0 0
> 962624926 0 0
> 1155149911 0 0
> 1386179893 0 0
> 1663415872 0 0
> 1996099046 0 0
> 2395318855 0 0
> 2874382626 0
> 3449259151 0
> 4139110981 0
> 4966933177 0
> 5960319812 0
> 7152383774 0
> 8582860529 0
> 10299432635 0
> 12359319162 0
> 14831182994 0
> 17797419593 0
> 21356903512 0
> 25628284214 0
> 30753941057 0
> 36904729268 0
> 44285675122 0
> 53142810146 0
> 63771372175 0
> 76525646610 0
> 91830775932 0
> 110196931118 0
> 132236317342 0
> 158683580810 0
> 190420296972 0
> 228504356366 0
> 274205227639 0
> 329046273167 0
> 394855527800 0
> 473826633360 0
> 568591960032 0
> 682310352038 0
> 818772422446 0
> 982526906935 0
> 1179032288322 0
> 1414838745986 0
> Estimated cardinality: 38
> EncodingStats minTTL: 0
> EncodingStats minLocalDeletionTime: 1442880000
> EncodingStats minTimestamp: 1544896864561707
> KeyType: org.apache.cassandra.db.marshal.Int32Type
> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
> StaticColumns: {}
> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
> You have new mail in /var/spool/mail/root
>
>
>
> 1228 :
>
> WARN 07:10:51,187 Only 45.238GiB free across all data volumes. Consider
> adding more capacity to your cluster or removing obsolete snapshots
> SSTable:
> /var/lib/cassandra/data/gil_test/my_test-e97d9860fd3011e8ad4d61ca1e38b0ef/mc-1228-big
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Bloom Filter FP chance: 0.010000
> Minimum timestamp: 1544896805142916
> Maximum timestamp: 1544903882074190
> SSTable min local deletion time: 2147483647
> SSTable max local deletion time: 2147483647
> Compressor: org.apache.cassandra.io.compress.LZ4Compressor
> Compression ratio: 0.6942828485456369
> TTL min: 0
> TTL max: 0
> First token: -9108684050423740263 (key=53)
> Last token: 8669466869824630438 (key=77)
> minClustringValues: [2018-12-15 18:00Z]
> maxClustringValues: [2018-12-15 19:58Z]
> Estimated droppable tombstones: 0.0
> SSTable Level: 0
> Repaired at: 0
> Replay positions covered: {CommitLogPosition(segmentId=1544706489001,
> position=19800215)=CommitLogPosition(segmentId=1544706489001,
> position=20586499)}
> totalColumnsSet: 113
> totalRows: 113
> Estimated tombstone drop times:
> Count Row Size Cell Count
> 1 0 41
> 2 0 21
> 3 0 7
> 4 0 1
> 5 0 1
> 6 0 0
> 7 0 0
> 8 0 0
> 10 0 0
> 12 0 0
> 14 0 0
> 17 0 0
> 20 0 0
> 24 0 0
> 29 0 0
> 35 0 0
> 42 0 0
> 50 16 0
> 60 25 0
> 72 0 0
> 86 17 0
> 103 4 0
> 124 7 0
> 149 1 0
> 179 0 0
> 215 1 0
> 258 0 0
> 310 0 0
> 372 0 0
> 446 0 0
> 535 0 0
> 642 0 0
> 770 0 0
> 924 0 0
> 1109 0 0
> 1331 0 0
> 1597 0 0
> 1916 0 0
> 2299 0 0
> 2759 0 0
> 3311 0 0
> 3973 0 0
> 4768 0 0
> 5722 0 0
> 6866 0 0
> 8239 0 0
> 9887 0 0
> 11864 0 0
> 14237 0 0
> 17084 0 0
> 20501 0 0
> 24601 0 0
> 29521 0 0
> 35425 0 0
> 42510 0 0
> 51012 0 0
> 61214 0 0
> 73457 0 0
> 88148 0 0
> 105778 0 0
> 126934 0 0
> 152321 0 0
> 182785 0 0
> 219342 0 0
> 263210 0 0
> 315852 0 0
> 379022 0 0
> 454826 0 0
> 545791 0 0
> 654949 0 0
> 785939 0 0
> 943127 0 0
> 1131752 0 0
> 1358102 0 0
> 1629722 0 0
> 1955666 0 0
> 2346799 0 0
> 2816159 0 0
> 3379391 0 0
> 4055269 0 0
> 4866323 0 0
> 5839588 0 0
> 7007506 0 0
> 8409007 0 0
> 10090808 0 0
> 12108970 0 0
> 14530764 0 0
> 17436917 0 0
> 20924300 0 0
> 25109160 0 0
> 30130992 0 0
> 36157190 0 0
> 43388628 0 0
> 52066354 0 0
> 62479625 0 0
> 74975550 0 0
> 89970660 0 0
> 107964792 0 0
> 129557750 0 0
> 155469300 0 0
> 186563160 0 0
> 223875792 0 0
> 268650950 0 0
> 322381140 0 0
> 386857368 0 0
> 464228842 0 0
> 557074610 0 0
> 668489532 0 0
> 802187438 0 0
> 962624926 0 0
> 1155149911 0 0
> 1386179893 0 0
> 1663415872 0 0
> 1996099046 0 0
> 2395318855 0 0
> 2874382626 0
> 3449259151 0
> 4139110981 0
> 4966933177 0
> 5960319812 0
> 7152383774 0
> 8582860529 0
> 10299432635 0
> 12359319162 0
> 14831182994 0
> 17797419593 0
> 21356903512 0
> 25628284214 0
> 30753941057 0
> 36904729268 0
> 44285675122 0
> 53142810146 0
> 63771372175 0
> 76525646610 0
> 91830775932 0
> 110196931118 0
> 132236317342 0
> 158683580810 0
> 190420296972 0
> 228504356366 0
> 274205227639 0
> 329046273167 0
> 394855527800 0
> 473826633360 0
> 568591960032 0
> 682310352038 0
> 818772422446 0
> 982526906935 0
> 1179032288322 0
> 1414838745986 0
> Estimated cardinality: 71
> EncodingStats minTTL: 0
> EncodingStats minLocalDeletionTime: 1442880000
> EncodingStats minTimestamp: 1544896805142916
> KeyType: org.apache.cassandra.db.marshal.Int32Type
> ClusteringTypes: [org.apache.cassandra.db.marshal.TimestampType]
> StaticColumns: {}
> RegularColumns: {name:org.apache.cassandra.db.marshal.UTF8Type}
>
> thanks
> gil
>
>
>>
>> ---------- Forwarded message ---------
>> From: Jeff Jirsa <jj...@gmail.com>
>> Date: Fri, Dec 14, 2018 at 8:56 AM
>> Subject: Re: TWCS sstables gets merged following node removal
>> To: <us...@cassandra.apache.org>
>>
>>
>> Remove node will stream data from all windows to remote nodes , so some
>> compaction is expected
>>
>> Would need to see the sstablemetadata to understand what’s happening
>> there.
>>
>> --
>> Jeff Jirsa
>>
>>
>> On Dec 13, 2018, at 10:26 PM, Roy Burstein <bu...@gmail.com>
>> wrote:
>>
>> Hi all ,
>>
>> My colleague opened Jira ticket for the issue but we are struggling with
>> this issue for a while and we have space issues :
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-14929
>>
>>
>> After removing a node from the cluster, a table that is defined as TWCS,
>> has sstables from different time windows merged together, making old and
>> new data sit in the same sstable(the full example is in the ticket).
>>
>> Any idea about this?workaround?Known issues?
>>
>> Thanks!
>>
>> Roy
>>
>>
Re: TWCS sstables gets merged following node removal
Posted by Jeff Jirsa <jj...@gmail.com>.
Remove node will stream data from all windows to remote nodes , so some compaction is expected
Would need to see the sstablemetadata to understand what’s happening there.
--
Jeff Jirsa
> On Dec 13, 2018, at 10:26 PM, Roy Burstein <bu...@gmail.com> wrote:
>
> Hi all ,
> My colleague opened Jira ticket for the issue but we are struggling with this issue for a while and we have space issues :
>
> https://issues.apache.org/jira/browse/CASSANDRA-14929
>
> After removing a node from the cluster, a table that is defined as TWCS, has sstables from different time windows merged together, making old and new data sit in the same sstable(the full example is in the ticket).
> Any idea about this?workaround?Known issues?
> Thanks!
> Roy