You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Graham Sanderson <gr...@vast.com> on 2016/01/01 03:09:27 UTC
Re: cassandra full gc too often
If you are lucky that might mask the real issue, but I doubt it… that is an insane number of compaction tasks and indicative of another problem. I would check release notes of 2.0.6+, if I recall that was not a stable version and may have had leaks.
Aside from that, just FYI, if you use native_objects for memtables you can run CMS fine even up to largish (20-30G) heap sizes without long GC pauses (at least over months) . That said look for promotion failures of 131074 DWORDS. Not sure what your GC logging options are, but what you posted doesn’t say why a full GC happened which would be totally tunable, but as I say I don’t think GC is actually your problem
> On Dec 31, 2015, at 10:16 AM, Ipremyadav <ip...@gmail.com> wrote:
>
> Simplest option is to use java 8 with G1 gc.
>
> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <chenatu2006@gmail.com <ma...@gmail.com>> wrote:
>
>> I have a cassandra 2.0.6 cluster with four nodes as backup database. The only operation is posting data into db. Recently, the full gc of the nodes increases apparently and blocks cluster operation.
>>
>> The load of each node is 10G. The heap is 8G each with default jvm memory settings. The cpu is 24 cores. The settings of cassandra.yaml are almost default.
>>
>> For debug, all the clients are disconnected, however the gc is still high.
>>
>> The GC is too often that it appears in every minute and blocks for nealy 30s.
>>
>> How to debug this situation
>>
>> The following is the snippets of gc log:
>>
>> before full gc histogram
>>
>>
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs]
>> Before GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> Before GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> ----------------------------------------------
>> 1: 61030245 1952967840 java.util.concurrent.FutureTask
>> 2: 61031398 1464753552 java.util.concurrent.Executors$RunnableAdapter
>> 3: 61030312 1464727488 java.util.concurrent.LinkedBlockingQueue$Node
>> 4: 61030244 1464725856 org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
>> 5: 214181 13386992 [B
>> 6: 202890 9738720 java.nio.HeapByteBuffer
>> 7: 37622 5994808 [C
>> 8: 42163 5722096 <constMethodKlass>
>> 9: 42163 5407152 <methodKlass>
>> 10: 4170 4648240 <constantPoolKlass>
>> 11: 100060 4002400 org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
>> 12: 4170 2860816 <instanceKlassKlass>
>> 13: 3664 2702720 <constantPoolCacheKlass>
>> 14: 4817 2701576 [J
>> 15: 2575 1056600 <methodDataKlass>
>> 16: 37431 898344 java.lang.String
>> 17: 2585 627352 [Ljava.lang.Object;
>> 18: 17728 567296 java.util.concurrent.ConcurrentHashMap$HashEntry
>> 19: 22111 530664 javax.management.ObjectName$Property
>> 20: 7595 463824 [S
>> 21: 4534 433560 java.lang.Class
>> 22: 11581 370592 java.util.HashMap$Entry
>>
>> ...
>> Total 289942930 8399196504
>> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs]
>> CMS: Large block 0x00000007f7da9588
>>
>> After GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 6335823
>> Max Chunk Size: 6335823
>> Number of Blocks: 1
>> Av. Block Size: 6335823
>> Tree Height: 1
>> After GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>>
>>
>> It seems related to objects of Futuretask.
>>
>> --
>> 陈硕 Shuo Chen
>> chenatu2006@gmail.com <ma...@gmail.com>
>> chenshuo@whaty.com <ma...@whaty.com>
Re: cassandra full gc too often
Posted by Shuo Chen <ch...@gmail.com>.
The following is the snippets of nodetool compactions history. It shows
many records relates to keyspace system.
Compaction History:
id keyspace_name
columnfamily_name compacted_at bytes_in
bytes_out rows_merged
8e4f8830-b04f-11e5-a211-45b7aa88107c system
sstable_activity 1451629144115 3342 915
{4:23}
96a6fcb0-b04b-11e5-a211-45b7aa88107c system hints
1451627440123 18970740 18970740 {1:1}
7c42c940-adac-11e5-8bd4-45b7aa88107c system hints
1451339203540 56969835 56782732 {2:3}
585b97a0-ad98-11e5-8bd4-45b7aa88107c system
sstable_activity 1451330553370 3700 956
{4:24}
aefc3f10-b1b2-11e5-a211-45b7aa88107c system
sstable_activity 1451781670273 3201 906
{4:23}
1e76f1b0-b180-11e5-a211-45b7aa88107c system
sstable_activity 1451759952971 3303 700
{4:23}
e7b75b70-aec2-11e5-8bd4-45b7aa88107c system hints
1451458783911 57690316 57497847 {2:3}
ad102280-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451532129448
45671877 41137664 {1:11, 3:1, 4:8}
60906970-aec7-11e5-8bd4-45b7aa88107c system
sstable_activity 1451460704647 3751 974
{4:25}
88aed310-ad91-11e5-8bd4-45b7aa88107c system hints
1451327627969 56984347 56765328 {2:3}
3ad14f00-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531937776
46696097 38827028 {1:8, 3:2, 4:9}
84df8fb0-b00f-11e5-a211-45b7aa88107c system hints
1451601640491 18970740 18970740 {1:1}
657482e0-ad33-11e5-8bd4-45b7aa88107c system
sstable_activity 1451287196174 3701 931
{4:24}
9cc8af70-b24a-11e5-a211-45b7aa88107c system
sstable_activity 1451846923239 3134 773
{4:23}
dcbe5e30-afd0-11e5-a211-45b7aa88107c system
sstable_activity 1451574729619 3357 790
{4:23}
b285ced0-afa0-11e5-84e3-45b7aa88107c system hints
1451554042941 43310718 42137761 {1:1,
2:2}
119770e0-ad4e-11e5-8bd4-45b7aa88107c system hints
1451298651886 57397441 57190519 {2:3}
f1bb37a0-b204-11e5-a211-45b7aa88107c system hints
1451817000986 17713746 17566349 {1:1}
dcd0c720-b0b4-11e5-a211-45b7aa88107c system
sstable_activity 1451672654993 3189 797
{4:23}
cadb2320-ae94-11e5-8bd4-45b7aa88107c system
sstable_activity 1451438978642 3642 959
{4:25}
321a84d0-af63-11e5-8bd4-45b7aa88107c system hints
1451527628189 57407115 57179555 {1:3}
5d760ca0-af20-11e5-8bd4-45b7aa88107c system hints
1451498924650 57405307 57234936 {1:3}
2e59a0b0-af6d-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531916859 464 190
{1:1, 2:1}
8723a290-adba-11e5-8bd4-45b7aa88107c system hints
1451345234745 57065795 56814910 {2:3}
add0a8d0-af6c-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531701213 464 42
{2:2}
04c0d2f0-af6d-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531847071 446 0
{2:2}
315dd170-af98-11e5-b1dc-45b7aa88107c system schema_columns
1451550390279 129142 45841 {2:1,
3:1, 4:9}
a17d8f80-b24b-11e5-a211-45b7aa88107c system hints
1451847360632 16746317 16600978 {1:1}
e22e3b90-ad39-11e5-8bd4-45b7aa88107c system hints
1451289982409 57240038 56984895 {2:3}
38591c10-b22b-11e5-a211-45b7aa88107c system hints
1451833440337 17351684 17002773 {1:1}
fc274930-af6c-11e5-b1dc-45b7aa88107c system hints
1451531832643 63540861 59716559 {2:3}
aba057f0-b1b4-11e5-a211-45b7aa88107c system hints
1451782523631 18743329 18577220 {1:1}
81d5e980-af5c-11e5-8bd4-45b7aa88107c system hints
1451524755480 57603966 57407115 {2:3}
a1801510-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451532110049
46735894 40839394 {1:11, 4:9}
f7e02540-b1bf-11e5-a211-45b7aa88107c system hints
1451787376020 18577220 18403648 {1:1}
3e9a72b0-af72-11e5-b1dc-45b7aa88107c system hints
1451534091611 60455761 43314155 {2:1,
3:2}
30c103c0-af6d-11e5-b1dc-45b7aa88107c system peers
1451531920892 1396 567 {2:3,
3:1}
65f47770-aed7-11e5-8bd4-45b7aa88107c system hints
1451467585639 57780382 57445632 {2:3}
ecabc460-afa6-11e5-a211-45b7aa88107c system hints
1451556717478 29776619 29776619 {1:2}
5da4df30-afac-11e5-a211-45b7aa88107c system hints
1451559054499 20309429 18970740 {1:1,
2:1}
f5ce2d30-afa6-11e5-a211-45b7aa88107c system hints
1451556732803 29776619 29776619 {1:2}
4ada6610-aeec-11e5-8bd4-45b7aa88107c system hints
1451476559601 57459788 57221330 {1:3}
74ffd7f0-ae32-11e5-8bd4-45b7aa88107c system hints
1451396743919 57210071 57053060 {2:3}
621f05e0-b0ce-11e5-a211-45b7aa88107c system
sstable_activity 1451683616062 3060 719
{4:23}
5ec0e5b0-af2c-11e5-8bd4-45b7aa88107c system
sstable_activity 1451504080779 3756 971
{4:25}
92707090-b09b-11e5-a211-45b7aa88107c system
sstable_activity 1451661792793 3149 689
{4:23}
c35b51f0-ae40-11e5-8bd4-45b7aa88107c system hints
1451402888335 57190905 57037697 {2:3}
2a813e10-ade4-11e5-8bd4-45b7aa88107c system
sstable_activity 1451363118193 3772 960
{4:24}
c99593a0-af6c-11e5-b1dc-45b7aa88107c system hints
1451531747802 57645358 57613272 {3:3}
50002a00-aee5-11e5-8bd4-45b7aa88107c system hints
1451473561760 57670199 57459788 {2:3}
c2cf2570-ae6a-11e5-8bd4-45b7aa88107c system hints
1451420926279 57375776 57227330 {2:3}
1b5e7bc0-ad1a-11e5-8bd4-45b7aa88107c system
sstable_activity 1451276334460 3606 861
{4:24}
bf2f8d90-adc7-11e5-8bd4-45b7aa88107c system hints
1451350912233 56901257 56731829 {2:3}
55e055e0-b263-11e5-a211-45b7aa88107c system hints
1451857541694 16600978 16364535 {1:1}
1a60f830-afa7-11e5-a211-45b7aa88107c system hints
1451556794163 29776619 29776619 {1:2}
7b193990-adfd-11e5-8bd4-45b7aa88107c system
sstable_activity 1451373990825 3679 939
{4:24}
6bf27720-aec9-11e5-8bd4-45b7aa88107c system hints
1451461582737 57571883 57372615 {2:3}
d68f67b0-af13-11e5-8bd4-45b7aa88107c system hints
1451493544363 57518024 57335312 {2:3}
7946c2a0-b162-11e5-a211-45b7aa88107c system hints
1451747220426 18970740 18970740 {1:1}
1b686990-ad98-11e5-8bd4-45b7aa88107c system hints
1451330451113 56988235 56854912 {2:3}
30883420-afa7-11e5-a211-45b7aa88107c system hints
1451556831330 29776619 29776619 {1:2}
e5651290-b129-11e5-a211-45b7aa88107c system hints
1451722920505 18970740 18970740 {1:1}
feedabd0-ad61-11e5-8bd4-45b7aa88107c system hints
1451307210509 57251764 57133776 {2:3}
d75a36b0-afa0-11e5-84e3-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451554104731
47893133 41298592 {1:10, 2:1, 4:9}
b2226820-b05b-11e5-a211-45b7aa88107c system hints
1451634358178 18970740 18970740 {1:1}
7c7e0d80-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451532047960
47584630 40254518 {1:10, 2:1, 4:9}
7d39a5d0-ad7f-11e5-8bd4-45b7aa88107c system
sstable_activity 1451319877805 3637 882
{4:24}
c05d07c0-b1fe-11e5-a211-45b7aa88107c system
sstable_activity 1451814341180 3329 852
{4:23}
a227fdb0-add4-11e5-8bd4-45b7aa88107c system hints
1451356446987 57070333 56869637 {2:3}
26a79a60-aeb5-11e5-8bd4-45b7aa88107c system hints
1451452876550 57637242 57408618 {2:3}
def14230-af6c-11e5-b1dc-45b7aa88107c system hints
1451531783635 57613272 57622817 {1:3}
ce8c20b0-afa6-11e5-a211-45b7aa88107c system local
1451556666939 1167 696 {4:1}
75ace380-b1e5-11e5-a211-45b7aa88107c system
sstable_activity 1451803478456 3171 820
{4:23}
88d05020-af86-11e5-b1dc-45b7aa88107c system hints
1451542806050 43921794 43937401 {2:3}
0a59ef20-b1f8-11e5-a211-45b7aa88107c system hints
1451811458834 17972574 17713746 {1:1}
7ace0730-afa7-11e5-a211-45b7aa88107c system hints
1451556955939 29776619 29776619 {1:2}
9e1ee8b0-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451532104379
5935114 2171959 {4:9}
0580cea0-ae5c-11e5-8bd4-45b7aa88107c system hints
1451414595722 57388662 57179650 {2:3}
378110c0-afa8-11e5-a211-45b7aa88107c system hints
1451557272524 28568485 24640663 {1:1,
2:1}
ef231ab0-af5e-11e5-8bd4-45b7aa88107c system
sstable_activity 1451525797851 3739 973
{4:25}
301fc6c0-ae25-11e5-8bd4-45b7aa88107c system hints
1451391044908 56996239 56821882 {2:3}
5ceccc10-afa7-11e5-a211-45b7aa88107c system hints
1451556905809 29776619 29776619 {1:2}
44713fe0-afa7-11e5-a211-45b7aa88107c system hints
1451556864734 29776619 29776619 {1:2}
0b0786b0-afa7-11e5-a211-45b7aa88107c system hints
1451556768411 29776619 29776619 {1:2}
5a667200-ae78-11e5-8bd4-45b7aa88107c system hints
1451426764064 57497152 57334697 {2:3}
d4306ba0-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531765594
43985647 32914710 {1:7, 3:1, 4:9}
7596f1a0-ad9f-11e5-8bd4-45b7aa88107c system hints
1451333608890 56854912 56608995 {1:3}
e31fc540-adee-11e5-8bd4-45b7aa88107c system hints
1451367722900 57057877 56867011 {2:3}
52458b10-b1e3-11e5-a211-45b7aa88107c system hints
1451802560065 18087954 17972574 {1:1}
278bf030-af13-11e5-8bd4-45b7aa88107c system
sstable_activity 1451493250739 3720 922
{4:25}
76443290-af55-11e5-8bd4-45b7aa88107c system hints
1451521729593 57500226 57294613 {2:3}
8065b450-af33-11e5-8bd4-45b7aa88107c system hints
1451507143701 57557563 57359219 {2:3}
ae29e180-b0ce-11e5-a211-45b7aa88107c system hints
1451683743640 18970740 18970740 {1:1}
d942a360-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531774102
8615536 4447310 {1:1, 3:1, 4:9}
6b039a10-b289-11e5-a211-45b7aa88107c system hints
1451873898033 15967755 15796743 {1:1}
3afa2ba0-b14d-11e5-a211-45b7aa88107c system
sstable_activity 1451738096474 3196 778
{4:23}
3d6e34a0-afa7-11e5-a211-45b7aa88107c system hints
1451556852970 29776619 29776619 {1:2}
b27e7970-af6c-11e5-b1dc-45b7aa88107c system local
1451531709063 1215 668 {4:1}
78715730-ae62-11e5-8bd4-45b7aa88107c system
sstable_activity 1451417365539 3749 950
{4:25}
46bd9b90-aed0-11e5-8bd4-45b7aa88107c system hints
1451464526793 57589209 57452596 {2:3}
d9030250-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_SCO 1451531773685
61554578 59278964 {1:76826, 2:11126, 3:604, 4:9}
c70ceb50-afcc-11e5-a211-45b7aa88107c system hints
1451572975237 18970740 18970740 {1:1}
69f8a090-b106-11e5-a211-45b7aa88107c system hints
1451707681048 18970740 18970740 {1:1}
7dd087b0-af79-11e5-b1dc-45b7aa88107c system hints
1451537204139 43314155 43324175 {1:3}
94404a50-b166-11e5-a211-45b7aa88107c system
sstable_activity 1451748983669 3364 926
{4:23}
d99598d0-ad10-11e5-8bd4-45b7aa88107c system hints
1451272358621 57162919 56981333 {2:3}
7fe03f20-b23d-11e5-a211-45b7aa88107c system hints
1451841291282 17002773 16746317 {1:1}
e664d240-b1cc-11e5-a211-45b7aa88107c system hints
1451792930148 18403648 18252171 {1:1}
04178e40-afa7-11e5-a211-45b7aa88107c system hints
1451556756772 29776619 29776619 {1:2}
1a414650-ada6-11e5-8bd4-45b7aa88107c system hints
1451336462133 56826857 56710683 {2:3}
ad9ef5f0-b026-11e5-a211-45b7aa88107c system hints
1451611587279 18970740 18970740 {1:1}
60a4e320-ad17-11e5-8bd4-45b7aa88107c system hints
1451275162194 57236552 57076664 {2:3}
917e6370-aeae-11e5-8bd4-45b7aa88107c system hints
1451450049319 57539934 57384675 {2:3}
eb1440d0-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531803997
43867391 36299816 {1:8, 4:10}
a20bfc70-aee0-11e5-8bd4-45b7aa88107c system
sstable_activity 1451471551927 3833 945
{4:25}
47cbe960-aef3-11e5-8bd4-45b7aa88107c system hints
1451479560950 57443969 57255886 {2:3}
e2a84f00-af25-11e5-8bd4-45b7aa88107c system hints
1451501295600 57449985 57298217 {2:3}
af1628a0-ade1-11e5-8bd4-45b7aa88107c system hints
1451362052138 57001606 56839416 {2:3}
b0fea4a0-ae16-11e5-8bd4-45b7aa88107c system
sstable_activity 1451384818666 3645 872
{3:1, 4:24}
15473980-aeae-11e5-8bd4-45b7aa88107c system
sstable_activity 1451449840920 3789 975
{4:25}
e5bdc7c0-afa6-11e5-a211-45b7aa88107c system hints
1451556705852 29776619 29776619 {1:2}
d7808360-afa0-11e5-84e3-45b7aa88107c system
compactions_in_progress 1451554104982 461 42
{2:2}
36a22720-ae63-11e5-8bd4-45b7aa88107c system hints
1451417684626 57282895 57148294 {2:3}
99222cc0-ad54-11e5-8bd4-45b7aa88107c system hints
1451301456268 57258179 57086571 {2:3}
a3480190-adb1-11e5-8bd4-45b7aa88107c system
sstable_activity 1451341416489 3683 925
{4:24}
1198f0c0-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531868620
8937482 3569558 {2:1, 3:1, 4:8}
422d8470-b036-11e5-a211-45b7aa88107c system
sstable_activity 1451618278967 3243 825
{4:23}
12aee840-b218-11e5-a211-45b7aa88107c system
sstable_activity 1451825216708 3176 819
{4:23}
85af2f30-adb3-11e5-8bd4-45b7aa88107c system hints
1451342225827 57028608 56811880 {2:3}
8b98e730-afa0-11e5-84e3-45b7aa88107c system hints
1451553977634 88625803 44359037 {4:2,
5:1}
0c2b9aa0-ae02-11e5-8bd4-45b7aa88107c system hints
1451375952202 56891360 56548629 {2:3}
ec9afff0-af00-11e5-8bd4-45b7aa88107c system hints
1451485420911 57594351 57326414 {2:3}
98a23cb0-b199-11e5-a211-45b7aa88107c system hints
1451770895355 18970740 18906200 {1:1}
8f90dde0-afb7-11e5-a211-45b7aa88107c system
sstable_activity 1451563862718 3508 827
{1:2, 2:8, 3:2, 4:19}
822b7da0-b14b-11e5-a211-45b7aa88107c system hints
1451737356922 18970740 18970740 {1:1}
f5dc9cb0-ae54-11e5-8bd4-45b7aa88107c system hints
1451411563003 57341080 57162518 {2:3}
83efd290-af8d-11e5-b1dc-45b7aa88107c system hints
1451545804345 43996474 44000895 {2:3}
ba28f830-af80-11e5-b1dc-45b7aa88107c system hints
1451540311859 43744401 43725750 {1:1,
2:2}
decb8560-afa6-11e5-a211-45b7aa88107c system hints
1451556694198 29776619 29776619 {1:2}
8fb80860-ad5b-11e5-8bd4-45b7aa88107c system hints
1451304446950 57324867 56998191 {2:3}
969514e0-afa7-11e5-a211-45b7aa88107c system hints
1451557002542 29776619 29776619 {1:2}
bf7d00d0-afb5-11e5-a211-45b7aa88107c system hints
1451563084125 18970740 18970740 {1:1}
113e2660-af48-11e5-8bd4-45b7aa88107c system hints
1451515976646 57413867 57219902 {2:3}
32ee9f00-adf5-11e5-8bd4-45b7aa88107c system hints
1451370433776 57034296 56618944 {2:3}
3c583550-afea-11e5-a211-45b7aa88107c system
sstable_activity 1451585627429 3073 720
{4:23}
58437cd0-af3a-11e5-8bd4-45b7aa88107c system hints
1451510082845 57369449 57188682 {2:3}
871b8c30-b0ef-11e5-a211-45b7aa88107c system hints
1451697851507 18970740 18970740 {1:1}
00522050-ae7f-11e5-8bd4-45b7aa88107c system hints
1451429619413 57450075 57258800 {2:3}
14e3fd10-af6d-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531874145 472 42
{2:2}
b3e23af0-b0e7-11e5-a211-45b7aa88107c system
sstable_activity 1451694490655 3013 726
{4:23}
aa5986d0-ad24-11e5-8bd4-45b7aa88107c system hints
1451280869309 57181046 56983904 {2:3}
29697550-aebc-11e5-8bd4-45b7aa88107c system hints
1451455887653 57617579 57451553 {2:3}
8201d200-ae8c-11e5-8bd4-45b7aa88107c system hints
1451435420448 57558970 57354959 {2:3}
45bc2770-addb-11e5-8bd4-45b7aa88107c system hints
1451359298407 56977263 56817508 {2:3}
ebd29a70-affe-11e5-a211-45b7aa88107c system hints
1451594511767 18970740 18970740 {1:1}
9530ef00-ad4c-11e5-8bd4-45b7aa88107c system
sstable_activity 1451298013680 3580 885
{4:24}
caa48640-afa6-11e5-a211-45b7aa88107c system local
1451556660388 1013 677 {4:1}
acee31c0-aea0-11e5-8bd4-45b7aa88107c system hints
1451444082396 57659639 57460800 {2:3}
aa2ad760-ae71-11e5-8bd4-45b7aa88107c system hints
1451423891414 57468574 57242109 {2:3}
9e29cbd0-b036-11e5-a211-45b7aa88107c system hints
1451618433293 18970740 18970740 {1:1}
df7b80d0-af6c-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531784541 542 172
{1:2, 2:2}
48b11430-ad7d-11e5-8bd4-45b7aa88107c system hints
1451318930675 56911864 56738503 {2:3}
6afb39b0-b199-11e5-a211-45b7aa88107c system
sstable_activity 1451770818763 3039 788
{4:23}
f77313f0-af40-11e5-8bd4-45b7aa88107c system hints
1451512926895 57410690 57197897 {2:3}
38c86a20-adc1-11e5-8bd4-45b7aa88107c system hints
1451348109762 56814910 56689825 {1:3}
c2406070-af0e-11e5-8bd4-45b7aa88107c system hints
1451491362807 57391505 57316604 {2:3}
610527f0-b26f-11e5-a211-45b7aa88107c system hints
1451862714351 16364535 16082982 {1:1}
23d18f10-ae49-11e5-8bd4-45b7aa88107c system
sstable_activity 1451406486145 3587 941
{4:25}
76303090-ad40-11e5-8bd4-45b7aa88107c system hints
1451292807705 57224902 57033897 {2:3}
787be5c0-ad76-11e5-8bd4-45b7aa88107c system hints
1451316004380 56853811 56717047 {2:3}
f446f990-aef9-11e5-8bd4-45b7aa88107c system hints
1451482427305 57466666 57373339 {2:3}
3ba17b30-ad47-11e5-8bd4-45b7aa88107c system hints
1451295715939 57320671 57178557 {2:3}
85f76e20-afa8-11e5-a211-45b7aa88107c system hints
1451557404162 24767833 20275337 {1:1,
2:1}
464b1550-af6d-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531957029 464 42
{2:2}
d39fbc30-af9f-11e5-b1dc-45b7aa88107c system
sstable_activity 1451553668979 3895 835
{4:26}
acf90bb0-afa7-11e5-a211-45b7aa88107c system hints
1451557040107 29214113 28457489 {1:1,
2:1}
d09de8e0-ad65-11e5-8bd4-45b7aa88107c system
sstable_activity 1451308850798 3685 899
{4:24}
6e371d30-b27d-11e5-a211-45b7aa88107c system hints
1451868749443 16082982 15967755 {1:1}
fca1eea0-ad83-11e5-8bd4-45b7aa88107c system hints
1451321809546 56925900 56697130 {2:3}
31372130-b0f9-11e5-a211-45b7aa88107c system hints
1451702002371 18970740 18970740 {1:1}
a0dd4c90-af45-11e5-8bd4-45b7aa88107c system
sstable_activity 1451514929113 3785 882
{4:25}
a180b150-af6d-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451532110053 446 0
{2:2}
46239020-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531956770
45648060 39245044 {1:8, 2:1, 3:1, 4:9}
76cf6680-afa0-11e5-84e3-45b7aa88107c system local
1451553942760 1265 674 {4:1}
e4c92910-b11b-11e5-a211-45b7aa88107c system hints
1451716906529 18970740 18970740 {1:1}
16e97320-b264-11e5-a211-45b7aa88107c system
sstable_activity 1451857865554 3114 726
{4:23}
fddd18f0-b100-11e5-a211-45b7aa88107c system
sstable_activity 1451705352191 2959 716
{4:23}
ea955a00-b173-11e5-a211-45b7aa88107c system hints
1451754711968 18970740 18970740 {1:1}
52156430-adce-11e5-8bd4-45b7aa88107c system hints
1451353735667 56989435 56824386 {2:3}
b9d73cf0-ae47-11e5-8bd4-45b7aa88107c system hints
1451405878847 57223358 57041982 {2:3}
2ab91b60-afaf-11e5-a211-45b7aa88107c system schema_triggers
1451560257558 134 60 {1:1,
3:1}
9712f8c0-ae39-11e5-8bd4-45b7aa88107c system hints
1451399807564 57261044 57081560 {2:3}
86ec5560-adfb-11e5-8bd4-45b7aa88107c system hints
1451373151670 56802348 56679018 {2:3}
4b9ffb20-b27d-11e5-a211-45b7aa88107c system
sstable_activity 1451868691410 3037 791
{4:23}
e0ed0e70-aef9-11e5-8bd4-45b7aa88107c system
sstable_activity 1451482394839 3749 941
{4:25}
28e78950-afa7-11e5-a211-45b7aa88107c system hints
1451556818533 29776619 29776619 {1:2}
47589f00-b231-11e5-a211-45b7aa88107c system
sstable_activity 1451836042480 3212 785
{4:23}
499c5d30-afaf-11e5-a211-45b7aa88107c system peers
1451560309379 1810 556 {3:4}
2d335080-af6a-11e5-8bd4-45b7aa88107c system hints
1451530626440 57406002 57261820 {2:3}
d46ed430-af6c-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531766003 608 268
{1:2, 2:1}
31cbd580-af98-11e5-b1dc-45b7aa88107c system local
1451550391000 905 741 {4:1}
95dec160-afa0-11e5-84e3-45b7aa88107c system hints
1451553994870 44359037 44359037 {1:3}
4726d640-afe4-11e5-a211-45b7aa88107c system hints
1451583068580 18970740 18970740 {1:1}
9ed33380-ae7b-11e5-8bd4-45b7aa88107c system
sstable_activity 1451428167352 3638 906
{4:25}
50fa9650-b1a8-11e5-a211-45b7aa88107c system hints
1451777217589 18906200 18743329 {1:1}
b66e4420-b183-11e5-a211-45b7aa88107c system hints
1451761496418 18970740 18970740 {1:1}
5a003890-ade8-11e5-8bd4-45b7aa88107c system hints
1451364915865 57066041 56952696 {2:3}
451f4710-ad1e-11e5-8bd4-45b7aa88107c system hints
1451278122497 57133634 56933373 {2:3}
b1e973b0-ad6f-11e5-8bd4-45b7aa88107c system hints
1451313094251 56935662 56616319 {1:3}
0ce9d500-ae17-11e5-8bd4-45b7aa88107c system hints
1451384972880 56788738 56649325 {2:3}
0fc24530-ae1e-11e5-8bd4-45b7aa88107c system hints
1451387984131 56905778 56758301 {2:3}
31d54e10-b0da-11e5-a211-45b7aa88107c system hints
1451688689009 18970740 18970740 {1:1}
cb322de0-b133-11e5-a211-45b7aa88107c system
sstable_activity 1451727171518 3177 862
{4:23}
4f987470-af1a-11e5-8bd4-45b7aa88107c system hints
1451496324407 57567325 57405307 {2:3}
bbad5970-af2c-11e5-8bd4-45b7aa88107c system hints
1451504236679 57519774 57367465 {2:3}
5e31ef40-aff4-11e5-a211-45b7aa88107c system hints
1451589979188 18970740
On Mon, Jan 4, 2016 at 9:54 AM, Shuo Chen <ch...@gmail.com> wrote:
> The attachment is the current system.log and config of the cassandra
> cluster. Over 8000 full gc happens in 3 days with 121 young gc.
>
> There are client operation in these days. Besides most columnfamily in the
> cluster are supercolumnfamily created by cassandra-cli. Most rows have
> average 30 sub-rows and each sub-row has 20 columns.
>
> You said that there are many compaction tasks. So how to check the source
> of this compactions? Thank you!
>
> On Fri, Jan 1, 2016 at 10:09 AM, Graham Sanderson <gr...@vast.com> wrote:
>
>> If you are lucky that might mask the real issue, but I doubt it… that is
>> an insane number of compaction tasks and indicative of another problem. I
>> would check release notes of 2.0.6+, if I recall that was not a stable
>> version and may have had leaks.
>>
>> Aside from that, just FYI, if you use native_objects for memtables you
>> can run CMS fine even up to largish (20-30G) heap sizes without long GC
>> pauses (at least over months) . That said look for promotion failures of
>> 131074 DWORDS. Not sure what your GC logging options are, but what you
>> posted doesn’t say why a full GC happened which would be totally tunable,
>> but as I say I don’t think GC is actually your problem
>>
>>
>> On Dec 31, 2015, at 10:16 AM, Ipremyadav <ip...@gmail.com> wrote:
>>
>> Simplest option is to use java 8 with G1 gc.
>>
>> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <ch...@gmail.com> wrote:
>>
>> I have a cassandra 2.0.6 cluster with four nodes as backup database. The
>> only operation is posting data into db. Recently, the full gc of the nodes
>> increases apparently and blocks cluster operation.
>>
>> The load of each node is 10G. The heap is 8G each with default jvm memory
>> settings. The cpu is 24 cores. The settings of cassandra.yaml are almost
>> default.
>>
>> For debug, all the clients are disconnected, however the gc is still high.
>>
>> The GC is too often that it appears in every minute and blocks for nealy
>> 30s.
>>
>> How to debug this situation
>>
>> The following is the snippets of gc log:
>>
>> before full gc histogram
>>
>>
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs]
>> Before GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> Before GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> ----------------------------------------------
>> 1: 61030245 1952967840 java.util.concurrent.FutureTask
>> 2: 61031398 1464753552
>> java.util.concurrent.Executors$RunnableAdapter
>> 3: 61030312 1464727488
>> java.util.concurrent.LinkedBlockingQueue$Node
>> 4: 61030244 1464725856
>> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
>> 5: 214181 13386992 [B
>> 6: 202890 9738720 java.nio.HeapByteBuffer
>> 7: 37622 5994808 [C
>> 8: 42163 5722096 <constMethodKlass>
>> 9: 42163 5407152 <methodKlass>
>> 10: 4170 4648240 <constantPoolKlass>
>> 11: 100060 4002400
>> org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
>> 12: 4170 2860816 <instanceKlassKlass>
>> 13: 3664 2702720 <constantPoolCacheKlass>
>> 14: 4817 2701576 [J
>> 15: 2575 1056600 <methodDataKlass>
>> 16: 37431 898344 java.lang.String
>> 17: 2585 627352 [Ljava.lang.Object;
>> 18: 17728 567296
>> java.util.concurrent.ConcurrentHashMap$HashEntry
>> 19: 22111 530664 javax.management.ObjectName$Property
>> 20: 7595 463824 [S
>> 21: 4534 433560 java.lang.Class
>> 22: 11581 370592 java.util.HashMap$Entry
>>
>> ...
>> Total 289942930 8399196504
>> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs]
>> CMS: Large block 0x00000007f7da9588
>>
>> After GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 6335823
>> Max Chunk Size: 6335823
>> Number of Blocks: 1
>> Av. Block Size: 6335823
>> Tree Height: 1
>> After GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>>
>>
>> It seems related to objects of Futuretask.
>>
>> --
>> *陈硕* *Shuo Chen*
>> chenatu2006@gmail.com
>> chenshuo@whaty.com
>>
>>
>>
>
>
> --
> *陈硕* *Shuo Chen*
> chenatu2006@gmail.com
> chenshuo@whaty.com
>
--
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com
Re: cassandra full gc too often
Posted by Robert Coli <rc...@eventbrite.com>.
On Sun, Jan 3, 2016 at 5:54 PM, Shuo Chen <ch...@gmail.com> wrote:
> There are client operation in these days. Besides most columnfamily in the
> cluster are supercolumnfamily created by cassandra-cli. Most rows have
> average 30 sub-rows and each sub-row has 20 columns.
>
Supercolumns, especially pre-CQL implementation, have serious performance
and memory concerns.
In general, one should not use them.
=Rob
Re: cassandra full gc too often
Posted by Shuo Chen <ch...@gmail.com>.
The attachment is the current system.log and config of the cassandra
cluster. Over 8000 full gc happens in 3 days with 121 young gc.
There are client operation in these days. Besides most columnfamily in the
cluster are supercolumnfamily created by cassandra-cli. Most rows have
average 30 sub-rows and each sub-row has 20 columns.
You said that there are many compaction tasks. So how to check the source
of this compactions? Thank you!
On Fri, Jan 1, 2016 at 10:09 AM, Graham Sanderson <gr...@vast.com> wrote:
> If you are lucky that might mask the real issue, but I doubt it… that is
> an insane number of compaction tasks and indicative of another problem. I
> would check release notes of 2.0.6+, if I recall that was not a stable
> version and may have had leaks.
>
> Aside from that, just FYI, if you use native_objects for memtables you can
> run CMS fine even up to largish (20-30G) heap sizes without long GC pauses
> (at least over months) . That said look for promotion failures of 131074
> DWORDS. Not sure what your GC logging options are, but what you posted
> doesn’t say why a full GC happened which would be totally tunable, but as I
> say I don’t think GC is actually your problem
>
>
> On Dec 31, 2015, at 10:16 AM, Ipremyadav <ip...@gmail.com> wrote:
>
> Simplest option is to use java 8 with G1 gc.
>
> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <ch...@gmail.com> wrote:
>
> I have a cassandra 2.0.6 cluster with four nodes as backup database. The
> only operation is posting data into db. Recently, the full gc of the nodes
> increases apparently and blocks cluster operation.
>
> The load of each node is 10G. The heap is 8G each with default jvm memory
> settings. The cpu is 24 cores. The settings of cassandra.yaml are almost
> default.
>
> For debug, all the clients are disconnected, however the gc is still high.
>
> The GC is too often that it appears in every minute and blocks for nealy
> 30s.
>
> How to debug this situation
>
> The following is the snippets of gc log:
>
> before full gc histogram
>
>
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs]
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
> ----------------------------------------------
> 1: 61030245 1952967840 java.util.concurrent.FutureTask
> 2: 61031398 1464753552
> java.util.concurrent.Executors$RunnableAdapter
> 3: 61030312 1464727488
> java.util.concurrent.LinkedBlockingQueue$Node
> 4: 61030244 1464725856
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
> 5: 214181 13386992 [B
> 6: 202890 9738720 java.nio.HeapByteBuffer
> 7: 37622 5994808 [C
> 8: 42163 5722096 <constMethodKlass>
> 9: 42163 5407152 <methodKlass>
> 10: 4170 4648240 <constantPoolKlass>
> 11: 100060 4002400
> org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
> 12: 4170 2860816 <instanceKlassKlass>
> 13: 3664 2702720 <constantPoolCacheKlass>
> 14: 4817 2701576 [J
> 15: 2575 1056600 <methodDataKlass>
> 16: 37431 898344 java.lang.String
> 17: 2585 627352 [Ljava.lang.Object;
> 18: 17728 567296
> java.util.concurrent.ConcurrentHashMap$HashEntry
> 19: 22111 530664 javax.management.ObjectName$Property
> 20: 7595 463824 [S
> 21: 4534 433560 java.lang.Class
> 22: 11581 370592 java.util.HashMap$Entry
>
> ...
> Total 289942930 8399196504
> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs]
> CMS: Large block 0x00000007f7da9588
>
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 6335823
> Max Chunk Size: 6335823
> Number of Blocks: 1
> Av. Block Size: 6335823
> Tree Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
>
>
> It seems related to objects of Futuretask.
>
> --
> *陈硕* *Shuo Chen*
> chenatu2006@gmail.com
> chenshuo@whaty.com
>
>
>
--
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com