You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Shuo Chen <ch...@gmail.com> on 2015/12/31 11:23:14 UTC
cassandra full gc too often
I have a cassandra 2.0.6 cluster with four nodes as backup database. The
only operation is posting data into db. Recently, the full gc of the nodes
increases apparently and blocks cluster operation.
The load of each node is 10G. The heap is 8G each with default jvm memory
settings. The cpu is 24 cores. The settings of cassandra.yaml are almost
default.
For debug, all the clients are disconnected, however the gc is still high.
The GC is too often that it appears in every minute and blocks for nealy
30s.
How to debug this situation
The following is the snippets of gc log:
before full gc histogram
Total Free Space: 0
Max Chunk Size: 0
Number of Blocks: 0
Tree Height: 0
1620.992: [GC 7377342K(8178944K), 0.1380260 secs]
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max Chunk Size: 0
Number of Blocks: 0
Tree Height: 0
Before GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max Chunk Size: 0
Number of Blocks: 0
Tree Height: 0
----------------------------------------------
1: 61030245 1952967840 java.util.concurrent.FutureTask
2: 61031398 1464753552
java.util.concurrent.Executors$RunnableAdapter
3: 61030312 1464727488
java.util.concurrent.LinkedBlockingQueue$Node
4: 61030244 1464725856
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
5: 214181 13386992 [B
6: 202890 9738720 java.nio.HeapByteBuffer
7: 37622 5994808 [C
8: 42163 5722096 <constMethodKlass>
9: 42163 5407152 <methodKlass>
10: 4170 4648240 <constantPoolKlass>
11: 100060 4002400
org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
12: 4170 2860816 <instanceKlassKlass>
13: 3664 2702720 <constantPoolCacheKlass>
14: 4817 2701576 [J
15: 2575 1056600 <methodDataKlass>
16: 37431 898344 java.lang.String
17: 2585 627352 [Ljava.lang.Object;
18: 17728 567296
java.util.concurrent.ConcurrentHashMap$HashEntry
19: 22111 530664 javax.management.ObjectName$Property
20: 7595 463824 [S
21: 4534 433560 java.lang.Class
22: 11581 370592 java.util.HashMap$Entry
...
Total 289942930 8399196504
1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs]
CMS: Large block 0x00000007f7da9588
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 6335823
Max Chunk Size: 6335823
Number of Blocks: 1
Av. Block Size: 6335823
Tree Height: 1
After GC:
Statistics for BinaryTreeDictionary:
------------------------------------
Total Free Space: 0
Max Chunk Size: 0
Number of Blocks: 0
Tree Height: 0
It seems related to objects of Futuretask.
--
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com
Re: cassandra full gc too often
Posted by Shuo Chen <ch...@gmail.com>.
The following is the snippets of nodetool compactions history. It shows
many records relates to keyspace system.
Compaction History:
id keyspace_name
columnfamily_name compacted_at bytes_in
bytes_out rows_merged
8e4f8830-b04f-11e5-a211-45b7aa88107c system
sstable_activity 1451629144115 3342 915
{4:23}
96a6fcb0-b04b-11e5-a211-45b7aa88107c system hints
1451627440123 18970740 18970740 {1:1}
7c42c940-adac-11e5-8bd4-45b7aa88107c system hints
1451339203540 56969835 56782732 {2:3}
585b97a0-ad98-11e5-8bd4-45b7aa88107c system
sstable_activity 1451330553370 3700 956
{4:24}
aefc3f10-b1b2-11e5-a211-45b7aa88107c system
sstable_activity 1451781670273 3201 906
{4:23}
1e76f1b0-b180-11e5-a211-45b7aa88107c system
sstable_activity 1451759952971 3303 700
{4:23}
e7b75b70-aec2-11e5-8bd4-45b7aa88107c system hints
1451458783911 57690316 57497847 {2:3}
ad102280-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451532129448
45671877 41137664 {1:11, 3:1, 4:8}
60906970-aec7-11e5-8bd4-45b7aa88107c system
sstable_activity 1451460704647 3751 974
{4:25}
88aed310-ad91-11e5-8bd4-45b7aa88107c system hints
1451327627969 56984347 56765328 {2:3}
3ad14f00-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531937776
46696097 38827028 {1:8, 3:2, 4:9}
84df8fb0-b00f-11e5-a211-45b7aa88107c system hints
1451601640491 18970740 18970740 {1:1}
657482e0-ad33-11e5-8bd4-45b7aa88107c system
sstable_activity 1451287196174 3701 931
{4:24}
9cc8af70-b24a-11e5-a211-45b7aa88107c system
sstable_activity 1451846923239 3134 773
{4:23}
dcbe5e30-afd0-11e5-a211-45b7aa88107c system
sstable_activity 1451574729619 3357 790
{4:23}
b285ced0-afa0-11e5-84e3-45b7aa88107c system hints
1451554042941 43310718 42137761 {1:1,
2:2}
119770e0-ad4e-11e5-8bd4-45b7aa88107c system hints
1451298651886 57397441 57190519 {2:3}
f1bb37a0-b204-11e5-a211-45b7aa88107c system hints
1451817000986 17713746 17566349 {1:1}
dcd0c720-b0b4-11e5-a211-45b7aa88107c system
sstable_activity 1451672654993 3189 797
{4:23}
cadb2320-ae94-11e5-8bd4-45b7aa88107c system
sstable_activity 1451438978642 3642 959
{4:25}
321a84d0-af63-11e5-8bd4-45b7aa88107c system hints
1451527628189 57407115 57179555 {1:3}
5d760ca0-af20-11e5-8bd4-45b7aa88107c system hints
1451498924650 57405307 57234936 {1:3}
2e59a0b0-af6d-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531916859 464 190
{1:1, 2:1}
8723a290-adba-11e5-8bd4-45b7aa88107c system hints
1451345234745 57065795 56814910 {2:3}
add0a8d0-af6c-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531701213 464 42
{2:2}
04c0d2f0-af6d-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531847071 446 0
{2:2}
315dd170-af98-11e5-b1dc-45b7aa88107c system schema_columns
1451550390279 129142 45841 {2:1,
3:1, 4:9}
a17d8f80-b24b-11e5-a211-45b7aa88107c system hints
1451847360632 16746317 16600978 {1:1}
e22e3b90-ad39-11e5-8bd4-45b7aa88107c system hints
1451289982409 57240038 56984895 {2:3}
38591c10-b22b-11e5-a211-45b7aa88107c system hints
1451833440337 17351684 17002773 {1:1}
fc274930-af6c-11e5-b1dc-45b7aa88107c system hints
1451531832643 63540861 59716559 {2:3}
aba057f0-b1b4-11e5-a211-45b7aa88107c system hints
1451782523631 18743329 18577220 {1:1}
81d5e980-af5c-11e5-8bd4-45b7aa88107c system hints
1451524755480 57603966 57407115 {2:3}
a1801510-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451532110049
46735894 40839394 {1:11, 4:9}
f7e02540-b1bf-11e5-a211-45b7aa88107c system hints
1451787376020 18577220 18403648 {1:1}
3e9a72b0-af72-11e5-b1dc-45b7aa88107c system hints
1451534091611 60455761 43314155 {2:1,
3:2}
30c103c0-af6d-11e5-b1dc-45b7aa88107c system peers
1451531920892 1396 567 {2:3,
3:1}
65f47770-aed7-11e5-8bd4-45b7aa88107c system hints
1451467585639 57780382 57445632 {2:3}
ecabc460-afa6-11e5-a211-45b7aa88107c system hints
1451556717478 29776619 29776619 {1:2}
5da4df30-afac-11e5-a211-45b7aa88107c system hints
1451559054499 20309429 18970740 {1:1,
2:1}
f5ce2d30-afa6-11e5-a211-45b7aa88107c system hints
1451556732803 29776619 29776619 {1:2}
4ada6610-aeec-11e5-8bd4-45b7aa88107c system hints
1451476559601 57459788 57221330 {1:3}
74ffd7f0-ae32-11e5-8bd4-45b7aa88107c system hints
1451396743919 57210071 57053060 {2:3}
621f05e0-b0ce-11e5-a211-45b7aa88107c system
sstable_activity 1451683616062 3060 719
{4:23}
5ec0e5b0-af2c-11e5-8bd4-45b7aa88107c system
sstable_activity 1451504080779 3756 971
{4:25}
92707090-b09b-11e5-a211-45b7aa88107c system
sstable_activity 1451661792793 3149 689
{4:23}
c35b51f0-ae40-11e5-8bd4-45b7aa88107c system hints
1451402888335 57190905 57037697 {2:3}
2a813e10-ade4-11e5-8bd4-45b7aa88107c system
sstable_activity 1451363118193 3772 960
{4:24}
c99593a0-af6c-11e5-b1dc-45b7aa88107c system hints
1451531747802 57645358 57613272 {3:3}
50002a00-aee5-11e5-8bd4-45b7aa88107c system hints
1451473561760 57670199 57459788 {2:3}
c2cf2570-ae6a-11e5-8bd4-45b7aa88107c system hints
1451420926279 57375776 57227330 {2:3}
1b5e7bc0-ad1a-11e5-8bd4-45b7aa88107c system
sstable_activity 1451276334460 3606 861
{4:24}
bf2f8d90-adc7-11e5-8bd4-45b7aa88107c system hints
1451350912233 56901257 56731829 {2:3}
55e055e0-b263-11e5-a211-45b7aa88107c system hints
1451857541694 16600978 16364535 {1:1}
1a60f830-afa7-11e5-a211-45b7aa88107c system hints
1451556794163 29776619 29776619 {1:2}
7b193990-adfd-11e5-8bd4-45b7aa88107c system
sstable_activity 1451373990825 3679 939
{4:24}
6bf27720-aec9-11e5-8bd4-45b7aa88107c system hints
1451461582737 57571883 57372615 {2:3}
d68f67b0-af13-11e5-8bd4-45b7aa88107c system hints
1451493544363 57518024 57335312 {2:3}
7946c2a0-b162-11e5-a211-45b7aa88107c system hints
1451747220426 18970740 18970740 {1:1}
1b686990-ad98-11e5-8bd4-45b7aa88107c system hints
1451330451113 56988235 56854912 {2:3}
30883420-afa7-11e5-a211-45b7aa88107c system hints
1451556831330 29776619 29776619 {1:2}
e5651290-b129-11e5-a211-45b7aa88107c system hints
1451722920505 18970740 18970740 {1:1}
feedabd0-ad61-11e5-8bd4-45b7aa88107c system hints
1451307210509 57251764 57133776 {2:3}
d75a36b0-afa0-11e5-84e3-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451554104731
47893133 41298592 {1:10, 2:1, 4:9}
b2226820-b05b-11e5-a211-45b7aa88107c system hints
1451634358178 18970740 18970740 {1:1}
7c7e0d80-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451532047960
47584630 40254518 {1:10, 2:1, 4:9}
7d39a5d0-ad7f-11e5-8bd4-45b7aa88107c system
sstable_activity 1451319877805 3637 882
{4:24}
c05d07c0-b1fe-11e5-a211-45b7aa88107c system
sstable_activity 1451814341180 3329 852
{4:23}
a227fdb0-add4-11e5-8bd4-45b7aa88107c system hints
1451356446987 57070333 56869637 {2:3}
26a79a60-aeb5-11e5-8bd4-45b7aa88107c system hints
1451452876550 57637242 57408618 {2:3}
def14230-af6c-11e5-b1dc-45b7aa88107c system hints
1451531783635 57613272 57622817 {1:3}
ce8c20b0-afa6-11e5-a211-45b7aa88107c system local
1451556666939 1167 696 {4:1}
75ace380-b1e5-11e5-a211-45b7aa88107c system
sstable_activity 1451803478456 3171 820
{4:23}
88d05020-af86-11e5-b1dc-45b7aa88107c system hints
1451542806050 43921794 43937401 {2:3}
0a59ef20-b1f8-11e5-a211-45b7aa88107c system hints
1451811458834 17972574 17713746 {1:1}
7ace0730-afa7-11e5-a211-45b7aa88107c system hints
1451556955939 29776619 29776619 {1:2}
9e1ee8b0-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451532104379
5935114 2171959 {4:9}
0580cea0-ae5c-11e5-8bd4-45b7aa88107c system hints
1451414595722 57388662 57179650 {2:3}
378110c0-afa8-11e5-a211-45b7aa88107c system hints
1451557272524 28568485 24640663 {1:1,
2:1}
ef231ab0-af5e-11e5-8bd4-45b7aa88107c system
sstable_activity 1451525797851 3739 973
{4:25}
301fc6c0-ae25-11e5-8bd4-45b7aa88107c system hints
1451391044908 56996239 56821882 {2:3}
5ceccc10-afa7-11e5-a211-45b7aa88107c system hints
1451556905809 29776619 29776619 {1:2}
44713fe0-afa7-11e5-a211-45b7aa88107c system hints
1451556864734 29776619 29776619 {1:2}
0b0786b0-afa7-11e5-a211-45b7aa88107c system hints
1451556768411 29776619 29776619 {1:2}
5a667200-ae78-11e5-8bd4-45b7aa88107c system hints
1451426764064 57497152 57334697 {2:3}
d4306ba0-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531765594
43985647 32914710 {1:7, 3:1, 4:9}
7596f1a0-ad9f-11e5-8bd4-45b7aa88107c system hints
1451333608890 56854912 56608995 {1:3}
e31fc540-adee-11e5-8bd4-45b7aa88107c system hints
1451367722900 57057877 56867011 {2:3}
52458b10-b1e3-11e5-a211-45b7aa88107c system hints
1451802560065 18087954 17972574 {1:1}
278bf030-af13-11e5-8bd4-45b7aa88107c system
sstable_activity 1451493250739 3720 922
{4:25}
76443290-af55-11e5-8bd4-45b7aa88107c system hints
1451521729593 57500226 57294613 {2:3}
8065b450-af33-11e5-8bd4-45b7aa88107c system hints
1451507143701 57557563 57359219 {2:3}
ae29e180-b0ce-11e5-a211-45b7aa88107c system hints
1451683743640 18970740 18970740 {1:1}
d942a360-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531774102
8615536 4447310 {1:1, 3:1, 4:9}
6b039a10-b289-11e5-a211-45b7aa88107c system hints
1451873898033 15967755 15796743 {1:1}
3afa2ba0-b14d-11e5-a211-45b7aa88107c system
sstable_activity 1451738096474 3196 778
{4:23}
3d6e34a0-afa7-11e5-a211-45b7aa88107c system hints
1451556852970 29776619 29776619 {1:2}
b27e7970-af6c-11e5-b1dc-45b7aa88107c system local
1451531709063 1215 668 {4:1}
78715730-ae62-11e5-8bd4-45b7aa88107c system
sstable_activity 1451417365539 3749 950
{4:25}
46bd9b90-aed0-11e5-8bd4-45b7aa88107c system hints
1451464526793 57589209 57452596 {2:3}
d9030250-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_SCO 1451531773685
61554578 59278964 {1:76826, 2:11126, 3:604, 4:9}
c70ceb50-afcc-11e5-a211-45b7aa88107c system hints
1451572975237 18970740 18970740 {1:1}
69f8a090-b106-11e5-a211-45b7aa88107c system hints
1451707681048 18970740 18970740 {1:1}
7dd087b0-af79-11e5-b1dc-45b7aa88107c system hints
1451537204139 43314155 43324175 {1:3}
94404a50-b166-11e5-a211-45b7aa88107c system
sstable_activity 1451748983669 3364 926
{4:23}
d99598d0-ad10-11e5-8bd4-45b7aa88107c system hints
1451272358621 57162919 56981333 {2:3}
7fe03f20-b23d-11e5-a211-45b7aa88107c system hints
1451841291282 17002773 16746317 {1:1}
e664d240-b1cc-11e5-a211-45b7aa88107c system hints
1451792930148 18403648 18252171 {1:1}
04178e40-afa7-11e5-a211-45b7aa88107c system hints
1451556756772 29776619 29776619 {1:2}
1a414650-ada6-11e5-8bd4-45b7aa88107c system hints
1451336462133 56826857 56710683 {2:3}
ad9ef5f0-b026-11e5-a211-45b7aa88107c system hints
1451611587279 18970740 18970740 {1:1}
60a4e320-ad17-11e5-8bd4-45b7aa88107c system hints
1451275162194 57236552 57076664 {2:3}
917e6370-aeae-11e5-8bd4-45b7aa88107c system hints
1451450049319 57539934 57384675 {2:3}
eb1440d0-af6c-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531803997
43867391 36299816 {1:8, 4:10}
a20bfc70-aee0-11e5-8bd4-45b7aa88107c system
sstable_activity 1451471551927 3833 945
{4:25}
47cbe960-aef3-11e5-8bd4-45b7aa88107c system hints
1451479560950 57443969 57255886 {2:3}
e2a84f00-af25-11e5-8bd4-45b7aa88107c system hints
1451501295600 57449985 57298217 {2:3}
af1628a0-ade1-11e5-8bd4-45b7aa88107c system hints
1451362052138 57001606 56839416 {2:3}
b0fea4a0-ae16-11e5-8bd4-45b7aa88107c system
sstable_activity 1451384818666 3645 872
{3:1, 4:24}
15473980-aeae-11e5-8bd4-45b7aa88107c system
sstable_activity 1451449840920 3789 975
{4:25}
e5bdc7c0-afa6-11e5-a211-45b7aa88107c system hints
1451556705852 29776619 29776619 {1:2}
d7808360-afa0-11e5-84e3-45b7aa88107c system
compactions_in_progress 1451554104982 461 42
{2:2}
36a22720-ae63-11e5-8bd4-45b7aa88107c system hints
1451417684626 57282895 57148294 {2:3}
99222cc0-ad54-11e5-8bd4-45b7aa88107c system hints
1451301456268 57258179 57086571 {2:3}
a3480190-adb1-11e5-8bd4-45b7aa88107c system
sstable_activity 1451341416489 3683 925
{4:24}
1198f0c0-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531868620
8937482 3569558 {2:1, 3:1, 4:8}
422d8470-b036-11e5-a211-45b7aa88107c system
sstable_activity 1451618278967 3243 825
{4:23}
12aee840-b218-11e5-a211-45b7aa88107c system
sstable_activity 1451825216708 3176 819
{4:23}
85af2f30-adb3-11e5-8bd4-45b7aa88107c system hints
1451342225827 57028608 56811880 {2:3}
8b98e730-afa0-11e5-84e3-45b7aa88107c system hints
1451553977634 88625803 44359037 {4:2,
5:1}
0c2b9aa0-ae02-11e5-8bd4-45b7aa88107c system hints
1451375952202 56891360 56548629 {2:3}
ec9afff0-af00-11e5-8bd4-45b7aa88107c system hints
1451485420911 57594351 57326414 {2:3}
98a23cb0-b199-11e5-a211-45b7aa88107c system hints
1451770895355 18970740 18906200 {1:1}
8f90dde0-afb7-11e5-a211-45b7aa88107c system
sstable_activity 1451563862718 3508 827
{1:2, 2:8, 3:2, 4:19}
822b7da0-b14b-11e5-a211-45b7aa88107c system hints
1451737356922 18970740 18970740 {1:1}
f5dc9cb0-ae54-11e5-8bd4-45b7aa88107c system hints
1451411563003 57341080 57162518 {2:3}
83efd290-af8d-11e5-b1dc-45b7aa88107c system hints
1451545804345 43996474 44000895 {2:3}
ba28f830-af80-11e5-b1dc-45b7aa88107c system hints
1451540311859 43744401 43725750 {1:1,
2:2}
decb8560-afa6-11e5-a211-45b7aa88107c system hints
1451556694198 29776619 29776619 {1:2}
8fb80860-ad5b-11e5-8bd4-45b7aa88107c system hints
1451304446950 57324867 56998191 {2:3}
969514e0-afa7-11e5-a211-45b7aa88107c system hints
1451557002542 29776619 29776619 {1:2}
bf7d00d0-afb5-11e5-a211-45b7aa88107c system hints
1451563084125 18970740 18970740 {1:1}
113e2660-af48-11e5-8bd4-45b7aa88107c system hints
1451515976646 57413867 57219902 {2:3}
32ee9f00-adf5-11e5-8bd4-45b7aa88107c system hints
1451370433776 57034296 56618944 {2:3}
3c583550-afea-11e5-a211-45b7aa88107c system
sstable_activity 1451585627429 3073 720
{4:23}
58437cd0-af3a-11e5-8bd4-45b7aa88107c system hints
1451510082845 57369449 57188682 {2:3}
871b8c30-b0ef-11e5-a211-45b7aa88107c system hints
1451697851507 18970740 18970740 {1:1}
00522050-ae7f-11e5-8bd4-45b7aa88107c system hints
1451429619413 57450075 57258800 {2:3}
14e3fd10-af6d-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531874145 472 42
{2:2}
b3e23af0-b0e7-11e5-a211-45b7aa88107c system
sstable_activity 1451694490655 3013 726
{4:23}
aa5986d0-ad24-11e5-8bd4-45b7aa88107c system hints
1451280869309 57181046 56983904 {2:3}
29697550-aebc-11e5-8bd4-45b7aa88107c system hints
1451455887653 57617579 57451553 {2:3}
8201d200-ae8c-11e5-8bd4-45b7aa88107c system hints
1451435420448 57558970 57354959 {2:3}
45bc2770-addb-11e5-8bd4-45b7aa88107c system hints
1451359298407 56977263 56817508 {2:3}
ebd29a70-affe-11e5-a211-45b7aa88107c system hints
1451594511767 18970740 18970740 {1:1}
9530ef00-ad4c-11e5-8bd4-45b7aa88107c system
sstable_activity 1451298013680 3580 885
{4:24}
caa48640-afa6-11e5-a211-45b7aa88107c system local
1451556660388 1013 677 {4:1}
acee31c0-aea0-11e5-8bd4-45b7aa88107c system hints
1451444082396 57659639 57460800 {2:3}
aa2ad760-ae71-11e5-8bd4-45b7aa88107c system hints
1451423891414 57468574 57242109 {2:3}
9e29cbd0-b036-11e5-a211-45b7aa88107c system hints
1451618433293 18970740 18970740 {1:1}
df7b80d0-af6c-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531784541 542 172
{1:2, 2:2}
48b11430-ad7d-11e5-8bd4-45b7aa88107c system hints
1451318930675 56911864 56738503 {2:3}
6afb39b0-b199-11e5-a211-45b7aa88107c system
sstable_activity 1451770818763 3039 788
{4:23}
f77313f0-af40-11e5-8bd4-45b7aa88107c system hints
1451512926895 57410690 57197897 {2:3}
38c86a20-adc1-11e5-8bd4-45b7aa88107c system hints
1451348109762 56814910 56689825 {1:3}
c2406070-af0e-11e5-8bd4-45b7aa88107c system hints
1451491362807 57391505 57316604 {2:3}
610527f0-b26f-11e5-a211-45b7aa88107c system hints
1451862714351 16364535 16082982 {1:1}
23d18f10-ae49-11e5-8bd4-45b7aa88107c system
sstable_activity 1451406486145 3587 941
{4:25}
76303090-ad40-11e5-8bd4-45b7aa88107c system hints
1451292807705 57224902 57033897 {2:3}
787be5c0-ad76-11e5-8bd4-45b7aa88107c system hints
1451316004380 56853811 56717047 {2:3}
f446f990-aef9-11e5-8bd4-45b7aa88107c system hints
1451482427305 57466666 57373339 {2:3}
3ba17b30-ad47-11e5-8bd4-45b7aa88107c system hints
1451295715939 57320671 57178557 {2:3}
85f76e20-afa8-11e5-a211-45b7aa88107c system hints
1451557404162 24767833 20275337 {1:1,
2:1}
464b1550-af6d-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531957029 464 42
{2:2}
d39fbc30-af9f-11e5-b1dc-45b7aa88107c system
sstable_activity 1451553668979 3895 835
{4:26}
acf90bb0-afa7-11e5-a211-45b7aa88107c system hints
1451557040107 29214113 28457489 {1:1,
2:1}
d09de8e0-ad65-11e5-8bd4-45b7aa88107c system
sstable_activity 1451308850798 3685 899
{4:24}
6e371d30-b27d-11e5-a211-45b7aa88107c system hints
1451868749443 16082982 15967755 {1:1}
fca1eea0-ad83-11e5-8bd4-45b7aa88107c system hints
1451321809546 56925900 56697130 {2:3}
31372130-b0f9-11e5-a211-45b7aa88107c system hints
1451702002371 18970740 18970740 {1:1}
a0dd4c90-af45-11e5-8bd4-45b7aa88107c system
sstable_activity 1451514929113 3785 882
{4:25}
a180b150-af6d-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451532110053 446 0
{2:2}
46239020-af6d-11e5-b1dc-45b7aa88107c
webtrn_study_log_formallySCORM_STU_COURSE 1451531956770
45648060 39245044 {1:8, 2:1, 3:1, 4:9}
76cf6680-afa0-11e5-84e3-45b7aa88107c system local
1451553942760 1265 674 {4:1}
e4c92910-b11b-11e5-a211-45b7aa88107c system hints
1451716906529 18970740 18970740 {1:1}
16e97320-b264-11e5-a211-45b7aa88107c system
sstable_activity 1451857865554 3114 726
{4:23}
fddd18f0-b100-11e5-a211-45b7aa88107c system
sstable_activity 1451705352191 2959 716
{4:23}
ea955a00-b173-11e5-a211-45b7aa88107c system hints
1451754711968 18970740 18970740 {1:1}
52156430-adce-11e5-8bd4-45b7aa88107c system hints
1451353735667 56989435 56824386 {2:3}
b9d73cf0-ae47-11e5-8bd4-45b7aa88107c system hints
1451405878847 57223358 57041982 {2:3}
2ab91b60-afaf-11e5-a211-45b7aa88107c system schema_triggers
1451560257558 134 60 {1:1,
3:1}
9712f8c0-ae39-11e5-8bd4-45b7aa88107c system hints
1451399807564 57261044 57081560 {2:3}
86ec5560-adfb-11e5-8bd4-45b7aa88107c system hints
1451373151670 56802348 56679018 {2:3}
4b9ffb20-b27d-11e5-a211-45b7aa88107c system
sstable_activity 1451868691410 3037 791
{4:23}
e0ed0e70-aef9-11e5-8bd4-45b7aa88107c system
sstable_activity 1451482394839 3749 941
{4:25}
28e78950-afa7-11e5-a211-45b7aa88107c system hints
1451556818533 29776619 29776619 {1:2}
47589f00-b231-11e5-a211-45b7aa88107c system
sstable_activity 1451836042480 3212 785
{4:23}
499c5d30-afaf-11e5-a211-45b7aa88107c system peers
1451560309379 1810 556 {3:4}
2d335080-af6a-11e5-8bd4-45b7aa88107c system hints
1451530626440 57406002 57261820 {2:3}
d46ed430-af6c-11e5-b1dc-45b7aa88107c system
compactions_in_progress 1451531766003 608 268
{1:2, 2:1}
31cbd580-af98-11e5-b1dc-45b7aa88107c system local
1451550391000 905 741 {4:1}
95dec160-afa0-11e5-84e3-45b7aa88107c system hints
1451553994870 44359037 44359037 {1:3}
4726d640-afe4-11e5-a211-45b7aa88107c system hints
1451583068580 18970740 18970740 {1:1}
9ed33380-ae7b-11e5-8bd4-45b7aa88107c system
sstable_activity 1451428167352 3638 906
{4:25}
50fa9650-b1a8-11e5-a211-45b7aa88107c system hints
1451777217589 18906200 18743329 {1:1}
b66e4420-b183-11e5-a211-45b7aa88107c system hints
1451761496418 18970740 18970740 {1:1}
5a003890-ade8-11e5-8bd4-45b7aa88107c system hints
1451364915865 57066041 56952696 {2:3}
451f4710-ad1e-11e5-8bd4-45b7aa88107c system hints
1451278122497 57133634 56933373 {2:3}
b1e973b0-ad6f-11e5-8bd4-45b7aa88107c system hints
1451313094251 56935662 56616319 {1:3}
0ce9d500-ae17-11e5-8bd4-45b7aa88107c system hints
1451384972880 56788738 56649325 {2:3}
0fc24530-ae1e-11e5-8bd4-45b7aa88107c system hints
1451387984131 56905778 56758301 {2:3}
31d54e10-b0da-11e5-a211-45b7aa88107c system hints
1451688689009 18970740 18970740 {1:1}
cb322de0-b133-11e5-a211-45b7aa88107c system
sstable_activity 1451727171518 3177 862
{4:23}
4f987470-af1a-11e5-8bd4-45b7aa88107c system hints
1451496324407 57567325 57405307 {2:3}
bbad5970-af2c-11e5-8bd4-45b7aa88107c system hints
1451504236679 57519774 57367465 {2:3}
5e31ef40-aff4-11e5-a211-45b7aa88107c system hints
1451589979188 18970740
On Mon, Jan 4, 2016 at 9:54 AM, Shuo Chen <ch...@gmail.com> wrote:
> The attachment is the current system.log and config of the cassandra
> cluster. Over 8000 full gc happens in 3 days with 121 young gc.
>
> There are client operation in these days. Besides most columnfamily in the
> cluster are supercolumnfamily created by cassandra-cli. Most rows have
> average 30 sub-rows and each sub-row has 20 columns.
>
> You said that there are many compaction tasks. So how to check the source
> of this compactions? Thank you!
>
> On Fri, Jan 1, 2016 at 10:09 AM, Graham Sanderson <gr...@vast.com> wrote:
>
>> If you are lucky that might mask the real issue, but I doubt it… that is
>> an insane number of compaction tasks and indicative of another problem. I
>> would check release notes of 2.0.6+, if I recall that was not a stable
>> version and may have had leaks.
>>
>> Aside from that, just FYI, if you use native_objects for memtables you
>> can run CMS fine even up to largish (20-30G) heap sizes without long GC
>> pauses (at least over months) . That said look for promotion failures of
>> 131074 DWORDS. Not sure what your GC logging options are, but what you
>> posted doesn’t say why a full GC happened which would be totally tunable,
>> but as I say I don’t think GC is actually your problem
>>
>>
>> On Dec 31, 2015, at 10:16 AM, Ipremyadav <ip...@gmail.com> wrote:
>>
>> Simplest option is to use java 8 with G1 gc.
>>
>> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <ch...@gmail.com> wrote:
>>
>> I have a cassandra 2.0.6 cluster with four nodes as backup database. The
>> only operation is posting data into db. Recently, the full gc of the nodes
>> increases apparently and blocks cluster operation.
>>
>> The load of each node is 10G. The heap is 8G each with default jvm memory
>> settings. The cpu is 24 cores. The settings of cassandra.yaml are almost
>> default.
>>
>> For debug, all the clients are disconnected, however the gc is still high.
>>
>> The GC is too often that it appears in every minute and blocks for nealy
>> 30s.
>>
>> How to debug this situation
>>
>> The following is the snippets of gc log:
>>
>> before full gc histogram
>>
>>
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs]
>> Before GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> Before GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> ----------------------------------------------
>> 1: 61030245 1952967840 java.util.concurrent.FutureTask
>> 2: 61031398 1464753552
>> java.util.concurrent.Executors$RunnableAdapter
>> 3: 61030312 1464727488
>> java.util.concurrent.LinkedBlockingQueue$Node
>> 4: 61030244 1464725856
>> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
>> 5: 214181 13386992 [B
>> 6: 202890 9738720 java.nio.HeapByteBuffer
>> 7: 37622 5994808 [C
>> 8: 42163 5722096 <constMethodKlass>
>> 9: 42163 5407152 <methodKlass>
>> 10: 4170 4648240 <constantPoolKlass>
>> 11: 100060 4002400
>> org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
>> 12: 4170 2860816 <instanceKlassKlass>
>> 13: 3664 2702720 <constantPoolCacheKlass>
>> 14: 4817 2701576 [J
>> 15: 2575 1056600 <methodDataKlass>
>> 16: 37431 898344 java.lang.String
>> 17: 2585 627352 [Ljava.lang.Object;
>> 18: 17728 567296
>> java.util.concurrent.ConcurrentHashMap$HashEntry
>> 19: 22111 530664 javax.management.ObjectName$Property
>> 20: 7595 463824 [S
>> 21: 4534 433560 java.lang.Class
>> 22: 11581 370592 java.util.HashMap$Entry
>>
>> ...
>> Total 289942930 8399196504
>> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs]
>> CMS: Large block 0x00000007f7da9588
>>
>> After GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 6335823
>> Max Chunk Size: 6335823
>> Number of Blocks: 1
>> Av. Block Size: 6335823
>> Tree Height: 1
>> After GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>>
>>
>> It seems related to objects of Futuretask.
>>
>> --
>> *陈硕* *Shuo Chen*
>> chenatu2006@gmail.com
>> chenshuo@whaty.com
>>
>>
>>
>
>
> --
> *陈硕* *Shuo Chen*
> chenatu2006@gmail.com
> chenshuo@whaty.com
>
--
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com
Re: cassandra full gc too often
Posted by Robert Coli <rc...@eventbrite.com>.
On Sun, Jan 3, 2016 at 5:54 PM, Shuo Chen <ch...@gmail.com> wrote:
> There are client operation in these days. Besides most columnfamily in the
> cluster are supercolumnfamily created by cassandra-cli. Most rows have
> average 30 sub-rows and each sub-row has 20 columns.
>
Supercolumns, especially pre-CQL implementation, have serious performance
and memory concerns.
In general, one should not use them.
=Rob
Re: cassandra full gc too often
Posted by Shuo Chen <ch...@gmail.com>.
The attachment is the current system.log and config of the cassandra
cluster. Over 8000 full gc happens in 3 days with 121 young gc.
There are client operation in these days. Besides most columnfamily in the
cluster are supercolumnfamily created by cassandra-cli. Most rows have
average 30 sub-rows and each sub-row has 20 columns.
You said that there are many compaction tasks. So how to check the source
of this compactions? Thank you!
On Fri, Jan 1, 2016 at 10:09 AM, Graham Sanderson <gr...@vast.com> wrote:
> If you are lucky that might mask the real issue, but I doubt it… that is
> an insane number of compaction tasks and indicative of another problem. I
> would check release notes of 2.0.6+, if I recall that was not a stable
> version and may have had leaks.
>
> Aside from that, just FYI, if you use native_objects for memtables you can
> run CMS fine even up to largish (20-30G) heap sizes without long GC pauses
> (at least over months) . That said look for promotion failures of 131074
> DWORDS. Not sure what your GC logging options are, but what you posted
> doesn’t say why a full GC happened which would be totally tunable, but as I
> say I don’t think GC is actually your problem
>
>
> On Dec 31, 2015, at 10:16 AM, Ipremyadav <ip...@gmail.com> wrote:
>
> Simplest option is to use java 8 with G1 gc.
>
> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <ch...@gmail.com> wrote:
>
> I have a cassandra 2.0.6 cluster with four nodes as backup database. The
> only operation is posting data into db. Recently, the full gc of the nodes
> increases apparently and blocks cluster operation.
>
> The load of each node is 10G. The heap is 8G each with default jvm memory
> settings. The cpu is 24 cores. The settings of cassandra.yaml are almost
> default.
>
> For debug, all the clients are disconnected, however the gc is still high.
>
> The GC is too often that it appears in every minute and blocks for nealy
> 30s.
>
> How to debug this situation
>
> The following is the snippets of gc log:
>
> before full gc histogram
>
>
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs]
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
> ----------------------------------------------
> 1: 61030245 1952967840 java.util.concurrent.FutureTask
> 2: 61031398 1464753552
> java.util.concurrent.Executors$RunnableAdapter
> 3: 61030312 1464727488
> java.util.concurrent.LinkedBlockingQueue$Node
> 4: 61030244 1464725856
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
> 5: 214181 13386992 [B
> 6: 202890 9738720 java.nio.HeapByteBuffer
> 7: 37622 5994808 [C
> 8: 42163 5722096 <constMethodKlass>
> 9: 42163 5407152 <methodKlass>
> 10: 4170 4648240 <constantPoolKlass>
> 11: 100060 4002400
> org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
> 12: 4170 2860816 <instanceKlassKlass>
> 13: 3664 2702720 <constantPoolCacheKlass>
> 14: 4817 2701576 [J
> 15: 2575 1056600 <methodDataKlass>
> 16: 37431 898344 java.lang.String
> 17: 2585 627352 [Ljava.lang.Object;
> 18: 17728 567296
> java.util.concurrent.ConcurrentHashMap$HashEntry
> 19: 22111 530664 javax.management.ObjectName$Property
> 20: 7595 463824 [S
> 21: 4534 433560 java.lang.Class
> 22: 11581 370592 java.util.HashMap$Entry
>
> ...
> Total 289942930 8399196504
> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs]
> CMS: Large block 0x00000007f7da9588
>
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 6335823
> Max Chunk Size: 6335823
> Number of Blocks: 1
> Av. Block Size: 6335823
> Tree Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
>
>
> It seems related to objects of Futuretask.
>
> --
> *陈硕* *Shuo Chen*
> chenatu2006@gmail.com
> chenshuo@whaty.com
>
>
>
--
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com
Re: cassandra full gc too often
Posted by Graham Sanderson <gr...@vast.com>.
If you are lucky that might mask the real issue, but I doubt it… that is an insane number of compaction tasks and indicative of another problem. I would check release notes of 2.0.6+, if I recall that was not a stable version and may have had leaks.
Aside from that, just FYI, if you use native_objects for memtables you can run CMS fine even up to largish (20-30G) heap sizes without long GC pauses (at least over months) . That said look for promotion failures of 131074 DWORDS. Not sure what your GC logging options are, but what you posted doesn’t say why a full GC happened which would be totally tunable, but as I say I don’t think GC is actually your problem
> On Dec 31, 2015, at 10:16 AM, Ipremyadav <ip...@gmail.com> wrote:
>
> Simplest option is to use java 8 with G1 gc.
>
> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <chenatu2006@gmail.com <ma...@gmail.com>> wrote:
>
>> I have a cassandra 2.0.6 cluster with four nodes as backup database. The only operation is posting data into db. Recently, the full gc of the nodes increases apparently and blocks cluster operation.
>>
>> The load of each node is 10G. The heap is 8G each with default jvm memory settings. The cpu is 24 cores. The settings of cassandra.yaml are almost default.
>>
>> For debug, all the clients are disconnected, however the gc is still high.
>>
>> The GC is too often that it appears in every minute and blocks for nealy 30s.
>>
>> How to debug this situation
>>
>> The following is the snippets of gc log:
>>
>> before full gc histogram
>>
>>
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs]
>> Before GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> Before GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>> ----------------------------------------------
>> 1: 61030245 1952967840 java.util.concurrent.FutureTask
>> 2: 61031398 1464753552 java.util.concurrent.Executors$RunnableAdapter
>> 3: 61030312 1464727488 java.util.concurrent.LinkedBlockingQueue$Node
>> 4: 61030244 1464725856 org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
>> 5: 214181 13386992 [B
>> 6: 202890 9738720 java.nio.HeapByteBuffer
>> 7: 37622 5994808 [C
>> 8: 42163 5722096 <constMethodKlass>
>> 9: 42163 5407152 <methodKlass>
>> 10: 4170 4648240 <constantPoolKlass>
>> 11: 100060 4002400 org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
>> 12: 4170 2860816 <instanceKlassKlass>
>> 13: 3664 2702720 <constantPoolCacheKlass>
>> 14: 4817 2701576 [J
>> 15: 2575 1056600 <methodDataKlass>
>> 16: 37431 898344 java.lang.String
>> 17: 2585 627352 [Ljava.lang.Object;
>> 18: 17728 567296 java.util.concurrent.ConcurrentHashMap$HashEntry
>> 19: 22111 530664 javax.management.ObjectName$Property
>> 20: 7595 463824 [S
>> 21: 4534 433560 java.lang.Class
>> 22: 11581 370592 java.util.HashMap$Entry
>>
>> ...
>> Total 289942930 8399196504
>> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs]
>> CMS: Large block 0x00000007f7da9588
>>
>> After GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 6335823
>> Max Chunk Size: 6335823
>> Number of Blocks: 1
>> Av. Block Size: 6335823
>> Tree Height: 1
>> After GC:
>> Statistics for BinaryTreeDictionary:
>> ------------------------------------
>> Total Free Space: 0
>> Max Chunk Size: 0
>> Number of Blocks: 0
>> Tree Height: 0
>>
>>
>> It seems related to objects of Futuretask.
>>
>> --
>> 陈硕 Shuo Chen
>> chenatu2006@gmail.com <ma...@gmail.com>
>> chenshuo@whaty.com <ma...@whaty.com>
Re: cassandra full gc too often
Posted by Ipremyadav <ip...@gmail.com>.
Simplest option is to use java 8 with G1 gc.
> On 31 Dec 2015, at 10:23 a.m., Shuo Chen <ch...@gmail.com> wrote:
>
> I have a cassandra 2.0.6 cluster with four nodes as backup database. The only operation is posting data into db. Recently, the full gc of the nodes increases apparently and blocks cluster operation.
>
> The load of each node is 10G. The heap is 8G each with default jvm memory settings. The cpu is 24 cores. The settings of cassandra.yaml are almost default.
>
> For debug, all the clients are disconnected, however the gc is still high.
>
> The GC is too often that it appears in every minute and blocks for nealy 30s.
>
> How to debug this situation
>
> The following is the snippets of gc log:
>
> before full gc histogram
>
>
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
> 1620.992: [GC 7377342K(8178944K), 0.1380260 secs]
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
> Before GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
> ----------------------------------------------
> 1: 61030245 1952967840 java.util.concurrent.FutureTask
> 2: 61031398 1464753552 java.util.concurrent.Executors$RunnableAdapter
> 3: 61030312 1464727488 java.util.concurrent.LinkedBlockingQueue$Node
> 4: 61030244 1464725856 org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask
> 5: 214181 13386992 [B
> 6: 202890 9738720 java.nio.HeapByteBuffer
> 7: 37622 5994808 [C
> 8: 42163 5722096 <constMethodKlass>
> 9: 42163 5407152 <methodKlass>
> 10: 4170 4648240 <constantPoolKlass>
> 11: 100060 4002400 org.apache.cassandra.io.sstable.IndexHelper$IndexInfo
> 12: 4170 2860816 <instanceKlassKlass>
> 13: 3664 2702720 <constantPoolCacheKlass>
> 14: 4817 2701576 [J
> 15: 2575 1056600 <methodDataKlass>
> 16: 37431 898344 java.lang.String
> 17: 2585 627352 [Ljava.lang.Object;
> 18: 17728 567296 java.util.concurrent.ConcurrentHashMap$HashEntry
> 19: 22111 530664 javax.management.ObjectName$Property
> 20: 7595 463824 [S
> 21: 4534 433560 java.lang.Class
> 22: 11581 370592 java.util.HashMap$Entry
>
> ...
> Total 289942930 8399196504
> 1658.022: [Full GC 8178943K->6241829K(8178944K), 27.8562520 secs]
> CMS: Large block 0x00000007f7da9588
>
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 6335823
> Max Chunk Size: 6335823
> Number of Blocks: 1
> Av. Block Size: 6335823
> Tree Height: 1
> After GC:
> Statistics for BinaryTreeDictionary:
> ------------------------------------
> Total Free Space: 0
> Max Chunk Size: 0
> Number of Blocks: 0
> Tree Height: 0
>
>
> It seems related to objects of Futuretask.
>
> --
> 陈硕 Shuo Chen
> chenatu2006@gmail.com
> chenshuo@whaty.com