You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ruchir Jha <ru...@gmail.com> on 2014/07/28 15:53:57 UTC
Full GC in cassandra
Really curious to know what's causing the spike in Columns and
DeletedColums below :
2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class
Histogram:
num #instances #bytes class name
----------------------------------------------
1: 132626060 6366050880 java.nio.HeapByteBuffer
2: 28194918 3920045528 [B
3: 78124737 3749987376
edu.stanford.ppl.concurrent.SnapTreeMap$Node
* 4: 67650128 2164804096
<2164804096> org.apache.cassandra.db.Column*
* 5: 16315310 522089920 org.apache.cassandra.db.DeletedColumn*
6: 6818 392489608 [I
7: 2844374 273059904
edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch
8: 5727000 229080000 java.util.TreeMap$Entry
9: 767742 182921376 [J
10: 2932832 140775936
edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder
11: 2844375 91020000
edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch
12: 4145131 66322096
java.util.concurrent.atomic.AtomicReference
13: 437874 64072392 [C
14: 2660844 63860256
java.util.concurrent.ConcurrentSkipListMap$Node
15: 4920 62849864 [[B
16: 1632063 52226016 edu.stanford.ppl.concurrent.SnapTreeMap
Re: Full GC in cassandra
Posted by graham sanderson <gr...@vast.com>.
1) Is that heap dump after full GC?
2) Are you doing a lot of concurrent writes to the same partition
3) Is the system hinting at the time of the spike
4) when you say “spike” do you mean this is unusually high?
On Jul 28, 2014, at 11:07 AM, Keith Wright <kw...@nanigans.com> wrote:
> What’s your cfhistograms look like?
>
> From: Ruchir Jha <ru...@gmail.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Monday, July 28, 2014 at 10:43 AM
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Re: Full GC in cassandra
>
> Also we do subsequent updates (atleat 4) for each piece of data that we write.
>
>
> On Mon, Jul 28, 2014 at 10:36 AM, Ruchir Jha <ru...@gmail.com> wrote:
>> Doing about 5K writes / second. Avg Data Size = 1.6 TB / node. Total Data Size = 21 TB.
>>
>> And this is the nodetool cfstats output for one of our busiest column families:
>>
>> SSTable count: 10
>> Space used (live): 43239294899
>> Space used (total): 43239419603
>> SSTable Compression Ratio: 0.2954468408497778
>> Number of Keys (estimate): 63729152
>> Memtable Columns Count: 1921620
>> Memtable Data Size: 257680020
>> Memtable Switch Count: 9
>> Read Count: 6167
>> Read Latency: NaN ms.
>> Write Count: 770984
>> Write Latency: 0.098 ms.
>> Pending Tasks: 0
>> Bloom Filter False Positives: 370
>> Bloom Filter False Ratio: 0.00000
>> Bloom Filter Space Used: 80103200
>> Compacted row minimum size: 180
>> Compacted row maximum size: 3311
>> Compacted row mean size: 2631
>> Average live cells per slice (last five minutes): 73.0
>> Average tombstones per slice (last five minutes): 13.0
>>
>>
>>
>> On Mon, Jul 28, 2014 at 10:14 AM, Mark Reddy <ma...@boxever.com> wrote:
>>> What is your data size and number of columns in Cassandra. Do you do many deletions?
>>>
>>>
>>> On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha <ru...@gmail.com> wrote:
>>>> Really curious to know what's causing the spike in Columns and DeletedColums below :
>>>>
>>>>
>>>> 2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class Histogram:
>>>> num #instances #bytes class name
>>>> ----------------------------------------------
>>>> 1: 132626060 6366050880 java.nio.HeapByteBuffer
>>>> 2: 28194918 3920045528 [B
>>>> 3: 78124737 3749987376 edu.stanford.ppl.concurrent.SnapTreeMap$Node
>>>> 4: 67650128 2164804096 org.apache.cassandra.db.Column
>>>> 5: 16315310 522089920 org.apache.cassandra.db.DeletedColumn
>>>> 6: 6818 392489608 [I
>>>> 7: 2844374 273059904 edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch
>>>> 8: 5727000 229080000 java.util.TreeMap$Entry
>>>> 9: 767742 182921376 [J
>>>> 10: 2932832 140775936 edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder
>>>> 11: 2844375 91020000 edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch
>>>> 12: 4145131 66322096 java.util.concurrent.atomic.AtomicReference
>>>> 13: 437874 64072392 [C
>>>> 14: 2660844 63860256 java.util.concurrent.ConcurrentSkipListMap$Node
>>>> 15: 4920 62849864 [[B
>>>> 16: 1632063 52226016 edu.stanford.ppl.concurrent.SnapTreeMap
>>>
>>
>
Re: Full GC in cassandra
Posted by Keith Wright <kw...@nanigans.com>.
What’s your cfhistograms look like?
From: Ruchir Jha <ru...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Monday, July 28, 2014 at 10:43 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Full GC in cassandra
Also we do subsequent updates (atleat 4) for each piece of data that we write.
On Mon, Jul 28, 2014 at 10:36 AM, Ruchir Jha <ru...@gmail.com>> wrote:
Doing about 5K writes / second. Avg Data Size = 1.6 TB / node. Total Data Size = 21 TB.
And this is the nodetool cfstats output for one of our busiest column families:
SSTable count: 10
Space used (live): 43239294899
Space used (total): 43239419603
SSTable Compression Ratio: 0.2954468408497778
Number of Keys (estimate): 63729152
Memtable Columns Count: 1921620
Memtable Data Size: 257680020
Memtable Switch Count: 9
Read Count: 6167
Read Latency: NaN ms.
Write Count: 770984
Write Latency: 0.098 ms.
Pending Tasks: 0
Bloom Filter False Positives: 370
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 80103200
Compacted row minimum size: 180
Compacted row maximum size: 3311
Compacted row mean size: 2631
Average live cells per slice (last five minutes): 73.0
Average tombstones per slice (last five minutes): 13.0
On Mon, Jul 28, 2014 at 10:14 AM, Mark Reddy <ma...@boxever.com>> wrote:
What is your data size and number of columns in Cassandra. Do you do many deletions?
On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha <ru...@gmail.com>> wrote:
Really curious to know what's causing the spike in Columns and DeletedColums below :
2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class Histogram:
num #instances #bytes class name
----------------------------------------------
1: 132626060 6366050880<tel:6366050880> java.nio.HeapByteBuffer
2: 28194918 3920045528 [B
3: 78124737 3749987376 edu.stanford.ppl.concurrent.SnapTreeMap$Node
4: 67650128 2164804096<tel:2164804096> org.apache.cassandra.db.Column
5: 16315310 522089920 org.apache.cassandra.db.DeletedColumn
6: 6818 392489608 [I
7: 2844374 273059904 edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch
8: 5727000 229080000 java.util.TreeMap$Entry
9: 767742 182921376 [J
10: 2932832 140775936 edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder
11: 2844375 91020000 edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch
12: 4145131 66322096 java.util.concurrent.atomic.AtomicReference
13: 437874 64072392 [C
14: 2660844 63860256 java.util.concurrent.ConcurrentSkipListMap$Node
15: 4920 62849864 [[B
16: 1632063 52226016 edu.stanford.ppl.concurrent.SnapTreeMap
Re: Full GC in cassandra
Posted by Ruchir Jha <ru...@gmail.com>.
Also we do subsequent updates (atleat 4) for each piece of data that we
write.
On Mon, Jul 28, 2014 at 10:36 AM, Ruchir Jha <ru...@gmail.com> wrote:
> Doing about 5K writes / second. Avg Data Size = 1.6 TB / node. Total Data
> Size = 21 TB.
>
> And this is the nodetool cfstats output for one of our busiest column
> families:
>
> SSTable count: 10
> Space used (live): 43239294899
> Space used (total): 43239419603
> SSTable Compression Ratio: 0.2954468408497778
> Number of Keys (estimate): 63729152
> Memtable Columns Count: 1921620
> Memtable Data Size: 257680020
> Memtable Switch Count: 9
> Read Count: 6167
> Read Latency: NaN ms.
> Write Count: 770984
> Write Latency: 0.098 ms.
> Pending Tasks: 0
> Bloom Filter False Positives: 370
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 80103200
> Compacted row minimum size: 180
> Compacted row maximum size: 3311
> Compacted row mean size: 2631
> Average live cells per slice (last five minutes): 73.0
> Average tombstones per slice (last five minutes): 13.0
>
>
>
> On Mon, Jul 28, 2014 at 10:14 AM, Mark Reddy <ma...@boxever.com>
> wrote:
>
>> What is your data size and number of columns in Cassandra. Do you do many
>> deletions?
>>
>>
>> On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha <ru...@gmail.com> wrote:
>>
>>> Really curious to know what's causing the spike in Columns and
>>> DeletedColums below :
>>>
>>>
>>> 2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class
>>> Histogram:
>>> num #instances #bytes class name
>>> ----------------------------------------------
>>> 1: 132626060 6366050880 java.nio.HeapByteBuffer
>>> 2: 28194918 3920045528 [B
>>> 3: 78124737 3749987376
>>> edu.stanford.ppl.concurrent.SnapTreeMap$Node
>>> * 4: 67650128 2164804096
>>> <2164804096> org.apache.cassandra.db.Column*
>>> * 5: 16315310 522089920
>>> org.apache.cassandra.db.DeletedColumn*
>>> 6: 6818 392489608 [I
>>> 7: 2844374 273059904
>>> edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch
>>> 8: 5727000 229080000 java.util.TreeMap$Entry
>>> 9: 767742 182921376 [J
>>> 10: 2932832 140775936
>>> edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder
>>> 11: 2844375 91020000
>>> edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch
>>> 12: 4145131 66322096
>>> java.util.concurrent.atomic.AtomicReference
>>> 13: 437874 64072392 [C
>>> 14: 2660844 63860256
>>> java.util.concurrent.ConcurrentSkipListMap$Node
>>> 15: 4920 62849864 [[B
>>> 16: 1632063 52226016
>>> edu.stanford.ppl.concurrent.SnapTreeMap
>>>
>>
>>
>
Re: Full GC in cassandra
Posted by Ruchir Jha <ru...@gmail.com>.
Doing about 5K writes / second. Avg Data Size = 1.6 TB / node. Total Data
Size = 21 TB.
And this is the nodetool cfstats output for one of our busiest column
families:
SSTable count: 10
Space used (live): 43239294899
Space used (total): 43239419603
SSTable Compression Ratio: 0.2954468408497778
Number of Keys (estimate): 63729152
Memtable Columns Count: 1921620
Memtable Data Size: 257680020
Memtable Switch Count: 9
Read Count: 6167
Read Latency: NaN ms.
Write Count: 770984
Write Latency: 0.098 ms.
Pending Tasks: 0
Bloom Filter False Positives: 370
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 80103200
Compacted row minimum size: 180
Compacted row maximum size: 3311
Compacted row mean size: 2631
Average live cells per slice (last five minutes): 73.0
Average tombstones per slice (last five minutes): 13.0
On Mon, Jul 28, 2014 at 10:14 AM, Mark Reddy <ma...@boxever.com> wrote:
> What is your data size and number of columns in Cassandra. Do you do many
> deletions?
>
>
> On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha <ru...@gmail.com> wrote:
>
>> Really curious to know what's causing the spike in Columns and
>> DeletedColums below :
>>
>>
>> 2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class
>> Histogram:
>> num #instances #bytes class name
>> ----------------------------------------------
>> 1: 132626060 6366050880 java.nio.HeapByteBuffer
>> 2: 28194918 3920045528 [B
>> 3: 78124737 3749987376
>> edu.stanford.ppl.concurrent.SnapTreeMap$Node
>> * 4: 67650128 2164804096
>> <2164804096> org.apache.cassandra.db.Column*
>> * 5: 16315310 522089920
>> org.apache.cassandra.db.DeletedColumn*
>> 6: 6818 392489608 [I
>> 7: 2844374 273059904
>> edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch
>> 8: 5727000 229080000 java.util.TreeMap$Entry
>> 9: 767742 182921376 [J
>> 10: 2932832 140775936
>> edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder
>> 11: 2844375 91020000
>> edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch
>> 12: 4145131 66322096
>> java.util.concurrent.atomic.AtomicReference
>> 13: 437874 64072392 [C
>> 14: 2660844 63860256
>> java.util.concurrent.ConcurrentSkipListMap$Node
>> 15: 4920 62849864 [[B
>> 16: 1632063 52226016
>> edu.stanford.ppl.concurrent.SnapTreeMap
>>
>
>
Re: Full GC in cassandra
Posted by Mark Reddy <ma...@boxever.com>.
What is your data size and number of columns in Cassandra. Do you do many
deletions?
On Mon, Jul 28, 2014 at 2:53 PM, Ruchir Jha <ru...@gmail.com> wrote:
> Really curious to know what's causing the spike in Columns and
> DeletedColums below :
>
>
> 2014-07-28T09:30:27.471-0400: 127335.928: [Full GC 127335.928: [Class
> Histogram:
> num #instances #bytes class name
> ----------------------------------------------
> 1: 132626060 6366050880 java.nio.HeapByteBuffer
> 2: 28194918 3920045528 [B
> 3: 78124737 3749987376
> edu.stanford.ppl.concurrent.SnapTreeMap$Node
> * 4: 67650128 2164804096
> <2164804096> org.apache.cassandra.db.Column*
> * 5: 16315310 522089920 org.apache.cassandra.db.DeletedColumn*
> 6: 6818 392489608 [I
> 7: 2844374 273059904
> edu.stanford.ppl.concurrent.CopyOnWriteManager$COWEpoch
> 8: 5727000 229080000 java.util.TreeMap$Entry
> 9: 767742 182921376 [J
> 10: 2932832 140775936
> edu.stanford.ppl.concurrent.SnapTreeMap$RootHolder
> 11: 2844375 91020000
> edu.stanford.ppl.concurrent.CopyOnWriteManager$Latch
> 12: 4145131 66322096
> java.util.concurrent.atomic.AtomicReference
> 13: 437874 64072392 [C
> 14: 2660844 63860256
> java.util.concurrent.ConcurrentSkipListMap$Node
> 15: 4920 62849864 [[B
> 16: 1632063 52226016 edu.stanford.ppl.concurrent.SnapTreeMap
>