You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Thakrar, Jayesh" <jt...@conversantmedia.com> on 2017/03/04 20:54:05 UTC

Any way to control/limit off-heap memory?

I have a situation where the off-heap memory is bloating the jvm process memory, making it a candidate to be killed by the oom_killer.
My server has 256 GB RAM and Cassandra heap memory of 16 GB

Below is the output of "nodetool info" and nodetool compactionstats for a culprit table which causes bloom filter bloat.
Ofcourse one option is to turnoff bloom filter, but I need to look into application access pattern, etc.


xss =  -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp -javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
ID                     : 2b9b4252-0760-49c1-8d14-544be0183271
Gossip active          : true
Thrift active          : false
Native Transport active: true
Load                   : 953.19 GB
Generation No          : 1488641545
Uptime (seconds)       : 15706
Heap Memory (MB)       : 7692.93 / 16309.00
Off Heap Memory (MB)   : 175115.07
Data Center            : ord
Rack                   : rack3
Exceptions             : 0
Key Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Token                  : (invoke with -T/--tokens to see all 256 tokens)


                Table: logs_by_user
                SSTable count: 622
                SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0]
                Space used (live): 313156769247
                Space used (total): 313156769247
                Space used by snapshots (total): 0
                Off heap memory used (total): 180354511884
                SSTable Compression Ratio: 0.25016314078395613
                Number of keys (estimate): 147261312
                Memtable cell count: 44796
                Memtable data size: 57578717
                Memtable off heap memory used: 0
                Memtable switch count: 21
                Local read count: 0
                Local read latency: NaN ms
                Local write count: 1148687
                Local write latency: 0.123 ms
                Pending flushes: 0
                Bloom filter false positives: 0
                Bloom filter false ratio: 0.00000
                Bloom filter space used: 180269125192
                Bloom filter off heap memory used: 180269120216
                Index summary off heap memory used: 24335340
                Compression metadata off heap memory used: 61056328
                Compacted partition minimum bytes: 150
                Compacted partition maximum bytes: 668489532
                Compacted partition mean bytes: 3539
                Average live cells per slice (last five minutes): NaN
                Maximum live cells per slice (last five minutes): 0
                Average tombstones per slice (last five minutes): NaN
                Maximum tombstones per slice (last five minutes): 0


From: Conversant <jt...@conversantmedia.com>

Re: Any way to control/limit off-heap memory?

Posted by Jeff Jirsa <jj...@apache.org>.

On 2017-03-06 07:04 (-0800), "Thakrar, Jayesh" <jt...@conversantmedia.com> wrote: 
> Thanks Hannu - also considered that option.
> However, that's a trial and error and will have to play with the collision/false-positive fraction.
> And each iteration will most likely result in a compaction storm - so I was hoping for a way to throttle/limit the max off-heap size.
> 
> The reason I was thinking of eliminating bloom filters is because due to application design, we search for data using a partial key (prefix columns),
> hence am thinking of completely eliminating the bloom filters as they do not add any value in such a use case.
> 

If you dont want to use the bloom filters, don't set the FP ratio to 0, set it to something like 0.1 or 0.5. A fp ratio of 0 says "no false positives", which is only possible with HUGE bloom filters. A high FP ratio (since you're not using them) basically says "don't try very hard" which corresponds to small arrays, which means low accuracy, but low offheap usage. 

We probably shouldn't even allow 0.0000 FP ratio. 




Re: Any way to control/limit off-heap memory?

Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
Thanks Hannu - also considered that option.
However, that's a trial and error and will have to play with the collision/false-positive fraction.
And each iteration will most likely result in a compaction storm - so I was hoping for a way to throttle/limit the max off-heap size.

The reason I was thinking of eliminating bloom filters is because due to application design, we search for data using a partial key (prefix columns),
hence am thinking of completely eliminating the bloom filters as they do not add any value in such a use case.

Is my assumption correct?

From: Hannu Kröger <hk...@gmail.com>
Date: Sunday, March 5, 2017 at 6:34 AM
To: <us...@cassandra.apache.org>
Subject: Re: Any way to control/limit off-heap memory?

If bloom filters are taking too much memory, you can adjust bloom filters:
https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_tuning_bloom_filters_c.html

Hannu

On 4 Mar 2017, at 22:54, Thakrar, Jayesh <jt...@conversantmedia.com>> wrote:

I have a situation where the off-heap memory is bloating the jvm process memory, making it a candidate to be killed by the oom_killer.
My server has 256 GB RAM and Cassandra heap memory of 16 GB

Below is the output of "nodetool info" and nodetool compactionstats for a culprit table which causes bloom filter bloat.
Ofcourse one option is to turnoff bloom filter, but I need to look into application access pattern, etc.


xss =  -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp -javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
ID                     : 2b9b4252-0760-49c1-8d14-544be0183271
Gossip active          : true
Thrift active          : false
Native Transport active: true
Load                   : 953.19 GB
Generation No          : 1488641545
Uptime (seconds)       : 15706
Heap Memory (MB)       : 7692.93 / 16309.00
Off Heap Memory (MB)   : 175115.07
Data Center            : ord
Rack                   : rack3
Exceptions             : 0
Key Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Token                  : (invoke with -T/--tokens to see all 256 tokens)


                Table: logs_by_user
                SSTable count: 622
                SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0]
                Space used (live): 313156769247
                Space used (total): 313156769247
                Space used by snapshots (total): 0
                Off heap memory used (total): 180354511884
                SSTable Compression Ratio: 0.25016314078395613
                Number of keys (estimate): 147261312
                Memtable cell count: 44796
                Memtable data size: 57578717
                Memtable off heap memory used: 0
                Memtable switch count: 21
                Local read count: 0
                Local read latency: NaN ms
                Local write count: 1148687
                Local write latency: 0.123 ms
                Pending flushes: 0
                Bloom filter false positives: 0
                Bloom filter false ratio: 0.00000
                Bloom filter space used: 180269125192
                Bloom filter off heap memory used: 180269120216
                Index summary off heap memory used: 24335340
                Compression metadata off heap memory used: 61056328
                Compacted partition minimum bytes: 150
                Compacted partition maximum bytes: 668489532
                Compacted partition mean bytes: 3539
                Average live cells per slice (last five minutes): NaN
                Maximum live cells per slice (last five minutes): 0
                Average tombstones per slice (last five minutes): NaN
                Maximum tombstones per slice (last five minutes): 0


From: Conversant <jt...@conversantmedia.com>


Re: Any way to control/limit off-heap memory?

Posted by Hannu Kröger <hk...@gmail.com>.
If bloom filters are taking too much memory, you can adjust bloom filters:
https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_tuning_bloom_filters_c.html <https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_tuning_bloom_filters_c.html>

Hannu

> On 4 Mar 2017, at 22:54, Thakrar, Jayesh <jt...@conversantmedia.com> wrote:
> 
> I have a situation where the off-heap memory is bloating the jvm process memory, making it a candidate to be killed by the oom_killer.
> My server has 256 GB RAM and Cassandra heap memory of 16 GB
>  
> Below is the output of "nodetool info" and nodetool compactionstats for a culprit table which causes bloom filter bloat.
> Ofcourse one option is to turnoff bloom filter, but I need to look into application access pattern, etc.
>  
>  
> xss =  -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp -javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
> ID                     : 2b9b4252-0760-49c1-8d14-544be0183271
> Gossip active          : true
> Thrift active          : false
> Native Transport active: true
> Load                   : 953.19 GB
> Generation No          : 1488641545
> Uptime (seconds)       : 15706
> Heap Memory (MB)       : 7692.93 / 16309.00
> Off Heap Memory (MB)   : 175115.07
> Data Center            : ord
> Rack                   : rack3
> Exceptions             : 0
> Key Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 14400 save period in seconds
> Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
> Counter Cache          : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
> Token                  : (invoke with -T/--tokens to see all 256 tokens)
>  
>  
>                 Table: logs_by_user
>                 SSTable count: 622
>                 SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0]
>                 Space used (live): 313156769247
>                 Space used (total): 313156769247
>                 Space used by snapshots (total): 0
>                 Off heap memory used (total): 180354511884
>                 SSTable Compression Ratio: 0.25016314078395613
>                 Number of keys (estimate): 147261312
>                 Memtable cell count: 44796
>                 Memtable data size: 57578717
>                 Memtable off heap memory used: 0
>                 Memtable switch count: 21
>                 Local read count: 0
>                 Local read latency: NaN ms
>                 Local write count: 1148687
>                 Local write latency: 0.123 ms
>                 Pending flushes: 0
>                 Bloom filter false positives: 0
>                 Bloom filter false ratio: 0.00000
>                 Bloom filter space used: 180269125192
>                 Bloom filter off heap memory used: 180269120216
>                 Index summary off heap memory used: 24335340
>                 Compression metadata off heap memory used: 61056328
>                 Compacted partition minimum bytes: 150
>                 Compacted partition maximum bytes: 668489532
>                 Compacted partition mean bytes: 3539
>                 Average live cells per slice (last five minutes): NaN
>                 Maximum live cells per slice (last five minutes): 0
>                 Average tombstones per slice (last five minutes): NaN
>                 Maximum tombstones per slice (last five minutes): 0
>  
>  
> From: Conversant <jthak <ma...@conversantmedia.com>