You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by "Thakrar, Jayesh" <jt...@conversantmedia.com> on 2017/03/04 20:54:05 UTC
Any way to control/limit off-heap memory?
I have a situation where the off-heap memory is bloating the jvm process memory, making it a candidate to be killed by the oom_killer.
My server has 256 GB RAM and Cassandra heap memory of 16 GB
Below is the output of "nodetool info" and nodetool compactionstats for a culprit table which causes bloom filter bloat.
Ofcourse one option is to turnoff bloom filter, but I need to look into application access pattern, etc.
xss = -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp -javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
ID : 2b9b4252-0760-49c1-8d14-544be0183271
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 953.19 GB
Generation No : 1488641545
Uptime (seconds) : 15706
Heap Memory (MB) : 7692.93 / 16309.00
Off Heap Memory (MB) : 175115.07
Data Center : ord
Rack : rack3
Exceptions : 0
Key Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Token : (invoke with -T/--tokens to see all 256 tokens)
Table: logs_by_user
SSTable count: 622
SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 313156769247
Space used (total): 313156769247
Space used by snapshots (total): 0
Off heap memory used (total): 180354511884
SSTable Compression Ratio: 0.25016314078395613
Number of keys (estimate): 147261312
Memtable cell count: 44796
Memtable data size: 57578717
Memtable off heap memory used: 0
Memtable switch count: 21
Local read count: 0
Local read latency: NaN ms
Local write count: 1148687
Local write latency: 0.123 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 180269125192
Bloom filter off heap memory used: 180269120216
Index summary off heap memory used: 24335340
Compression metadata off heap memory used: 61056328
Compacted partition minimum bytes: 150
Compacted partition maximum bytes: 668489532
Compacted partition mean bytes: 3539
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
From: Conversant <jt...@conversantmedia.com>
Re: Any way to control/limit off-heap memory?
Posted by Jeff Jirsa <jj...@apache.org>.
On 2017-03-06 07:04 (-0800), "Thakrar, Jayesh" <jt...@conversantmedia.com> wrote:
> Thanks Hannu - also considered that option.
> However, that's a trial and error and will have to play with the collision/false-positive fraction.
> And each iteration will most likely result in a compaction storm - so I was hoping for a way to throttle/limit the max off-heap size.
>
> The reason I was thinking of eliminating bloom filters is because due to application design, we search for data using a partial key (prefix columns),
> hence am thinking of completely eliminating the bloom filters as they do not add any value in such a use case.
>
If you dont want to use the bloom filters, don't set the FP ratio to 0, set it to something like 0.1 or 0.5. A fp ratio of 0 says "no false positives", which is only possible with HUGE bloom filters. A high FP ratio (since you're not using them) basically says "don't try very hard" which corresponds to small arrays, which means low accuracy, but low offheap usage.
We probably shouldn't even allow 0.0000 FP ratio.
Re: Any way to control/limit off-heap memory?
Posted by "Thakrar, Jayesh" <jt...@conversantmedia.com>.
Thanks Hannu - also considered that option.
However, that's a trial and error and will have to play with the collision/false-positive fraction.
And each iteration will most likely result in a compaction storm - so I was hoping for a way to throttle/limit the max off-heap size.
The reason I was thinking of eliminating bloom filters is because due to application design, we search for data using a partial key (prefix columns),
hence am thinking of completely eliminating the bloom filters as they do not add any value in such a use case.
Is my assumption correct?
From: Hannu Kröger <hk...@gmail.com>
Date: Sunday, March 5, 2017 at 6:34 AM
To: <us...@cassandra.apache.org>
Subject: Re: Any way to control/limit off-heap memory?
If bloom filters are taking too much memory, you can adjust bloom filters:
https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_tuning_bloom_filters_c.html
Hannu
On 4 Mar 2017, at 22:54, Thakrar, Jayesh <jt...@conversantmedia.com>> wrote:
I have a situation where the off-heap memory is bloating the jvm process memory, making it a candidate to be killed by the oom_killer.
My server has 256 GB RAM and Cassandra heap memory of 16 GB
Below is the output of "nodetool info" and nodetool compactionstats for a culprit table which causes bloom filter bloat.
Ofcourse one option is to turnoff bloom filter, but I need to look into application access pattern, etc.
xss = -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp -javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
ID : 2b9b4252-0760-49c1-8d14-544be0183271
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 953.19 GB
Generation No : 1488641545
Uptime (seconds) : 15706
Heap Memory (MB) : 7692.93 / 16309.00
Off Heap Memory (MB) : 175115.07
Data Center : ord
Rack : rack3
Exceptions : 0
Key Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Token : (invoke with -T/--tokens to see all 256 tokens)
Table: logs_by_user
SSTable count: 622
SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 313156769247
Space used (total): 313156769247
Space used by snapshots (total): 0
Off heap memory used (total): 180354511884
SSTable Compression Ratio: 0.25016314078395613
Number of keys (estimate): 147261312
Memtable cell count: 44796
Memtable data size: 57578717
Memtable off heap memory used: 0
Memtable switch count: 21
Local read count: 0
Local read latency: NaN ms
Local write count: 1148687
Local write latency: 0.123 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.00000
Bloom filter space used: 180269125192
Bloom filter off heap memory used: 180269120216
Index summary off heap memory used: 24335340
Compression metadata off heap memory used: 61056328
Compacted partition minimum bytes: 150
Compacted partition maximum bytes: 668489532
Compacted partition mean bytes: 3539
Average live cells per slice (last five minutes): NaN
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): NaN
Maximum tombstones per slice (last five minutes): 0
From: Conversant <jt...@conversantmedia.com>
Re: Any way to control/limit off-heap memory?
Posted by Hannu Kröger <hk...@gmail.com>.
If bloom filters are taking too much memory, you can adjust bloom filters:
https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_tuning_bloom_filters_c.html <https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_tuning_bloom_filters_c.html>
Hannu
> On 4 Mar 2017, at 22:54, Thakrar, Jayesh <jt...@conversantmedia.com> wrote:
>
> I have a situation where the off-heap memory is bloating the jvm process memory, making it a candidate to be killed by the oom_killer.
> My server has 256 GB RAM and Cassandra heap memory of 16 GB
>
> Below is the output of "nodetool info" and nodetool compactionstats for a culprit table which causes bloom filter bloat.
> Ofcourse one option is to turnoff bloom filter, but I need to look into application access pattern, etc.
>
>
> xss = -ea -Dorg.xerial.snappy.tempdir=/home/vchadoop/var/tmp -javaagent:/home/vchadoop/apps/apache-cassandra-2.2.5//lib/jamm-0.3.0.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms16G -Xmx16G -Xmn4800M -XX:+HeapDumpOnOutOfMemoryError -Xss256k
> ID : 2b9b4252-0760-49c1-8d14-544be0183271
> Gossip active : true
> Thrift active : false
> Native Transport active: true
> Load : 953.19 GB
> Generation No : 1488641545
> Uptime (seconds) : 15706
> Heap Memory (MB) : 7692.93 / 16309.00
> Off Heap Memory (MB) : 175115.07
> Data Center : ord
> Rack : rack3
> Exceptions : 0
> Key Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 14400 save period in seconds
> Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
> Counter Cache : entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
> Token : (invoke with -T/--tokens to see all 256 tokens)
>
>
> Table: logs_by_user
> SSTable count: 622
> SSTables in each level: [174/4, 447/10, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 313156769247
> Space used (total): 313156769247
> Space used by snapshots (total): 0
> Off heap memory used (total): 180354511884
> SSTable Compression Ratio: 0.25016314078395613
> Number of keys (estimate): 147261312
> Memtable cell count: 44796
> Memtable data size: 57578717
> Memtable off heap memory used: 0
> Memtable switch count: 21
> Local read count: 0
> Local read latency: NaN ms
> Local write count: 1148687
> Local write latency: 0.123 ms
> Pending flushes: 0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.00000
> Bloom filter space used: 180269125192
> Bloom filter off heap memory used: 180269120216
> Index summary off heap memory used: 24335340
> Compression metadata off heap memory used: 61056328
> Compacted partition minimum bytes: 150
> Compacted partition maximum bytes: 668489532
> Compacted partition mean bytes: 3539
> Average live cells per slice (last five minutes): NaN
> Maximum live cells per slice (last five minutes): 0
> Average tombstones per slice (last five minutes): NaN
> Maximum tombstones per slice (last five minutes): 0
>
>
> From: Conversant <jthak <ma...@conversantmedia.com>