[Off heap memory used (total)] in cfstats

[Kai Wang <de...@gmail.com>]

Hi,

When I switch a big table from STCS to LCS, I notice high off heap memory
usage using nodetool cfstats. "*Off heap memory used (total)*" shows +10G
usage.

Eventually my nodes died because of OOM. How do I throttle off heap usage?
The only thing I see in cassandra.yaml is *memtable_offheap_space_in_mb. *Is
that the only knob?

Thanks.

Re: [Off heap memory used (total)] in cfstats

[Kai Wang <de...@gmail.com>]

Update:

I check the source code:
https://github.com/apache/cassandra/blob/4a0d1caa262af3b6f2b6d329e45766b4df845a88/src/java/org/apache/cassandra/tools/nodetool/Info.java#L154

Off heap memory usage comes from four parts. I rechecked cfstats to find
out Bloom filter takes most of the off heap memory. I increased
bloom_filter_fp_chance from the default 0.01 to 0.1 and restarted nodes.
The memory usage comes down now.

My table is kinda special in that it has lots of partitions but each
partition is very small (maximum <1M). I guess for such a table, Bloom
filter can be very large compared to the actual data size.


On Fri, Jan 8, 2016 at 6:39 PM, Kai Wang <de...@gmail.com> wrote:

> Hi,
>
> When I switch a big table from STCS to LCS, I notice high off heap memory
> usage using nodetool cfstats. "*Off heap memory used (total)*" shows +10G
> usage.
>
> Eventually my nodes died because of OOM. How do I throttle off heap usage?
> The only thing I see in cassandra.yaml is *memtable_offheap_space_in_mb. *Is
> that the only knob?
>
> Thanks.
>