Heap stuck at 98% while restarting node

[Philippe <wa...@gmail.com>]

While restarting a 1.1.12 node, I've run into this while it's replaying the
commit log:
GCInspector.java (line 145) Heap is 0.9778097528951517 full.  You may need
to reduce memtable and/or cache sizes.  Cassandra will now flush up to the
two largest memtables to free up memory.  Adjust flush_largest_memtables_at
threshold in cassandra.yaml if you don't want Cassandra to do this
automatically
I see it flush one column family but that's not enough as the CPU stays
locked at one thread running at 100%

Config : 3 nodes running 1.1.3 (x2) and one node running 1.1.12. No caches,
lots of column families.  memtable_total_space_in_mb set to 4096MB
Cassandra is started with a 32GB heap, yes I know it's too high but it
works on the other 2 nodes.

Can someone explain how I can "reduce memtable size" ? Any other ways of
getting this node to restart apart from increasing the HEAP SIZE ?

Thanks

Re: Heap stuck at 98% while restarting node

[aaron morton <aa...@thelastpickle.com>]

> Config : 3 nodes running 1.1.3 (x2) and one node running 1.1.12. No caches, lots of column families.  memtable_total_space_in_mb set to 4096MB
Reduce this to 2048 so it will flush to disk more frequently and avoid growing the heap so much. 

From what you've said I would be looking at the size of the row caches and the bloom filters. 

> Cassandra is started with a 32GB heap, yes I know it's too high but it works on the other 2 nodes.
Even hippies would say thats high http://www.youtube.com/watch?v=yoSwOrytf_M

Keep an eye on how much time is spent in GC. And remember that the memory you do not allocate to the JVM is memory that can be used to cache the files. 

Hope that helps. 

-----------------
Aaron Morton
Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 29/07/2013, at 2:25 AM, Philippe <wa...@gmail.com> wrote:

> While restarting a 1.1.12 node, I've run into this while it's replaying the commit log:
> GCInspector.java (line 145) Heap is 0.9778097528951517 full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
> I see it flush one column family but that's not enough as the CPU stays locked at one thread running at 100%
> 
> Config : 3 nodes running 1.1.3 (x2) and one node running 1.1.12. No caches, lots of column families.  memtable_total_space_in_mb set to 4096MB
> Cassandra is started with a 32GB heap, yes I know it's too high but it works on the other 2 nodes.
> 
> Can someone explain how I can "reduce memtable size" ? Any other ways of getting this node to restart apart from increasing the HEAP SIZE ?
> 
> Thanks
>