You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Alain RODRIGUEZ <ar...@gmail.com> on 2012/11/07 10:56:30 UTC
Questions around the heap
Hi,
We just had some issue in production that we finally solve upgrading
hardware and increasing the heap.
Now we have 3 xLarge servers from AWS (15G RAM, 4 cpu - 8 cores). We add
them and then removed the old ones.
With full default configuration, 0.75 threshold of 4G was being reach
continuously, so I was obliged to increase the heap to 8G:
Memtable : 2G (Manually configured)
Key cache : 0.1G (min(5% of Heap (in MB), 100MB))
System : 1G (more or less, from datastax doc)
It should use about 3 G and it actually use between 4 and 6 G.
So here are my questions:
How can we know how the heap is being used, monitor it ?
Why have I that much memory used in the heap of my new servers ?
All configurations not specified are default from 1.1.2 Cassandra.
Here is what happen to us before, why we change our hardware, if you have
any clue on what happen we would be glad to learn and maybe come back to
our old hardware.
-------------------------------- User experience
------------------------------------------------------------------------
We had a Cassandra 1.1.2 2 nodes cluster with RF2 and CL.ONE (R&W) running
on 2 m1.Large aws (7.5G RAM, 2 cpu - 4 cores dedicated to Cassandra only).
Cassandra.yaml was configured with 1.1.2 default options and in
cassandra-env.sh I configured a 4G heap with a 200M "new size".
That is the heap that was supposed to be used.
Memtable : 1.4G (1/3 of the heap)
Key cache : 0.1G (min(5% of Heap (in MB), 100MB))
System : 1G (more or less, from datastax doc)
So we are around 2.5G max in theory out of 3G usable (threshold 0.75 of the
heap before flushing memtable because of pressure)
I thought it was ok regarding Datastax documentation:
"Regardless of how much RAM your hardware has, you should keep the JVM heap
size constrained by the following formula and allow the operating system’s
file cache to do the rest:
(memtable_total_space_in_mb) + 1GB + (cache_size_estimate)"
After adding a third node and changing the RF from 2 to 3 (to allow using
CL.QUORUM and still be able to restart a node whenever we want), things
went really bad. Even if I still don't get how any of these operations
could possibly affect the heap needed.
All the 3 nodes reached the 0.75 heap threshold (I tried to increase it to
0.85, but it was still reached). And they never came down. So my cluster
started flushing a lot and the load increased because of
unceasing compactions. This unexpected load produced latency that broke
down our service for a while. Even with the service down, Cassandra was
unable to recover.
Re: Questions around the heap
Posted by aaron morton <aa...@thelastpickle.com>.
For background, this thread discusses the working for cassandra http://www.mail-archive.com/user@cassandra.apache.org/msg25762.html
tl;dr you can work it out or guess based on the tenured usage after CMS.
> How can we know how the heap is being used, monitor it ?
My favourite is to turn on the gc logging in cassandra-env.sh
I can also recommend the GC coverage in this book http://amzn.com/0137142528
You can also use JConsole or anything else that reads the JVM metrics via JMX.
> Why have I that much memory used in the heap of my new servers ?
IMHO the m1.xlarge is the best EC2 node (apart from ssd) to use.
> I configured a 4G heap with a 200M "new size".
That is a *very* low new heap size. I would expect it to result it frequent premature promotion into the tenured heap. Which will make it look like you are using more memory.
> That is the heap that was supposed to be used.
>
> Memtable : 1.4G (1/3 of the heap)
> Key cache : 0.1G (min(5% of Heap (in MB), 100MB))
> System : 1G (more or less, from datastax doc)
>
> So we are around 2.5G max in theory out of 3G usable (threshold 0.75 of the heap before flushing memtable because of pressure)
The memtable usage is the maxium value, if all the memtables are full and the flush queue is full. It's not the working size used for memtables. The code tries to avoid ever hitting the maximum.
Not sure if the 1G for "system" is still current or what it's actually referring to.
I suggest:
* returning the configuration to the defaults.
* if you have a high number of rows looking at the working set calculations linked above.
* monitoring the servers to look for triggers for the GC activity, such as compaction or repair
* looking at your code base for read queries that read a lot of data. May be write but it's often read.
* if you are using default compaction strategy, looking at the data model rows that have a high number of deletes and or overwrites over a longtime. These can have a high tombstone count.
GC activity is relative to the workload. Try to find things that cause a lot of columns to be read from disk.
I've found the following JVM tweeks sometimes helpful:
MAX_HEAP_SIZE="4G"
HEAP_NEWSIZE="1200M"
SurvivorRatio=4
MaxTenuringThreshold=4
Hope that helps.
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 12/11/2012, at 10:26 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> It's been Does anybody has an answer to any of these questions ?
>
> Alain
>
>
> 2012/11/7 Hiller, Dean <De...@nrel.gov>
> +1, I am interested in this answer as well.
>
> From: Alain RODRIGUEZ <ar...@gmail.com>>
> Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
> Date: Wednesday, November 7, 2012 9:45 AM
> To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
> Subject: Re: Questions around the heap
>
> s application that heavily scans a particular column family, you would want to inhibit or disable the Bloom filter on the column family by setting it high"
>
Re: Questions around the heap
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
It's been Does anybody has an answer to any of these questions ?
Alain
2012/11/7 Hiller, Dean <De...@nrel.gov>
> +1, I am interested in this answer as well.
>
> From: Alain RODRIGUEZ <ar...@gmail.com>>
> Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org>>
> Date: Wednesday, November 7, 2012 9:45 AM
> To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <
> user@cassandra.apache.org<ma...@cassandra.apache.org>>
> Subject: Re: Questions around the heap
>
> s application that heavily scans a particular column family, you would
> want to inhibit or disable the Bloom filter on the column family by setting
> it high"
>
Re: Questions around the heap
Posted by "Hiller, Dean" <De...@nrel.gov>.
+1, I am interested in this answer as well.
From: Alain RODRIGUEZ <ar...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Wednesday, November 7, 2012 9:45 AM
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Questions around the heap
s application that heavily scans a particular column family, you would want to inhibit or disable the Bloom filter on the column family by setting it high"
Re: Questions around the heap
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
I have to say that I have no idea on how to tune them.
I discover the existence of bloom filters a few month ago and even after
reading http://wiki.apache.org/cassandra/ArchitectureOverview#line-132 and
http://spyced.blogspot.com/2009/01/all-you-ever-wanted-to-know-about.html I
am not sure what would be the impacts (positives and negatives) of tuning
the bloom filters.
>From my reads I understand that with a bloom_filter_fp_chance > 0 I
introduce a chance to get a false positive from a SSTable inducing
eventually more latency while answering queries but using less memory. Is
that right ?
"What are your bloom filter settings on your CFs?"
They are default (0 - which seems to mean fully enabled
http://www.datastax.com/docs/1.1/configuration/storage_configuration#bloom-filter-fp-chance
)
Cant they grow indefinitely or is there a threshold?
Is there a way to "explore" the heap to be sure that bloom filters are
causing this intensive use of the memory inside the heap before tuning them?
>From http://www.datastax.com/docs/1.1/operations/tuning#tuning-bloomfilters
:
"For example, to run an analytics application that heavily scans a
particular column family, you would want to inhibit or disable the Bloom
filter on the column family by setting it high"
Why would I do that, won't it slow the display of analytics?
Alain
2012/11/7 Bryan <br...@appssavvy.com>
> What are your bloom filter settings on your CFs? Maybe look here:
> http://www.datastax.com/docs/1.1/operations/tuning#tuning-bloomfilters
>
>
>
> On Nov 7, 2012, at 4:56 AM, Alain RODRIGUEZ wrote:
>
> Hi,
>
> We just had some issue in production that we finally solve upgrading
> hardware and increasing the heap.
>
> Now we have 3 xLarge servers from AWS (15G RAM, 4 cpu - 8 cores). We add
> them and then removed the old ones.
>
> With full default configuration, 0.75 threshold of 4G was being reach
> continuously, so I was obliged to increase the heap to 8G:
>
> Memtable : 2G (Manually configured)
> Key cache : 0.1G (min(5% of Heap (in MB), 100MB))
> System : 1G (more or less, from datastax doc)
>
> It should use about 3 G and it actually use between 4 and 6 G.
>
> So here are my questions:
>
> How can we know how the heap is being used, monitor it ?
> Why have I that much memory used in the heap of my new servers ?
>
> All configurations not specified are default from 1.1.2 Cassandra.
>
> Here is what happen to us before, why we change our hardware, if you have
> any clue on what happen we would be glad to learn and maybe come back to
> our old hardware.
>
> -------------------------------- User experience
> ------------------------------------------------------------------------
>
> We had a Cassandra 1.1.2 2 nodes cluster with RF2 and CL.ONE (R&W) running
> on 2 m1.Large aws (7.5G RAM, 2 cpu - 4 cores dedicated to Cassandra only).
>
> Cassandra.yaml was configured with 1.1.2 default options and in
> cassandra-env.sh I configured a 4G heap with a 200M "new size".
>
> That is the heap that was supposed to be used.
>
> Memtable : 1.4G (1/3 of the heap)
> Key cache : 0.1G (min(5% of Heap (in MB), 100MB))
> System : 1G (more or less, from datastax doc)
>
> So we are around 2.5G max in theory out of 3G usable (threshold 0.75 of
> the heap before flushing memtable because of pressure)
>
> I thought it was ok regarding Datastax documentation:
>
> "Regardless of how much RAM your hardware has, you should keep the JVM
> heap size constrained by the following formula and allow the operating
> system’s file cache to do the rest:
>
> (memtable_total_space_in_mb) + 1GB + (cache_size_estimate)"
> After adding a third node and changing the RF from 2 to 3 (to allow using
> CL.QUORUM and still be able to restart a node whenever we want), things
> went really bad. Even if I still don't get how any of these operations
> could possibly affect the heap needed.
>
> All the 3 nodes reached the 0.75 heap threshold (I tried to increase it to
> 0.85, but it was still reached). And they never came down. So my cluster
> started flushing a lot and the load increased because of
> unceasing compactions. This unexpected load produced latency that broke
> down our service for a while. Even with the service down, Cassandra was
> unable to recover.
>
>
>
Re: Questions around the heap
Posted by Bryan <br...@appssavvy.com>.
What are your bloom filter settings on your CFs? Maybe look here: http://www.datastax.com/docs/1.1/operations/tuning#tuning-bloomfilters
On Nov 7, 2012, at 4:56 AM, Alain RODRIGUEZ wrote:
> Hi,
>
> We just had some issue in production that we finally solve upgrading hardware and increasing the heap.
>
> Now we have 3 xLarge servers from AWS (15G RAM, 4 cpu - 8 cores). We add them and then removed the old ones.
>
> With full default configuration, 0.75 threshold of 4G was being reach continuously, so I was obliged to increase the heap to 8G:
>
> Memtable : 2G (Manually configured)
> Key cache : 0.1G (min(5% of Heap (in MB), 100MB))
> System : 1G (more or less, from datastax doc)
>
> It should use about 3 G and it actually use between 4 and 6 G.
>
> So here are my questions:
>
> How can we know how the heap is being used, monitor it ?
> Why have I that much memory used in the heap of my new servers ?
>
> All configurations not specified are default from 1.1.2 Cassandra.
>
> Here is what happen to us before, why we change our hardware, if you have any clue on what happen we would be glad to learn and maybe come back to our old hardware.
>
> -------------------------------- User experience ------------------------------------------------------------------------
>
> We had a Cassandra 1.1.2 2 nodes cluster with RF2 and CL.ONE (R&W) running on 2 m1.Large aws (7.5G RAM, 2 cpu - 4 cores dedicated to Cassandra only).
>
> Cassandra.yaml was configured with 1.1.2 default options and in cassandra-env.sh I configured a 4G heap with a 200M "new size".
>
> That is the heap that was supposed to be used.
>
> Memtable : 1.4G (1/3 of the heap)
> Key cache : 0.1G (min(5% of Heap (in MB), 100MB))
> System : 1G (more or less, from datastax doc)
>
> So we are around 2.5G max in theory out of 3G usable (threshold 0.75 of the heap before flushing memtable because of pressure)
>
> I thought it was ok regarding Datastax documentation:
>
> "Regardless of how much RAM your hardware has, you should keep the JVM heap size constrained by the following formula and allow the operating system’s file cache to do the rest:
> (memtable_total_space_in_mb) + 1GB + (cache_size_estimate)"
>
> After adding a third node and changing the RF from 2 to 3 (to allow using CL.QUORUM and still be able to restart a node whenever we want), things went really bad. Even if I still don't get how any of these operations could possibly affect the heap needed.
>
> All the 3 nodes reached the 0.75 heap threshold (I tried to increase it to 0.85, but it was still reached). And they never came down. So my cluster started flushing a lot and the load increased because of unceasing compactions. This unexpected load produced latency that broke down our service for a while. Even with the service down, Cassandra was unable to recover.
>