You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Huy Le <hu...@springpartners.com> on 2011/02/09 17:15:01 UTC
Out of control memory consumption
Hi,
There is already an email thread on memory issue on this email list, but I
creating a new thread as we are experiencing a different memory consumption
issue.
We are 12-server cluster. We use random partitioner with manually generated
server tokens. Memory usage on one server keeps growing out of control. We
ran flush and cleared key and row caches but and ran GC but heap memory
usage won't go down. The only way to heap memory usage to go down is the
restart cassandra. We have to do this one a day. All other servers have
heap memory usage less than 500MB. This issue happened on both Cassandra
0.6.6 and 0.6.11.
Our JVM info:
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
And JVM memory allocation: -Xms3G -Xmx3G
Non-heap memory usage is 138MB.
Any recommendation where should look to see why memory usage keep growing?
Thanks!
Huy
Re: Out of control memory consumption
Posted by Huy Le <hu...@springpartners.com>.
>
> If the heap usages continues to grow an OOM will eventually be thrown.
> Are you experiencing OOMs on these boxes? If you are not OOMing, then
> what problem are you experiencing (excessive CPU use garbage collection
> for one example)?
>
>
>
No OOM. The JVM just too busy doing GC when the used heap size is big
making this node unresponsive to its peers on the cluster.
>
> > Our JVM info:
> >
> > java version "1.6.0_21"
> > Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
> > Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
> >
> > And JVM memory allocation: -Xms3G -Xmx3G
> >
> > Non-heap memory usage is 138MB.
> >
> > Any recommendation where should look to see why memory usage keep
> growing?
> >
> > Thanks!
> >
> > Huy
>
> Are you using standard, mmap_index_only, or mmap io? Are you using JNA?
>
>
We use standard disk access mode with JNA.
Huy
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: Out of control memory consumption
Posted by Chris Burroughs <ch...@gmail.com>.
On 02/09/2011 11:15 AM, Huy Le wrote:
> There is already an email thread on memory issue on this email list, but I
> creating a new thread as we are experiencing a different memory consumption
> issue.
>
> We are 12-server cluster. We use random partitioner with manually generated
> server tokens. Memory usage on one server keeps growing out of control. We
> ran flush and cleared key and row caches but and ran GC but heap memory
> usage won't go down. The only way to heap memory usage to go down is the
> restart cassandra. We have to do this one a day. All other servers have
> heap memory usage less than 500MB. This issue happened on both Cassandra
> 0.6.6 and 0.6.11.
>
If the heap usages continues to grow an OOM will eventually be thrown.
Are you experiencing OOMs on these boxes? If you are not OOMing, then
what problem are you experiencing (excessive CPU use garbage collection
for one example)?
> Our JVM info:
>
> java version "1.6.0_21"
> Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)
>
> And JVM memory allocation: -Xms3G -Xmx3G
>
> Non-heap memory usage is 138MB.
>
> Any recommendation where should look to see why memory usage keep growing?
>
> Thanks!
>
> Huy
Are you using standard, mmap_index_only, or mmap io? Are you using JNA?
Re: Out of control memory consumption
Posted by Jonathan Ellis <jb...@gmail.com>.
Sounds like you need to use a larger heap or put less stuff in it
(memtables, caches).
On Thu, Feb 10, 2011 at 11:17 AM, Huy Le <hu...@springpartners.com> wrote:
> Yes, we had setting at 75 but JVM did not have enough time to do GC, so it
> abort GC'ing. We lowered it to 50, but still had issue, so we lowered it
> again to 35.
>
> On Thu, Feb 10, 2011 at 12:11 PM, Oleg Anastasyev <ol...@gmail.com>
> wrote:
>>
>> Huy Le <huyle <at> springpartners.com> writes:
>>
>> > Our CMS settings are: -XX:CMSInitiatingOccupancyFraction=35
>> > \
>> -XX:+UseCMSInitiatingOccupancyOnly \
>> >
>>
>> Occupancy Fraction = 35 seems very low value. You instructed GC to make
>> collection as soon as memory usage is at 35% - i.e. about 1G. This seems
>> very
>> low for production.
>> Did you tried another setting for this parameter ? 75 seems safe one.
>>
>
>
>
> --
> Huy Le
> Spring Partners, Inc.
> http://springpadit.com
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Out of control memory consumption
Posted by Robert Coli <rc...@digg.com>.
On Thu, Feb 10, 2011 at 9:17 AM, Huy Le <hu...@springpartners.com> wrote:
> Yes, we had setting at 75 but JVM did not have enough time to do GC, so it
> abort GC'ing. We lowered it to 50, but still had issue, so we lowered it
> again to 35.
If you lower CMSInitiatingOccupancyFraction below the size of your
typical working set, the only thing you will do is burn a core's worth
of CPU on constant GC, which is probably not accompanied by an
improvement in overall system performance. ;D
=Rob
Re: Out of control memory consumption
Posted by Huy Le <hu...@springpartners.com>.
Yes, we had setting at 75 but JVM did not have enough time to do GC, so it
abort GC'ing. We lowered it to 50, but still had issue, so we lowered it
again to 35.
On Thu, Feb 10, 2011 at 12:11 PM, Oleg Anastasyev <ol...@gmail.com>wrote:
> Huy Le <huyle <at> springpartners.com> writes:
>
> > Our CMS settings are: -XX:CMSInitiatingOccupancyFraction=35
> \
> -XX:+UseCMSInitiatingOccupancyOnly \
> >
>
> Occupancy Fraction = 35 seems very low value. You instructed GC to make
> collection as soon as memory usage is at 35% - i.e. about 1G. This seems
> very
> low for production.
> Did you tried another setting for this parameter ? 75 seems safe one.
>
>
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: Out of control memory consumption
Posted by Oleg Anastasyev <ol...@gmail.com>.
Huy Le <huyle <at> springpartners.com> writes:
> Our CMS settings are: -XX:CMSInitiatingOccupancyFraction=35 \
-XX:+UseCMSInitiatingOccupancyOnly \
>
Occupancy Fraction = 35 seems very low value. You instructed GC to make
collection as soon as memory usage is at 35% - i.e. about 1G. This seems very
low for production.
Did you tried another setting for this parameter ? 75 seems safe one.
Re: Out of control memory consumption
Posted by Huy Le <hu...@springpartners.com>.
We use Cassandra version 0.6.11. Our cache size is very small. 11 out of 12
servers have used heap size less than 500MB of 3GB allocated. Just one
server that had memory usage run out of control.
The issue is isolated. It turn out that one CF has a row with compacted row
size of 50MB. And this row happens to be very active row that get updated
every few seconds. By deleting this row, heap memory usage is consistently
low across servers on the cluster.
It came to my surprise how expensive it is to have this row of 50MB
compacted size being updated every few seconds. Does anyone have any
insight as to why the row mentioned is so expensive to have?
Thanks!
Huy
On Wed, Feb 9, 2011 at 2:34 PM, Robert Coli <rc...@digg.com> wrote:
> On Wed, Feb 9, 2011 at 11:04 AM, Huy Le <hu...@springpartners.com> wrote:
> > Memory usage grows overtime.
>
> It is relatively typical for caches to exert memory pressure over time
> as they fill. What are your cache settings, for how many
> columnfamilies, and with what sized memtables? What version of
> Cassandra?
>
> =Rob
>
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: Out of control memory consumption
Posted by Robert Coli <rc...@digg.com>.
On Wed, Feb 9, 2011 at 11:04 AM, Huy Le <hu...@springpartners.com> wrote:
> Memory usage grows overtime.
It is relatively typical for caches to exert memory pressure over time
as they fill. What are your cache settings, for how many
columnfamilies, and with what sized memtables? What version of
Cassandra?
=Rob
Re: Out of control memory consumption
Posted by Huy Le <hu...@springpartners.com>.
>
> To be clear: You are not talking about the size of the Java process in
> top, but the actual amount of heap used as reported by the JVM via
> jmx/jconsole/etc?
>
> This is memory usage shows in JMX that we are talking about.
> Is the memory amount of memory that you consider high, the heap size
> just after a concurrent mark/sweep?
>
>
Memory usage grows overtime.
> Are you actually seeing OOM:s or are you restarting the node
> pre-emptively in response to seeing heap usage go up?
>
>
>
No OOM. We pre-emptively restart it before it become unresponsive due to
GC.
> > And JVM memory allocation: -Xms3G -Xmx3G
>
> Just FYI: So it is entirely expected that the JVM will be 3G (a bit
> higher) in size (even with standard I/O) and further that the amount
> of live data in the heap be approaching 3G. The concurrent mark/sweep
> GC won't trigger until the initial occupancy reaches the limit (if
> modern Cassandra with default settings).
>
>
Our CMS settings are:
-XX:CMSInitiatingOccupancyFraction=35 \
-XX:+UseCMSInitiatingOccupancyOnly \
> If you've got a 3 gig heap size and the other nodes stay at 500 mb,
> the question is why *don't* they increase in heap usage. Unless your
> 500 mb is the report of the actual live data set as evidenced by
> post-CMS heap usage.
>
>
What's considered to be "live data"? If we clear caches, run flush on the
key space, shouldn't that free up memory?
Thanks!
Huy
> --
> / Peter Schuller
>
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: Out of control memory consumption
Posted by Huy Le <hu...@springpartners.com>.
We do have graph of memory usage. Attached is a sample snapshot.
Huy
On Wed, Feb 9, 2011 at 12:57 PM, Peter Schuller <peter.schuller@infidyne.com
> wrote:
> (If you're looking at e.g. jconsole graphs a screenshot of the graph
> would not hurt.)
>
>
> --
> / Peter Schuller
>
--
Huy Le
Spring Partners, Inc.
http://springpadit.com
Re: Out of control memory consumption
Posted by Peter Schuller <pe...@infidyne.com>.
(If you're looking at e.g. jconsole graphs a screenshot of the graph
would not hurt.)
--
/ Peter Schuller
Re: Out of control memory consumption
Posted by Peter Schuller <pe...@infidyne.com>.
> We are 12-server cluster. We use random partitioner with manually generated
> server tokens. Memory usage on one server keeps growing out of control. We
> ran flush and cleared key and row caches but and ran GC but heap memory
> usage won't go down. The only way to heap memory usage to go down is the
> restart cassandra. We have to do this one a day. All other servers have
> heap memory usage less than 500MB. This issue happened on both Cassandra
> 0.6.6 and 0.6.11.
To be clear: You are not talking about the size of the Java process in
top, but the actual amount of heap used as reported by the JVM via
jmx/jconsole/etc?
Is the memory amount of memory that you consider high, the heap size
just after a concurrent mark/sweep?
Are you actually seeing OOM:s or are you restarting the node
pre-emptively in response to seeing heap usage go up?
> And JVM memory allocation: -Xms3G -Xmx3G
Just FYI: So it is entirely expected that the JVM will be 3G (a bit
higher) in size (even with standard I/O) and further that the amount
of live data in the heap be approaching 3G. The concurrent mark/sweep
GC won't trigger until the initial occupancy reaches the limit (if
modern Cassandra with default settings).
If you've got a 3 gig heap size and the other nodes stay at 500 mb,
the question is why *don't* they increase in heap usage. Unless your
500 mb is the report of the actual live data set as evidenced by
post-CMS heap usage.
--
/ Peter Schuller