You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ran Tavory <ra...@gmail.com> on 2010/10/11 16:13:30 UTC
Nodes getting slowed down after a few days of smooth operation
In my production cluster I've been seeing the following pattern.
When a node goes up it operates smoothly for a few days but then, after a
few days the node start to show excessive CPU usage, I see GC activity (and
it may also be excessive, not sure) and sometimes the node is dropped out of
the ring due to unresponsiveness. If I let things go untouched for a few
more days eventually all nodes in the cluster end up serving slow reads and
clients start timing out. A node restart solves the problem for a few more
days.
So far the only trigger for this behavior is "number of days that have
passed since last node restart", usually it's 4-5 days. One easy solution
is to restart nodes every couple of days but that's lame...
My current theory is this:
After a node gets restarted it compacts the sstable files on disk. I'm not
sure whether compactions always take place after restart, maybe it's just
minor compactions, I'm a little confused here, but my story would work best
if (major) compactions were always to run at server restart.
Then as the node operates, after a few days, enough sstable files start to
pile up and the node's filesystem cache gets full. This doesn't happen on
all the nodes, just in one DC in which I have hosts with 16G ram as opposed
to the other DC where I have 32G ram.
When the filesystem cache is full bad things start to happen. It doesn't
start right away, it may take 2-3 more days of high CPU and more or less
full fscache until shit starts to happen but eventually I start seeing more
GC activity, tpstats starts showing long list of pending tasks (usually the
row-read-stage), nodes get kicked out and back into the ring b/c they are
too busy to respond and clients start to time out or just have very bad
response times.
So I'm thinking what's the right thing to do?
If I increase memtable size would that help? (it'll take me a week to
experiment...). I'm not sure it'll actually help b/c at the end of the day
the filesystem cache will still get full, only that this time it'll be a
smaller number of larger files.
Actually I'm only evidently asserting that it's about the filesystem cache
getting full but I'm not sure this is the root cause, so what else could it
be? Maybe it's the sheer number of sstable files? At first I was thinking a
memory leak but I haven't seen any evidence of that.
If I run nodetool compact in a crontab every day instead of restart, would
that help? (that'll be lame too, I know...)
The documentation at http://wiki.apache.org/cassandra/MemtableSSTable states
that there's a number N of minimum files for compaction, default 4. Should I
try to decrease this number to maybe 2? (where is this setting anyway, I
don't think it's in storage-config, is this code only?)
Shooting in the dark, perhaps decreasing GCGraceSeconds from 10 days to 1
day will have a positive effect?
-------
Number of hosts: 6 in two DCs, 3 in each. RF=2, one copy of data in each DC.
Current version is 0.6.5 but it'd been like that with 0.6.2 and possibly
even before.
$ nodetool -port 9004 -host cass1 ring
Address Status Load Range
Ring
170141183460469231731687303715884105727
192.168.252.88Up 10.07 GB
28356863910078205288614550619314017621 |<--|
192.168.254.57Up 11.2 GB
56713727820156410577229101238628035242 | ^
192.168.252.124Up 10.03 GB
85070591730234615865843651857942052863 v |
192.168.254.58Up 11.26 GB
113427455640312821154458202477256070484 | ^
192.168.252.125Up 10.1 GB
141784319550391026443072753096570088105 v |
192.168.254.59Up 11.41 GB
170141183460469231731687303715884105727 |-->|
$ cat /proc/sys/vm/swappiness
0
$ java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
JVM_OPTS=" \
-ea \
-Xms4G \
-Xmx6G \
-XX:+UseParNewGC \
-XX:+UseConcMarkSweepGC \
-XX:+CMSParallelRemarkEnabled \
-XX:SurvivorRatio=8 \
-XX:MaxTenuringThreshold=1 \
-XX:CMSInitiatingOccupancyFraction=80 \
-XX:+HeapDumpOnOutOfMemoryError \
-Dcom.sun.management.jmxremote.port=9004 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false"
<HintedHandoffEnabled>false</HintedHandoffEnabled>
...
<ColumnFamily CompareWith="BytesType" Name="KvAds"
KeysCached="0"
RowsCached="10000000"/>
<ReplicaPlacementStrategy>org.apache.cassandra.locator.RackAwareStrategy</ReplicaPlacementStrategy>
<ReplicationFactor>2</ReplicationFactor>
<EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
</Keyspace>
...
<Partitioner>org.apache.cassandra.dht.RandomPartitioner</Partitioner>
...
<DiskAccessMode>standard</DiskAccessMode>
<RowWarningThresholdInMB>512</RowWarningThresholdInMB>
<SlicedBufferSizeInKB>64</SlicedBufferSizeInKB>
<FlushDataBufferSizeInMB>32</FlushDataBufferSizeInMB>
<FlushIndexBufferSizeInMB>8</FlushIndexBufferSizeInMB>
<ColumnIndexSizeInKB>64</ColumnIndexSizeInKB>
<MemtableThroughputInMB>64</MemtableThroughputInMB>
<BinaryMemtableThroughputInMB>256</BinaryMemtableThroughputInMB>
<MemtableOperationsInMillions>0.3</MemtableOperationsInMillions>
<MemtableFlushAfterMinutes>60</MemtableFlushAfterMinutes>
<ConcurrentReads>8</ConcurrentReads>
<ConcurrentWrites>32</ConcurrentWrites>
<CommitLogSync>periodic</CommitLogSync>
<CommitLogSyncPeriodInMS>10000</CommitLogSyncPeriodInMS>
Thanks!
--
/Ran
Re: Nodes getting slowed down after a few days of smooth operation
Posted by Ran Tavory <ra...@gmail.com>.
Peter, you're my JVM GC hero!
Thank you!
On Tue, Oct 12, 2010 at 12:38 AM, Peter Schuller <
peter.schuller@infidyne.com> wrote:
> > My motivation was that since I don't have too much data (10G each node)
> then
> > why don't I cache the hell out of it, so I started with a cache size of
> 100%
> > and a much larger heap size (started with 12G out of the 16G ram). Over
> time
> > I've learned that too much heap for the JVM is like a kid in a candy
> shop,
> > it'll eat as much as it can and then throw up (the kid was GC storming),
>
> In general CMS will tend to gobble up the maximum heap size unless
> your workload is such that the heuristics really work well and don't
> expand the heap beyond some level, but it won't magically fill the
> heap with data that doesn't exist. If you were reaching the maximum
> heap size with 12 GB, making the heap 6 GB instead won't make it
> better.
>
> Also, just be sure that you're really having an issue with GC. For
> example frequent young-generation GC:s are fully expected and normal.
> If you are seeing extremely frequent concurrent mark/sweep phases that
> do not free up a lot of data - that is an indication that the heap is
> too small.
>
> So, with respect to "GC storming", a bigger heap is generally better.
> The bigger the heap, the more effective GC is and the less often a
> concurrent mark/sweep has to happen.
>
> But this does not mean you want to give it too big a heap either,
> since whatever is gobbled up by the heap *won't* be used by the
> operating system for buffer caching.
>
> Keeping a big row cache may or may not be a good idea depending on
> circumstances, but if you have one, that directly implies additional
> heap usage and the heap must be sized accordingly. The row cache are
> just objects in memory; there is no automatic row cache size
> adjustment in response to heap pressure.
>
> If 10 million rows is your entire data set, and if that dataset is 10
> GB on disk (without in-memory object overhead), then I am not
> surprised at all that you're seeing issues after a few days of uptime.
> Likely the row cache is just much too big for the heap.
>
> > so
> > I started lowering the max heap until I reached 6G. with 4G I ran OOM
> BTW.
>
> Note that OOM and GC storming are often equivalent in terms of their
> cause (unless the OOM is caused by a single huge allocation or
> something). It's just that actually determining whether you are "out
> of memory" is difficult for the JVM, so there are heuristics involved.
> You may be sufficiently out of memory that you see excessive GC
> activity, but not so much as to trigger the threshold of GC
> inefficiency at which the JVM decides to actually through an OOM.
>
> > So now I have row cach capacity of effectively 100%, a heap size of 6G,
> data
> > of 10G and so I wonder how come the heap doesn't explode?
>
> Well, everything up to now has suggested to me that it *is* exploding ;)
>
> But:
>
> > Well, as it turns out, although I have 10G data on each node, the row
> cache
> > effective size is only about 681 * 2377203 = 1.6G (bytes)
> > Key cache: disabled
> > Row cache capacity: 10000000
> > Row cache size: 2377203
> > Row cache hit rate: 0.7017551635100059
> > Compacted row minimum size: 392
> > Compacted row maximum size: 102961
> > Compacted row mean size: 681
> > This strengthens what both Peter and Brandon have suggested that the row
> > cache is generating too much GC b/c it gets invalidated too frequently.
>
> Note that the compacted row size is not directly indicative of
> in-memory row size. I'm not sure what the overhead is expected to be
> though off hand; but you can probably assume a factor of 2 just from
> general fragmentation issue. Add to that overhead from the
> representation in object form itself etc. 1.6x2 = 3.2. Now we're
> starting to get close, especially taking into account additional
> overhead and other things on the heap.
>
> > That's certainly possible, so I'll try to set a 50% row cache size on one
> of
> > the nodes (and wait about a week...) and see what happens, and if this
> > proves to be the answer then this means that my dream of "I have so
> little
> > data and so much ram, why don't I cache the hell out of it" isn't going
> to
> > come true b/c too much of the row cache gets invalidated and hence GCed
> > which creates too much overhead for the JVM. (well, at least I was
> getting
> > nice read performance while it lasted ;)
>
> Given that you're not hitting your maximum cache size, data isn't
> evicted from the cache except as it is updated. Presumably that means
> you're actually not hitting the worst-case scenario, which is LRU
> eviction. Even then though, it's not as simple as it just being too
> much for the JVM. Especially given the rows/second that you'd expect
> to be evicted in Cassandra. A high rate of eviction does mean you need
> more margin in terms of free heap, but I seriously doubt the
> fundamental problem here is GC throughput vs. eviction rate.
>
> In general, I cannot stress enough - use jconsol/visual vm to observe
> heap usage, or at least check system logs for the results of GC and
> keep track of the heap usage after concurrent mark/sweep collections
> (not ParNew:s) to get a sense for what the actual amount of heap space
> "needed" is.
>
> > If this is true, then how would you recommend optimizing the row cache
> size
> > for maximum utility and minimum GC overhead?
>
> For one thing, I doubt a row cache as large as you have is very
> useful. If it takes several days to fill up to the point of you seeing
> memory problems, that suggests to me that it's far larger than
> actually needed. Presumably you will never want to have a system which
> is unable to function until after *days* of warm-up period.
>
> Simply decreasing it's size significantly would be my recommendation,
> if you keep it at all. Significantly being, I dunno; 100? A rule of
> thumb may be to watch how many rows are populated by X minutes or
> hours of operation (some reasonable warm-up period). Then just take
> that number and use as max (or less).
>
> Remember that caching will be done by the OS anyway, though that does
> not make the row cache useless (in particular the row cache survives
> compactions, meaning that compactions may have less of an impact when
> there is a row cache involved).
>
> On minimum GC overhead: Again in terms of GC overhead, the key point
> is to monitor heap usage. Unless you're doing cache eviction at 50k+
> rows/second or something along those lines, I don't think there should
> be any issue unless the cache is too small for the heap.
>
> > I've pasted here a log snippet from one of the servers while it was at
> high
> > CPU and GCing http://pastebin.com/U1cszFKv
>
> So you have it here:
>
> INFO [GC inspection] 2010-10-11 02:05:22,857 GCInspector.java (line
> 129) GC for ConcurrentMarkSweep: 27428 ms, 183140360 reclaimed leaving
> 6253188640 used; max is 6552551424
>
> At the end of a collection cycle, which took 27 seconds, it only freed
> below a couple of hundred megs, leaving an almost full heap. This
> simply means flat-out that you have too much data (Java objects) in
> the heap relative to the heap size. Heap size must be increased, or
> memory use decreased. In your case almost certainly the latter.
>
> > GC runs every like 20-40 seconds and almost for the entire duration of
> that
> > 20-40 secs. I'm not sure what to make of all the other numbers such
> as: GC
> > for ConcurrentMarkSweep: 22742 ms, 181335192 reclaimed leaving 6254994856
> > used; max is 6552551424
>
> See above; these are the critical lines that tell you probably the
> most out of any other line in the log file about the memories issues.
> Those tell you what the actual "live set" ("XXX used, max is YYY"
> means that XXX out of YYY bytes in the heap is live).
>
> (I am making *some* simplifications here because the concurrent nature
> of CMS means that you never get a snapshot view; but for practical
> purposes you can consider the above to be true with Cassandra.)
>
> Similar lines for "ParNew" you can mostly ignore for the purpose of
> monitoring heap usage unless you specifically know what you're looking
> for in those. The ConcurrentMarkSweep ones are what tell you what the
> actual amount of live data in the heap is.
>
> --
> / Peter Schuller
>
--
/Ran
Re: Nodes getting slowed down after a few days of smooth operation
Posted by Peter Schuller <pe...@infidyne.com>.
> My motivation was that since I don't have too much data (10G each node) then
> why don't I cache the hell out of it, so I started with a cache size of 100%
> and a much larger heap size (started with 12G out of the 16G ram). Over time
> I've learned that too much heap for the JVM is like a kid in a candy shop,
> it'll eat as much as it can and then throw up (the kid was GC storming),
In general CMS will tend to gobble up the maximum heap size unless
your workload is such that the heuristics really work well and don't
expand the heap beyond some level, but it won't magically fill the
heap with data that doesn't exist. If you were reaching the maximum
heap size with 12 GB, making the heap 6 GB instead won't make it
better.
Also, just be sure that you're really having an issue with GC. For
example frequent young-generation GC:s are fully expected and normal.
If you are seeing extremely frequent concurrent mark/sweep phases that
do not free up a lot of data - that is an indication that the heap is
too small.
So, with respect to "GC storming", a bigger heap is generally better.
The bigger the heap, the more effective GC is and the less often a
concurrent mark/sweep has to happen.
But this does not mean you want to give it too big a heap either,
since whatever is gobbled up by the heap *won't* be used by the
operating system for buffer caching.
Keeping a big row cache may or may not be a good idea depending on
circumstances, but if you have one, that directly implies additional
heap usage and the heap must be sized accordingly. The row cache are
just objects in memory; there is no automatic row cache size
adjustment in response to heap pressure.
If 10 million rows is your entire data set, and if that dataset is 10
GB on disk (without in-memory object overhead), then I am not
surprised at all that you're seeing issues after a few days of uptime.
Likely the row cache is just much too big for the heap.
> so
> I started lowering the max heap until I reached 6G. with 4G I ran OOM BTW.
Note that OOM and GC storming are often equivalent in terms of their
cause (unless the OOM is caused by a single huge allocation or
something). It's just that actually determining whether you are "out
of memory" is difficult for the JVM, so there are heuristics involved.
You may be sufficiently out of memory that you see excessive GC
activity, but not so much as to trigger the threshold of GC
inefficiency at which the JVM decides to actually through an OOM.
> So now I have row cach capacity of effectively 100%, a heap size of 6G, data
> of 10G and so I wonder how come the heap doesn't explode?
Well, everything up to now has suggested to me that it *is* exploding ;)
But:
> Well, as it turns out, although I have 10G data on each node, the row cache
> effective size is only about 681 * 2377203 = 1.6G (bytes)
> Key cache: disabled
> Row cache capacity: 10000000
> Row cache size: 2377203
> Row cache hit rate: 0.7017551635100059
> Compacted row minimum size: 392
> Compacted row maximum size: 102961
> Compacted row mean size: 681
> This strengthens what both Peter and Brandon have suggested that the row
> cache is generating too much GC b/c it gets invalidated too frequently.
Note that the compacted row size is not directly indicative of
in-memory row size. I'm not sure what the overhead is expected to be
though off hand; but you can probably assume a factor of 2 just from
general fragmentation issue. Add to that overhead from the
representation in object form itself etc. 1.6x2 = 3.2. Now we're
starting to get close, especially taking into account additional
overhead and other things on the heap.
> That's certainly possible, so I'll try to set a 50% row cache size on one of
> the nodes (and wait about a week...) and see what happens, and if this
> proves to be the answer then this means that my dream of "I have so little
> data and so much ram, why don't I cache the hell out of it" isn't going to
> come true b/c too much of the row cache gets invalidated and hence GCed
> which creates too much overhead for the JVM. (well, at least I was getting
> nice read performance while it lasted ;)
Given that you're not hitting your maximum cache size, data isn't
evicted from the cache except as it is updated. Presumably that means
you're actually not hitting the worst-case scenario, which is LRU
eviction. Even then though, it's not as simple as it just being too
much for the JVM. Especially given the rows/second that you'd expect
to be evicted in Cassandra. A high rate of eviction does mean you need
more margin in terms of free heap, but I seriously doubt the
fundamental problem here is GC throughput vs. eviction rate.
In general, I cannot stress enough - use jconsol/visual vm to observe
heap usage, or at least check system logs for the results of GC and
keep track of the heap usage after concurrent mark/sweep collections
(not ParNew:s) to get a sense for what the actual amount of heap space
"needed" is.
> If this is true, then how would you recommend optimizing the row cache size
> for maximum utility and minimum GC overhead?
For one thing, I doubt a row cache as large as you have is very
useful. If it takes several days to fill up to the point of you seeing
memory problems, that suggests to me that it's far larger than
actually needed. Presumably you will never want to have a system which
is unable to function until after *days* of warm-up period.
Simply decreasing it's size significantly would be my recommendation,
if you keep it at all. Significantly being, I dunno; 100? A rule of
thumb may be to watch how many rows are populated by X minutes or
hours of operation (some reasonable warm-up period). Then just take
that number and use as max (or less).
Remember that caching will be done by the OS anyway, though that does
not make the row cache useless (in particular the row cache survives
compactions, meaning that compactions may have less of an impact when
there is a row cache involved).
On minimum GC overhead: Again in terms of GC overhead, the key point
is to monitor heap usage. Unless you're doing cache eviction at 50k+
rows/second or something along those lines, I don't think there should
be any issue unless the cache is too small for the heap.
> I've pasted here a log snippet from one of the servers while it was at high
> CPU and GCing http://pastebin.com/U1cszFKv
So you have it here:
INFO [GC inspection] 2010-10-11 02:05:22,857 GCInspector.java (line
129) GC for ConcurrentMarkSweep: 27428 ms, 183140360 reclaimed leaving
6253188640 used; max is 6552551424
At the end of a collection cycle, which took 27 seconds, it only freed
below a couple of hundred megs, leaving an almost full heap. This
simply means flat-out that you have too much data (Java objects) in
the heap relative to the heap size. Heap size must be increased, or
memory use decreased. In your case almost certainly the latter.
> GC runs every like 20-40 seconds and almost for the entire duration of that
> 20-40 secs. I'm not sure what to make of all the other numbers such as: GC
> for ConcurrentMarkSweep: 22742 ms, 181335192 reclaimed leaving 6254994856
> used; max is 6552551424
See above; these are the critical lines that tell you probably the
most out of any other line in the log file about the memories issues.
Those tell you what the actual "live set" ("XXX used, max is YYY"
means that XXX out of YYY bytes in the heap is live).
(I am making *some* simplifications here because the concurrent nature
of CMS means that you never get a snapshot view; but for practical
purposes you can consider the above to be true with Cassandra.)
Similar lines for "ParNew" you can mostly ignore for the purpose of
monitoring heap usage unless you specifically know what you're looking
for in those. The ConcurrentMarkSweep ones are what tell you what the
actual amount of live data in the heap is.
--
/ Peter Schuller
Re: Nodes getting slowed down after a few days of smooth operation
Posted by Ran Tavory <ra...@gmail.com>.
Thanks Peter, Robert and Brandon.
So it seems that the only suspect by now is my excessive caching ;)
I'll get a better look at the GC activity next time shit starts to happen,
but in the mean time, as for the cache size (cassandra's internal cache),
it's row cache capacity is set to 10,000,000. I actually wanted to say 100%
but at the time there was a bug that interpreted 100% as just 1 so I used
10M instead.
My motivation was that since I don't have too much data (10G each node) then
why don't I cache the hell out of it, so I started with a cache size of 100%
and a much larger heap size (started with 12G out of the 16G ram). Over time
I've learned that too much heap for the JVM is like a kid in a candy shop,
it'll eat as much as it can and then throw up (the kid was GC storming), so
I started lowering the max heap until I reached 6G. with 4G I ran OOM BTW.
So now I have row cach capacity of effectively 100%, a heap size of 6G, data
of 10G and so I wonder how come the heap doesn't explode?
Well, as it turns out, although I have 10G data on each node, the row cache
effective size is only about 681 * 2377203 = 1.6G (bytes)
Key cache: disabled
Row cache capacity: 10000000
Row cache size: *2377203*
Row cache hit rate: 0.7017551635100059
Compacted row minimum size: 392
Compacted row maximum size: 102961
Compacted row mean size: *681*
This strengthens what both Peter and Brandon have suggested that the row
cache is generating too much GC b/c it gets invalidated too frequently.
That's certainly possible, so I'll try to set a 50% row cache size on one of
the nodes (and wait about a week...) and see what happens, and if this
proves to be the answer then this means that my dream of "I have so little
data and so much ram, why don't I cache the hell out of it" isn't going to
come true b/c too much of the row cache gets invalidated and hence GCed
which creates too much overhead for the JVM. (well, at least I was getting
nice read performance while it lasted ;)
If this is true, then how would you recommend optimizing the row cache size
for maximum utility and minimum GC overhead?
I've pasted here a log snippet from one of the servers while it was at high
CPU and GCing http://pastebin.com/U1cszFKv
You can see a large number of pending reads as well as other pending tasks
(response stage or consistency manager).
GC runs every like 20-40 seconds and almost for the entire duration of that
20-40 secs. I'm not sure what to make of all the other numbers such as: GC
for ConcurrentMarkSweep: 22742 ms, 181335192 reclaimed leaving 6254994856
used; max is 6552551424
Thanks!
On Mon, Oct 11, 2010 at 7:42 PM, Peter Schuller <peter.schuller@infidyne.com
> wrote:
> > 170141183460469231731687303715884105727
> > 192.168.252.88Up 10.07 GB
>
> Firstly, I second the point raised about the row cache size (very
> frequent concurrent GC:s is definitely an indicator that the JVM heap
> size is too small, and the row cache seems like a likely contender -
> especially given that you say it builds up over days). Note though
> that you have to look at the GCInspector's output with respect to the
> concurrent mark/sweep GC phases to judge the live set in your heap,
> rather than system memory. Attaching with jconsole or visualvm to the
> JVM will also give you a pretty good view of what's going on. Look for
> the heap usage as it appears after one of the "major" dips in the
> graph (not the regular sawtooth dips, which are young generation
> collections and won't help indicate actual live set).
>
> That said, with respect to caching effects: Your total data size seems
> to be about in the same ballpark as memory. Your maximum heap size is
> 6 gig; on a 16 gig machine, taking into account varous overhead, maybe
> you've got something like 8 GB for buffer cache? It doesn't sound
> strange at all that there would be a significant difference between a
> 32 GB machine and a 16 GB machine given your ~ 10 GB data size given
> that buffer cache size goes from "slightly below data size" to "almost
> three times data size". Especially when major or almost-major
> compactions are triggered; on the small machine you would expect to
> evict everything from cache during a compaction (except that touched
> *during* the compaction) while on the larger machine the newly written
> sstables effectively fit the cache too.
>
> But note that these are two pretty different conditions; the first is
> about making sure your JVM heap size is appropriate. The second can be
> tested for by observing I/O load (iostat -x -k 1) and correlating with
> compactions. So e.g., what's the average utilization and queue size in
> iostat just before a compaction vs. just after it? That difference
> should be due to cache eviction (assuming you're not servicing a
> built-up backlog). There is also the impact of compaction itself, as
> it is happening, and the I/O it generates. In general, the higher your
> disk load is prior to compaction, the less margin there is to deal
> with compaction happening concurrently.
>
> In general, whether or not you are willing to make the assumption that
> activley used data fits in RAM will severely affect the hardware
> requirements for serving your load.
>
> --
> / Peter Schuller
>
--
/Ran
Re: Nodes getting slowed down after a few days of smooth operation
Posted by Peter Schuller <pe...@infidyne.com>.
> 170141183460469231731687303715884105727
> 192.168.252.88Up 10.07 GB
Firstly, I second the point raised about the row cache size (very
frequent concurrent GC:s is definitely an indicator that the JVM heap
size is too small, and the row cache seems like a likely contender -
especially given that you say it builds up over days). Note though
that you have to look at the GCInspector's output with respect to the
concurrent mark/sweep GC phases to judge the live set in your heap,
rather than system memory. Attaching with jconsole or visualvm to the
JVM will also give you a pretty good view of what's going on. Look for
the heap usage as it appears after one of the "major" dips in the
graph (not the regular sawtooth dips, which are young generation
collections and won't help indicate actual live set).
That said, with respect to caching effects: Your total data size seems
to be about in the same ballpark as memory. Your maximum heap size is
6 gig; on a 16 gig machine, taking into account varous overhead, maybe
you've got something like 8 GB for buffer cache? It doesn't sound
strange at all that there would be a significant difference between a
32 GB machine and a 16 GB machine given your ~ 10 GB data size given
that buffer cache size goes from "slightly below data size" to "almost
three times data size". Especially when major or almost-major
compactions are triggered; on the small machine you would expect to
evict everything from cache during a compaction (except that touched
*during* the compaction) while on the larger machine the newly written
sstables effectively fit the cache too.
But note that these are two pretty different conditions; the first is
about making sure your JVM heap size is appropriate. The second can be
tested for by observing I/O load (iostat -x -k 1) and correlating with
compactions. So e.g., what's the average utilization and queue size in
iostat just before a compaction vs. just after it? That difference
should be due to cache eviction (assuming you're not servicing a
built-up backlog). There is also the impact of compaction itself, as
it is happening, and the I/O it generates. In general, the higher your
disk load is prior to compaction, the less margin there is to deal
with compaction happening concurrently.
In general, whether or not you are willing to make the assumption that
activley used data fits in RAM will severely affect the hardware
requirements for serving your load.
--
/ Peter Schuller
Re: Nodes getting slowed down after a few days of smooth operation
Posted by Brandon Williams <dr...@gmail.com>.
On Mon, Oct 11, 2010 at 9:13 AM, Ran Tavory <ra...@gmail.com> wrote:
> In my production cluster I've been seeing the following pattern.
> When a node goes up it operates smoothly for a few days but then, after a
> few days the node start to show excessive CPU usage, I see GC activity (and
> it may also be excessive, not sure) and sometimes the node is dropped out of
> the ring due to unresponsiveness. If I let things go untouched for a few
> more days eventually all nodes in the cluster end up serving slow reads and
> clients start timing out. A node restart solves the problem for a few more
> days.
> So far the only trigger for this behavior is "number of days that have
> passed since last node restart", usually it's 4-5 days. One easy solution
> is to restart nodes every couple of days but that's lame...
>
*snip*
> <ColumnFamily CompareWith="BytesType" Name="KvAds"
> KeysCached="0"
> RowsCached="10000000"/>
>
That's a large row cache, it might be exerting heavy GC pressure on the JVM.
Try setting the key cache to 100% and disabling the row cache instead.
-Brandon
Re: Nodes getting slowed down after a few days of smooth operation
Posted by Peter Schuller <pe...@infidyne.com>.
> I have wondered before whether there is any technical reason why the commit
> log replay should end with a flush, and from what I can tell, there isn't
> one other than the general goal of not having a large commit log. My
> personal feeling is that the last thing you want your production node doing
> after you've restarted it (blowing out the internal key and row caches) is
> compacting and blowing out your only remaining cache (the OS page cache) as
> a result. If this is a problem for others in production, I could submit a
> JIRA with a patch to disable the memtable flush part of startup?
While I don't argue against such a change, one argument for *having*
the flush is that subsequent restarts become faster and it might help
if you're in some kind of situation where nodes continually die
shortly after having been started.
(But I totally agree about blowing caches.)
--
/ Peter Schuller
Re: Nodes getting slowed down after a few days of smooth operation
Posted by Robert Coli <rc...@digg.com>.
On 10/11/10 7:13 AM, Ran Tavory wrote:
> After a node gets restarted it compacts the sstable files on disk. I'm
> not sure whether compactions always take place after restart, maybe
> it's just minor compactions, I'm a little confused here, but my story
> would work best if (major) compactions were always to run at server
> restart.
Node startup contains (among other steps) :
- commit log replay
- memtable flush of replayed transactions
The memtable flush can and does trigger minor compactions, and this is
almost certainly what you're seeing.
I have wondered before whether there is any technical reason why the
commit log replay should end with a flush, and from what I can tell,
there isn't one other than the general goal of not having a large commit
log. My personal feeling is that the last thing you want your production
node doing after you've restarted it (blowing out the internal key and
row caches) is compacting and blowing out your only remaining cache (the
OS page cache) as a result. If this is a problem for others in
production, I could submit a JIRA with a patch to disable the memtable
flush part of startup?
=Rob