You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mikio Braun <mi...@cs.tu-berlin.de> on 2010/08/24 17:29:44 UTC
Follow-up post on cassandra configuration with some experiments on
GC tuning
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Dear all,
thanks again for all the comments I got on my last post. I've played a
bit with different GC settings and got my Cassandra instance to run
very nicely with 8GB of heap.
I summarized my experiences with GC tuning in this follow-up post:
http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html
- -M
- --
Dr. Mikio Braun email: mikio@cs.tu-berlin.de
TU Berlin web: ml.cs.tu-berlin.de/~mikio
Franklinstr. 28/29 tel: +49 30 314 78627
10587 Berlin, Germany
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkxz5WcACgkQtnXKX8rQtgDiiwCeLknuTcr65eehwIcsivInjv4W
LaQAn3RY9pH19r8SuUhVBvtE6LeyFUvB
=MYsY
-----END PGP SIGNATURE-----
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Mikio Braun <mi...@cs.tu-berlin.de>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Carsten,
> In this regard, what I personally miss in Mikios - however nice - analysis, is what are the effects on the application stop times due to any garbage collection runs for the cases tested. In most cases, I prefer having low pauses due to any garbage collection runs and don't care too much about the shape of the memory usage, and I guess, that's the reason why the low pause collector is used by default for running cassandra.
I see your point. I haven't explicitly tested those pauses. You can get
them from the gc logs (with some amount of perl parsing).
Subjectively speaking, I saw a higher probability of timeouts if GC took
too long. One other thing to look out for would be CMS failures (when
you start a CMS cycle but the young generation GCs run out of memory to
promote objects) which then results in a full GC cycle.
Probably I can rerun the tests and save the gc logs as well and put them
somewhere.
- -M
- --
Dr. Mikio Braun email: mikio@cs.tu-berlin.de
TU Berlin web: ml.cs.tu-berlin.de/~mikio
Franklinstr. 28/29 tel: +49 30 314 78627
10587 Berlin, Germany
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkx/0wkACgkQtnXKX8rQtgC7CwCfSHyh4+6mMxKIbcmNCUegeY8P
0cwAnAhQrFKomDJ96P1ZQ3cZowDmrim1
=Lwuj
-----END PGP SIGNATURE-----
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Jonathan Ellis <jb...@gmail.com>.
On Mon, Aug 30, 2010 at 5:18 PM, Peter Schuller
<pe...@infidyne.com> wrote:
> Has anyone run Cassandra with G1 in production for prolonged periods
> of time?
Not AFAIK.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Peter Schuller <pe...@infidyne.com>.
> collection runs for the cases tested. In most cases, I prefer having low
> pauses due to any garbage collection runs and don't care too much about the
> shape of the memory usage, and I guess, that's the reason why the low pause
> collector is used by default for running cassandra. For myself, I have mixed
> feelings regarding the low pause collector, because I found it difficult to
> find good young generation sizings, which are suitable to different load
> patterns. Therefor I mostly prefer the throughput collector, which
> adaptively sizes the young generation, doing a good job to avoiding that too
> much data goes to the tenured generation.
Well, if you care about pause times, usually the best bet would be to
have the young gen be as large as possible to yield what you consider
to be a pause time within an acceptable range. I.e., as large as
acceptable but no larger.
> I would be interested in, what are
> the differences concerning the stop times between the different GC variants,
> when running cassandra. Is it really much better to use the low pause
> collector in regard to get stabile response times, even if I use
> XX:+UseParallelOldGC and XX:MaxGCPauseMillis=nnn flags? Any experiences with
> this?
If you use the default (for the JVM, not for cassandra) throughput
collector, you *will* take full stop-the-world collections, period.
You can enable parallel GC, but with that collector there's no way
around the fact that full collections will pause the application for
the full duration of such full GC:s. In general, the larger the heap
(relative to speed of the collection), the more of a problem this will
be. If you deem the pause times acceptable for your particular
use-case, I don't see an obvious reason to prefer the CMS collector.
MaxGCPauseMillis won't help; the throughput collector just doesn't
have any way to ader to it. A full GC is a full GC.
For CMS, I'm not sure what, if any, effect the MaxGCPauseMillis has.
In my very limited testing I didn't see any obvious effect on e.g.
sizing choice for the young generation (but I have not checked the
code to see if CMS uses it).
It is definitely used by the G1 collector; typically MaxGCPauseMillis
and GCPauseIntervalMillis are the two most importants settings to
tweak. They are directly used to decide the young generation size, as
well as limit the number of non-young regions that are picked for GC
during a partial (not young-only) GC.
Has anyone run Cassandra with G1 in production for prolonged periods
of time? One thing that concerns me is the reliance on GC to remove
obsolete SS tables. That relies on certain GC behavior that is true
for CMS and the throughput collector, but not with G1. With CMS, an
unreachable sstable will be detected when concurrent mark/sweep
triggers; but with G1, there is not necessarily any expectation at all
that some particular region that happens to contain the reference in
question will be collected - *ever* - since G1 always picks the "best"
regions first (best in terms of "bang for the buck" - the most memory
reclaimed at the lowest cost).
--
/ Peter Schuller
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Edward Capriolo <ed...@gmail.com>.
On Sun, Aug 29, 2010 at 1:53 PM, Carsten Krebs <ca...@gmx.net> wrote:
>
>
> Also, note that lack of saw-toothing is not a goal in and of itself
> and may even be bad. For example, with respect to the young generation
> the situation is essentially:
>
> (1) The larger the young generation, the more significant the saw-tooth.
> (2) The larger the young generation, the more efficient the GC (if the
> application behaves according to the weak generational hypothesis -
> google it if you want a ref) because less data is promoted to old gen
> and because the overhead of stop-the-world is lessened.
> (3) The larger the young generation, the longer the pause times to do
> collections of the young generation.
>
> In this regard, what I personally miss in Mikios - however nice - analysis,
> is what are the effects on the application stop times due to any garbage
> collection runs for the cases tested. In most cases, I prefer having low
> pauses due to any garbage collection runs and don't care too much about the
> shape of the memory usage, and I guess, that's the reason why the low pause
> collector is used by default for running cassandra. For myself, I have mixed
> feelings regarding the low pause collector, because I found it difficult to
> find good young generation sizings, which are suitable to different load
> patterns. Therefor I mostly prefer the throughput collector, which
> adaptively sizes the young generation, doing a good job to avoiding that too
> much data goes to the tenured generation. I would be interested in, what are
> the differences concerning the stop times between the different GC variants,
> when running cassandra. Is it really much better to use the low pause
> collector in regard to get stabile response times, even if I use
> XX:+UseParallelOldGC and XX:MaxGCPauseMillis=nnn flags? Any experiences with
> this?
> Regards,
> Carsten
>
>
I know that the saw-toothing is not necessarily a bad thing. If you
notice my graph the memory goes from 9-12 GB every 30 minutes.
Assuming throughput/performance is roughly the same , doesn't adaptive
power management kick in, thus keeping the earth greener and leaving
more power for other servers in your rack? (maybe that does not apply
in this case since java memory appears used to the OS)
I like these settings just because they seem to be more proactive.
This seems to keep my memory at a solid number rather then moving from
8-12GB on thirty minute intervals. If some event like a large row or
load hits the node I feel it helps to have less memory used.
In any case, very interesting thread. Keep it going!
Re: Follow-up post on cassandra configuration with some experiments on GC tuning
Posted by Carsten Krebs <ca...@gmx.net>.
>
> Also, note that lack of saw-toothing is not a goal in and of itself
> and may even be bad. For example, with respect to the young generation
> the situation is essentially:
>
> (1) The larger the young generation, the more significant the saw-tooth.
> (2) The larger the young generation, the more efficient the GC (if the
> application behaves according to the weak generational hypothesis -
> google it if you want a ref) because less data is promoted to old gen
> and because the overhead of stop-the-world is lessened.
> (3) The larger the young generation, the longer the pause times to do
> collections of the young generation.
>
In this regard, what I personally miss in Mikios - however nice - analysis, is what are the effects on the application stop times due to any garbage collection runs for the cases tested. In most cases, I prefer having low pauses due to any garbage collection runs and don't care too much about the shape of the memory usage, and I guess, that's the reason why the low pause collector is used by default for running cassandra. For myself, I have mixed feelings regarding the low pause collector, because I found it difficult to find good young generation sizings, which are suitable to different load patterns. Therefor I mostly prefer the throughput collector, which adaptively sizes the young generation, doing a good job to avoiding that too much data goes to the tenured generation. I would be interested in, what are the differences concerning the stop times between the different GC variants, when running cassandra. Is it really much better to use the low pause collector in regard to get stabile response times, even if I use XX:+UseParallelOldGC and XX:MaxGCPauseMillis=nnn flags? Any experiences with this?
Regards,
Carsten
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Peter Schuller <pe...@infidyne.com>.
> Before I applied these changes the young gen and the survivor space
> were very spiky. Now they both seem very low all the time. As you see
> from my screen shot, before these changes my JVM memory would make
> large saw tooths, now all three pools young, eden, perm seem smoother.
I'm not sure what's going on on Mikio's original graph (why CMS-i
would somehow cause lower average memory usage; I think something else
is going on there).
WIth respect to your graph:
http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/tune_that_jvm
That just looksl ike a concurrent/mark sweep has not completed at all
after the dip after around 14:00. Sooner or later it should either
complete a concurrent mark/sweep resulting on a dip, or it should fail
to complete it time causing a fallback to a full GC. Unless the CMS-i
does something *completely* different that I have completely missed,
you definitely should be expecting a sudden dip once a mark/sweep
finishes.
The minor saw-toothy behavior seen on the slope is mostly going to be
dictated by the young generation size chosen by the collector.
Possibly it chooses a smaller young generation when cms-i is enabled
(speculation).
Also, note that lack of saw-toothing is not a goal in and of itself
and may even be bad. For example, with respect to the young generation
the situation is essentially:
(1) The larger the young generation, the more significant the saw-tooth.
(2) The larger the young generation, the more efficient the GC (if the
application behaves according to the weak generational hypothesis -
google it if you want a ref) because less data is promoted to old gen
and because the overhead of stop-the-world is lessened.
(3) The larger the young generation, the longer the pause times to do
collections of the young generation.
A few consequences of those include:
* Assuming a parallel collection of the young generation, more CPU:s
mean that the optimal size of the young generation given a certain
pause time goal is higher. In other words, more CPU:s -> more saw
tooth.
* Lack of saw tooth may just indicate that the majority or of data
survives the young generation collections. This is not a good thing;
at best it's neutral because the application is simply such that it
does not generate a lot of temporary garbage (i.e., it does NOT adher
to the weak generational hypothesis). At worst it means GC will be
more expensive overall because the "per object" cost of collection the
old generation is significantly higher than the "per object" cost of
collecting the young generation. That said, if most data is truly very
transient in nature, a smaller young generation may still be "big
enough".
* The previous point highlights the trade-off between low pause times
and GC efficiency. One might force a smaller young generation in an
attempt to achieve shorter pause times with CMS, but the trade-off is
that a larger percentage of allocated data will survive into the old
generation and be collected there - more expensively.
> I am worried that the cms descriptions talk about systems with 1-2
> processor machines, being my system shows up at 16 processors after
> hyper threading.
My assumption has been that this recommendation is due to the fact
that the more processors you have, the less impact the CMS mark/sweep
phase may have on application throughput provided that an appropriate
number of threads is selected. So for example, if you have an 8 core
machine and have CMS use only a single thread for the mark/sweep
phase, the very fact that it is only using 1 out of 8 cores should
severely limit its impact. (Of course cache coherency issues
presumably negate this somewhat.)
Under such circumstances, incremental CMS does not seem worth it. On
the other, suppose you're running on a single CPU system. Disregarding
CPU cache issues, the concurrent mark/sweep phase would now
effectively halve the CPU resources available to the application. A
50% decrease is significant, and under such circumstances the
incremental mode is potentially interesting.
A trade-off is, presumably (again I don't know a lot about how
incremental mode is implemented, but I doubt they've avoided this), is
that the total time needed for the mark/sweep onces it does run is
higher, such that you retain more floating garbage that might
otherwise have been collected.
--
/ Peter Schuller
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Edward Capriolo <ed...@gmail.com>.
On Fri, Aug 27, 2010 at 6:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> I supsect something else is making the difference for ecapriolo. The
> documentation says,
>
> The incremental mode is meant to lessen the impact of long concurrent
> phases by periodically stopping the concurrent phase to yield back the
> processor to the application. [Remember, "concurrent" means "not
> blocking other threads" in this context.] This mode (referred to here
> as “i-cms”) divides the work done by concurrently by the collector
> into small chunks of time which are scheduled between young generation
> collections. This feature is useful when applications that need the
> low pause times provided by the concurrent collector are run on
> machines with small numbers of processors (e.g., 1 or 2)
>
> There's also close to a dozen tuning options for i-cms. It seems a
> little fragile to me.
>
> On Fri, Aug 27, 2010 at 4:53 PM, Benjamin Black <b...@b3k.us> wrote:
>> ecapriolo's testing seemed to indicate it _did_ change the behavior.
>> wonder what the difference is?
>>
>> On Fri, Aug 27, 2010 at 6:23 AM, Mikio Braun <mi...@cs.tu-berlin.de> wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Dear all,
>>>
>>> thanks for your comments, and I'm glad that you found my post helpful.
>>>
>>> Concerning the incremental CMS, I've recently updated my post and added
>>> the experiments repeated on one of our cluster nodes, and for some
>>> reason incremental CMS doesn't look that different anymore. So I guess
>>> it's ok to stick with the non-incremental CMS for now.
>>>
>>> - -M
>>>
>>> On 08/27/2010 09:12 AM, Peter Schuller wrote:
>>>>> Whether or not this is likely to happen with Cassandra I don't know. I
>>>>> don't know much about the incremental duty cycles are scheduled and it
>>>>> may be the case that Cassandra is not even remotely close to having a
>>>>> problem with incremental mode.
>>>>
>>>> I should further weaken my statement by pointing out that I never did
>>>> any exhaustive tweaking to get around the problem (other than
>>>> disabling incremental mode, since my primary goal has tended to be
>>>> ensure low pause times and not so much even GC activity). It may be
>>>> the case that even in stressful cases where it fails by default it is
>>>> simply a matter of tweaking.
>>>>
>>>> So, I guess I should re-phrase: In terms of just turning on
>>>> incremental mode without at least application specific tweaking (if
>>>> not deployment specific testing), I would suggest caution.
>>>>
>>>>
>>>
>>>
>>> - --
>>> Dr. Mikio Braun email: mikio@cs.tu-berlin.de
>>> TU Berlin web: ml.cs.tu-berlin.de/~mikio
>>> Franklinstr. 28/29 tel: +49 30 314 78627
>>> 10587 Berlin, Germany
>>>
>>>
>>>
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1.4.9 (GNU/Linux)
>>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>>
>>> iEYEARECAAYFAkx3vFUACgkQtnXKX8rQtgDUlgCfWb/euA2mgVJAWDY2tBSyAN+I
>>> 604AoKVua1+5bYK2yF9CWwFQmLHDt0Fn
>>> =CIal
>>> -----END PGP SIGNATURE-----
>>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Before I applied these changes the young gen and the survivor space
were very spiky. Now they both seem very low all the time. As you see
from my screen shot, before these changes my JVM memory would make
large saw tooths, now all three pools young, eden, perm seem smoother.
I am worried that the cms descriptions talk about systems with 1-2
processor machines, being my system shows up at 16 processors after
hyper threading.
Is something else going on. :) I am not in a lab so I have many
external factors. Being that it takes a long time to build up a row
cache its hard to compare for hours.
(On a side node I have started working on 'nodetool savecache' so
hopefully I figure out a way to save the cache to an external file and
recall it after start up. Seems like there could be edge cases where
bring up a cache even from moments ago could be be, but will see)
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Jonathan Ellis <jb...@gmail.com>.
I supsect something else is making the difference for ecapriolo. The
documentation says,
The incremental mode is meant to lessen the impact of long concurrent
phases by periodically stopping the concurrent phase to yield back the
processor to the application. [Remember, "concurrent" means "not
blocking other threads" in this context.] This mode (referred to here
as “i-cms”) divides the work done by concurrently by the collector
into small chunks of time which are scheduled between young generation
collections. This feature is useful when applications that need the
low pause times provided by the concurrent collector are run on
machines with small numbers of processors (e.g., 1 or 2)
There's also close to a dozen tuning options for i-cms. It seems a
little fragile to me.
On Fri, Aug 27, 2010 at 4:53 PM, Benjamin Black <b...@b3k.us> wrote:
> ecapriolo's testing seemed to indicate it _did_ change the behavior.
> wonder what the difference is?
>
> On Fri, Aug 27, 2010 at 6:23 AM, Mikio Braun <mi...@cs.tu-berlin.de> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Dear all,
>>
>> thanks for your comments, and I'm glad that you found my post helpful.
>>
>> Concerning the incremental CMS, I've recently updated my post and added
>> the experiments repeated on one of our cluster nodes, and for some
>> reason incremental CMS doesn't look that different anymore. So I guess
>> it's ok to stick with the non-incremental CMS for now.
>>
>> - -M
>>
>> On 08/27/2010 09:12 AM, Peter Schuller wrote:
>>>> Whether or not this is likely to happen with Cassandra I don't know. I
>>>> don't know much about the incremental duty cycles are scheduled and it
>>>> may be the case that Cassandra is not even remotely close to having a
>>>> problem with incremental mode.
>>>
>>> I should further weaken my statement by pointing out that I never did
>>> any exhaustive tweaking to get around the problem (other than
>>> disabling incremental mode, since my primary goal has tended to be
>>> ensure low pause times and not so much even GC activity). It may be
>>> the case that even in stressful cases where it fails by default it is
>>> simply a matter of tweaking.
>>>
>>> So, I guess I should re-phrase: In terms of just turning on
>>> incremental mode without at least application specific tweaking (if
>>> not deployment specific testing), I would suggest caution.
>>>
>>>
>>
>>
>> - --
>> Dr. Mikio Braun email: mikio@cs.tu-berlin.de
>> TU Berlin web: ml.cs.tu-berlin.de/~mikio
>> Franklinstr. 28/29 tel: +49 30 314 78627
>> 10587 Berlin, Germany
>>
>>
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.9 (GNU/Linux)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iEYEARECAAYFAkx3vFUACgkQtnXKX8rQtgDUlgCfWb/euA2mgVJAWDY2tBSyAN+I
>> 604AoKVua1+5bYK2yF9CWwFQmLHDt0Fn
>> =CIal
>> -----END PGP SIGNATURE-----
>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Benjamin Black <b...@b3k.us>.
ecapriolo's testing seemed to indicate it _did_ change the behavior.
wonder what the difference is?
On Fri, Aug 27, 2010 at 6:23 AM, Mikio Braun <mi...@cs.tu-berlin.de> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dear all,
>
> thanks for your comments, and I'm glad that you found my post helpful.
>
> Concerning the incremental CMS, I've recently updated my post and added
> the experiments repeated on one of our cluster nodes, and for some
> reason incremental CMS doesn't look that different anymore. So I guess
> it's ok to stick with the non-incremental CMS for now.
>
> - -M
>
> On 08/27/2010 09:12 AM, Peter Schuller wrote:
>>> Whether or not this is likely to happen with Cassandra I don't know. I
>>> don't know much about the incremental duty cycles are scheduled and it
>>> may be the case that Cassandra is not even remotely close to having a
>>> problem with incremental mode.
>>
>> I should further weaken my statement by pointing out that I never did
>> any exhaustive tweaking to get around the problem (other than
>> disabling incremental mode, since my primary goal has tended to be
>> ensure low pause times and not so much even GC activity). It may be
>> the case that even in stressful cases where it fails by default it is
>> simply a matter of tweaking.
>>
>> So, I guess I should re-phrase: In terms of just turning on
>> incremental mode without at least application specific tweaking (if
>> not deployment specific testing), I would suggest caution.
>>
>>
>
>
> - --
> Dr. Mikio Braun email: mikio@cs.tu-berlin.de
> TU Berlin web: ml.cs.tu-berlin.de/~mikio
> Franklinstr. 28/29 tel: +49 30 314 78627
> 10587 Berlin, Germany
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkx3vFUACgkQtnXKX8rQtgDUlgCfWb/euA2mgVJAWDY2tBSyAN+I
> 604AoKVua1+5bYK2yF9CWwFQmLHDt0Fn
> =CIal
> -----END PGP SIGNATURE-----
>
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Mikio Braun <mi...@cs.tu-berlin.de>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Dear all,
thanks for your comments, and I'm glad that you found my post helpful.
Concerning the incremental CMS, I've recently updated my post and added
the experiments repeated on one of our cluster nodes, and for some
reason incremental CMS doesn't look that different anymore. So I guess
it's ok to stick with the non-incremental CMS for now.
- -M
On 08/27/2010 09:12 AM, Peter Schuller wrote:
>> Whether or not this is likely to happen with Cassandra I don't know. I
>> don't know much about the incremental duty cycles are scheduled and it
>> may be the case that Cassandra is not even remotely close to having a
>> problem with incremental mode.
>
> I should further weaken my statement by pointing out that I never did
> any exhaustive tweaking to get around the problem (other than
> disabling incremental mode, since my primary goal has tended to be
> ensure low pause times and not so much even GC activity). It may be
> the case that even in stressful cases where it fails by default it is
> simply a matter of tweaking.
>
> So, I guess I should re-phrase: In terms of just turning on
> incremental mode without at least application specific tweaking (if
> not deployment specific testing), I would suggest caution.
>
>
- --
Dr. Mikio Braun email: mikio@cs.tu-berlin.de
TU Berlin web: ml.cs.tu-berlin.de/~mikio
Franklinstr. 28/29 tel: +49 30 314 78627
10587 Berlin, Germany
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkx3vFUACgkQtnXKX8rQtgDUlgCfWb/euA2mgVJAWDY2tBSyAN+I
604AoKVua1+5bYK2yF9CWwFQmLHDt0Fn
=CIal
-----END PGP SIGNATURE-----
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Peter Schuller <pe...@infidyne.com>.
> Whether or not this is likely to happen with Cassandra I don't know. I
> don't know much about the incremental duty cycles are scheduled and it
> may be the case that Cassandra is not even remotely close to having a
> problem with incremental mode.
I should further weaken my statement by pointing out that I never did
any exhaustive tweaking to get around the problem (other than
disabling incremental mode, since my primary goal has tended to be
ensure low pause times and not so much even GC activity). It may be
the case that even in stressful cases where it fails by default it is
simply a matter of tweaking.
So, I guess I should re-phrase: In terms of just turning on
incremental mode without at least application specific tweaking (if
not deployment specific testing), I would suggest caution.
--
/ Peter Schuller
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Peter Schuller <pe...@infidyne.com>.
> Any reason not to add the incremental mode settings? It's unusual to
> see a Cassandra node out of CPU, but wild heap oscillations are pretty
> common.
FWIW, while I have never tried it with Cassandra, the CMS incremental
mode has never ever worked well for me when I have applied it to
anything that stresses the GC (but these have been small test cases,
not real software). Typically it's just not doing the incremental duty
cycles aggressively enough and you end up falling back to full gc.
Whether or not this is likely to happen with Cassandra I don't know. I
don't know much about the incremental duty cycles are scheduled and it
may be the case that Cassandra is not even remotely close to having a
problem with incremental mode. But I would suggest caution and testing
prior to making it the default.
--
/ Peter Schuller
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Benjamin Black <b...@b3k.us>.
Any reason not to add the incremental mode settings? It's unusual to
see a Cassandra node out of CPU, but wild heap oscillations are pretty
common.
On Thu, Aug 26, 2010 at 8:54 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> added CMSInitiatingOccupancyFraction=80 for 0.6.6 and 0.7beta2
>
> On Thu, Aug 26, 2010 at 9:57 PM, Benjamin Black <b...@b3k.us> wrote:
>> imo, these should be part of the defaults.
>>
>> On Tue, Aug 24, 2010 at 8:29 AM, Mikio Braun <mi...@cs.tu-berlin.de> wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> Dear all,
>>>
>>> thanks again for all the comments I got on my last post. I've played a
>>> bit with different GC settings and got my Cassandra instance to run
>>> very nicely with 8GB of heap.
>>>
>>> I summarized my experiences with GC tuning in this follow-up post:
>>>
>>> http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html
>>>
>>> - -M
>>>
>>> - --
>>> Dr. Mikio Braun email: mikio@cs.tu-berlin.de
>>> TU Berlin web: ml.cs.tu-berlin.de/~mikio
>>> Franklinstr. 28/29 tel: +49 30 314 78627
>>> 10587 Berlin, Germany
>>>
>>>
>>>
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG v1.4.9 (GNU/Linux)
>>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>>
>>> iEYEARECAAYFAkxz5WcACgkQtnXKX8rQtgDiiwCeLknuTcr65eehwIcsivInjv4W
>>> LaQAn3RY9pH19r8SuUhVBvtE6LeyFUvB
>>> =MYsY
>>> -----END PGP SIGNATURE-----
>>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: Follow-up post on Cassandra configuration with some experiments
on GC tuning
Posted by Jonathan Ellis <jb...@gmail.com>.
about a month. it's not like this is a difficult change to apply yourself.
On Fri, Aug 27, 2010 at 1:43 AM, David Dabbs <dm...@gmail.com> wrote:
>>Jonathan Ellis
>>
>>added CMSInitiatingOccupancyFraction=80 for 0.6.6 and 0.7beta2
>
> When do you expect 0.6.6 to land?
>
>
> Thanks,
>
> David
>
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
RE: Follow-up post on Cassandra configuration with some experiments on GC tuning
Posted by David Dabbs <dm...@gmail.com>.
>Jonathan Ellis
>
>added CMSInitiatingOccupancyFraction=80 for 0.6.6 and 0.7beta2
When do you expect 0.6.6 to land?
Thanks,
David
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Jonathan Ellis <jb...@gmail.com>.
added CMSInitiatingOccupancyFraction=80 for 0.6.6 and 0.7beta2
On Thu, Aug 26, 2010 at 9:57 PM, Benjamin Black <b...@b3k.us> wrote:
> imo, these should be part of the defaults.
>
> On Tue, Aug 24, 2010 at 8:29 AM, Mikio Braun <mi...@cs.tu-berlin.de> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Dear all,
>>
>> thanks again for all the comments I got on my last post. I've played a
>> bit with different GC settings and got my Cassandra instance to run
>> very nicely with 8GB of heap.
>>
>> I summarized my experiences with GC tuning in this follow-up post:
>>
>> http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html
>>
>> - -M
>>
>> - --
>> Dr. Mikio Braun email: mikio@cs.tu-berlin.de
>> TU Berlin web: ml.cs.tu-berlin.de/~mikio
>> Franklinstr. 28/29 tel: +49 30 314 78627
>> 10587 Berlin, Germany
>>
>>
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.9 (GNU/Linux)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iEYEARECAAYFAkxz5WcACgkQtnXKX8rQtgDiiwCeLknuTcr65eehwIcsivInjv4W
>> LaQAn3RY9pH19r8SuUhVBvtE6LeyFUvB
>> =MYsY
>> -----END PGP SIGNATURE-----
>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Benjamin Black <b...@b3k.us>.
imo, these should be part of the defaults.
On Tue, Aug 24, 2010 at 8:29 AM, Mikio Braun <mi...@cs.tu-berlin.de> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dear all,
>
> thanks again for all the comments I got on my last post. I've played a
> bit with different GC settings and got my Cassandra instance to run
> very nicely with 8GB of heap.
>
> I summarized my experiences with GC tuning in this follow-up post:
>
> http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html
>
> - -M
>
> - --
> Dr. Mikio Braun email: mikio@cs.tu-berlin.de
> TU Berlin web: ml.cs.tu-berlin.de/~mikio
> Franklinstr. 28/29 tel: +49 30 314 78627
> 10587 Berlin, Germany
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkxz5WcACgkQtnXKX8rQtgDiiwCeLknuTcr65eehwIcsivInjv4W
> LaQAn3RY9pH19r8SuUhVBvtE6LeyFUvB
> =MYsY
> -----END PGP SIGNATURE-----
>
Re: Follow-up post on cassandra configuration with some experiments
on GC tuning
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Aug 24, 2010 at 11:29 AM, Mikio Braun <mi...@cs.tu-berlin.de> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Dear all,
>
> thanks again for all the comments I got on my last post. I've played a
> bit with different GC settings and got my Cassandra instance to run
> very nicely with 8GB of heap.
>
> I summarized my experiences with GC tuning in this follow-up post:
>
> http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html
>
> - -M
>
> - --
> Dr. Mikio Braun email: mikio@cs.tu-berlin.de
> TU Berlin web: ml.cs.tu-berlin.de/~mikio
> Franklinstr. 28/29 tel: +49 30 314 78627
> 10587 Berlin, Germany
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkxz5WcACgkQtnXKX8rQtgDiiwCeLknuTcr65eehwIcsivInjv4W
> LaQAn3RY9pH19r8SuUhVBvtE6LeyFUvB
> =MYsY
> -----END PGP SIGNATURE-----
>
Awesome! I applied your suggestions on one of my nodes. It looks
really really good.
http://www.edwardcapriolo.com/roller/edwardcapriolo/entry/tune_that_jvm