Re: CMS GC initial-mark taking 6 seconds , bad?

[Maxim Potekhin <po...@bnl.gov>]

Hello Aaron,

I happen to have 48GB on each machines I use in the cluster. Can I 
assume that I can't really use all of this memory productively? Do you 
have any suggestion related to that? Can I run more than one instance on 
Cassandra on the same box (using different ports) to take advantage of 
this memory, assuming the disk has enough bandwidth?

Thanks,
Maxim

On 9/25/2011 11:37 AM, aaron morton wrote:
> It does seem long and will be felt by your application.
>
> Are you running a 47GB heap ? Most peeps seem to think 8 to 12 is about the viable maximum.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 25/09/2011, at 7:14 PM, Yang wrote:
>
>> I see the following in my GC log
>>
>> 1910.513: [GC [1 CMS-initial-mark: 2598619K(26214400K)]
>> 13749939K(49807360K), 6.0696680 secs] [Times: user=6.10 sys=0.00,
>> real=6.07 secs]
>>
>> so there is a stop-the-world period of 6 seconds. does this sound bad
>> ? or 6 seconds is OK  and we should expect the built-in
>> fault-tolerance of Cassandra handle this?
>>
>> Thanks
>> Yang

Re: CMS GC initial-mark taking 6 seconds , bad?

[Chris Burroughs <ch...@gmail.com>]

On 10/20/2011 09:38 AM, Maxim Potekhin wrote:
> I happen to have 48GB on each machines I use in the cluster. Can I
> assume that I can't really use all of this memory productively? Do you
> have any suggestion related to that? Can I run more than one instance on
> Cassandra on the same box (using different ports) to take advantage of
> this memory, assuming the disk has enough bandwidth?

You are likely to not have good luck with a JVM heap that large.  But
you can:
 - Leave all that memory to the OS page cache.
 - mmap index files
 - use an off heap cache

All of those are productive uses.

Re: CMS GC initial-mark taking 6 seconds , bad?

[Todd Burruss <bb...@expedia.com>]

Are you using cassandra's caching?  If you are then you will need to play
around with the RAM setting to find a sweet spot.  A low hit rate on the
cache (which is counter productive anyway) will cause more GC.  A high hit
rate, less GC.

If you are not caching, no need to use a large heap as the OS will do a
fine job of caching the data files for you.  And in fact I would run with
a smaller'ish heap to give the OS more room - how much depends on a
variety of things (how many column families, how often repair, write/read
ratio) - so testing is the best course of action


On 10/20/11 6:38 AM, "Maxim Potekhin" <po...@bnl.gov> wrote:

>Hello Aaron,
>
>I happen to have 48GB on each machines I use in the cluster. Can I
>assume that I can't really use all of this memory productively? Do you
>have any suggestion related to that? Can I run more than one instance on
>Cassandra on the same box (using different ports) to take advantage of
>this memory, assuming the disk has enough bandwidth?
>
>Thanks,
>Maxim
>
>On 9/25/2011 11:37 AM, aaron morton wrote:
>> It does seem long and will be felt by your application.
>>
>> Are you running a 47GB heap ? Most peeps seem to think 8 to 12 is about
>>the viable maximum.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 25/09/2011, at 7:14 PM, Yang wrote:
>>
>>> I see the following in my GC log
>>>
>>> 1910.513: [GC [1 CMS-initial-mark: 2598619K(26214400K)]
>>> 13749939K(49807360K), 6.0696680 secs] [Times: user=6.10 sys=0.00,
>>> real=6.07 secs]
>>>
>>> so there is a stop-the-world period of 6 seconds. does this sound bad
>>> ? or 6 seconds is OK  and we should expect the built-in
>>> fault-tolerance of Cassandra handle this?
>>>
>>> Thanks
>>> Yang
>