You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Phillip Michalak <ph...@digitalreasoning.com> on 2010/01/21 04:10:20 UTC

'large' node configuration question

Does anyone have a recommendation for configuring cassandra on a  
cluster with 'large' nodes?
i.e. multiple nodes, each >64G RAM, 8 cores, >4 disks

Would it be better to run multiple cassandra processes per node  
(configured on separate IP addresses, separate data directories, etc)  
or just a single mammoth cassandra process on each node? Is there an  
upper bound on what you'd want to allocate to a cassandra process?

Thanks,
Phil

Re: 'large' node configuration question

Posted by Brandon Williams <dr...@gmail.com>.
2010/1/21 Ted Zlatanov <tz...@lifelogs.com>

> On Wed, 20 Jan 2010 21:14:27 -0600 Jonathan Ellis <jb...@gmail.com>
> wrote:
>
> JE> (I only mention that caveat because I don't know how well the JVM
> JE> scales to heaps that large.)
>
> Sun mentions garbage collection issues in the HotSpot FAQ:
> http://java.sun.com/docs/hotspot/HotSpotFAQ.html#64bit_description
>
> Based on that, it seems like a good idea to enable the parallel or
> concurrent garbage collectors with large heaps.  We're looking at this
> at our site as well so I'm curious about people's experiences.


Cassandra already uses the ParNew and CMS GCs by default (in cassandra.in.sh
)

-Brandon

Re: 'large' node configuration question

Posted by Dan Di Spaltro <da...@gmail.com>.
Isn't the -server option the default on 1.6?

2010/1/21 Ted Zlatanov <tz...@lifelogs.com>

> On Thu, 21 Jan 2010 11:04:58 -0600 Jonathan Ellis <jb...@gmail.com>
> wrote:
>
> JE> 2010/1/21 Ted Zlatanov <tz...@lifelogs.com>:
> >> Based on that, it seems like a good idea to enable the parallel or
> >> concurrent garbage collectors with large heaps.  We're looking at this
> >> at our site as well so I'm curious about people's experiences.
>
> JE> Cassandra's default jvm options (bin/cassandra.in.sh) enable the
> concurrent GC.
>
> On Thu, 21 Jan 2010 11:04:35 -0600 Brandon Williams <dr...@gmail.com>
> wrote:
>
> BW> Cassandra already uses the ParNew and CMS GCs by default (in
> cassandra.in.sh)
>
> Are those the best GC choices for Cassandra on a machine like what the
> OP mentioned?  There are many more tuning options:
>
> http://java.sun.com/performance/reference/whitepapers/6_performance.html,
> http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html
> http://blogs.sun.com/watt/resource/jvm-options-list.html
>
> plus the specific settings Cassandra uses:
>
>        -XX:SurvivorRatio=8 \
>        -XX:TargetSurvivorRatio=90 \
>        -XX:+AggressiveOpts \
>        -XX:+UseParNewGC \
>        -XX:+UseConcMarkSweepGC \
>        -XX:+CMSParallelRemarkEnabled \
>        -XX:SurvivorRatio=128 \
>        -XX:MaxTenuringThreshold=0 \
>
> may not be right for a heap 16-64 times larger than the 1 GB heap
> specified in cassandra.in.sh.
>
> Also, maybe these options:
>
>        -ea \
>        -Xdebug \
>        -XX:+HeapDumpOnOutOfMemoryError \
>        -Xrunjdwp:transport=dt_socket,server=y,address=8888,suspend=n \
>
> should go in a "debugging" configuration, triggered by setting
> $CASSANDRA_DEBUG?  With a 60+ GB heap, dumping it to a file could be
> very painful.  It's pretty bad with a smaller heap too.
>
> Finally, is there a reason the -server option is not used?
>
> Thanks
> Ted
>
>


-- 
Dan Di Spaltro

Re: 'large' node configuration question

Posted by Ted Zlatanov <tz...@lifelogs.com>.
On Thu, 21 Jan 2010 14:47:41 -0600 Brandon Williams <dr...@gmail.com> wrote: 

BW> 2010/1/21 Ted Zlatanov <tz...@lifelogs.com>
>> Also, maybe these options:
>> 
>> -ea \
>> -Xdebug \
>> -XX:+HeapDumpOnOutOfMemoryError \
>> -Xrunjdwp:transport=dt_socket,server=y,address=8888,suspend=n \
>> 
>> should go in a "debugging" configuration, triggered by setting
>> $CASSANDRA_DEBUG?  With a 60+ GB heap, dumping it to a file could be
>> very painful.  It's pretty bad with a smaller heap too.

BW> You can always override the CASSANDRA_INCLUDE environment variable
BW> and point it at a file with your own options.

I suggested it as a convenience for the end users who don't want to
override CASSANDRA_INCLUDE and don't need the debugging options on by
default.  Not everyone is an expert on the JVM options.

Ted

Re: 'large' node configuration question

Posted by Brandon Williams <dr...@gmail.com>.
2010/1/21 Ted Zlatanov <tz...@lifelogs.com>

> may not be right for a heap 16-64 times larger than the 1 GB heap
> specified in cassandra.in.sh.
>

Using a heap that large probably does not make sense; you want that ram for
filesystem cache.


> Also, maybe these options:
>
>        -ea \
>        -Xdebug \
>        -XX:+HeapDumpOnOutOfMemoryError \
>        -Xrunjdwp:transport=dt_socket,server=y,address=8888,suspend=n \
>
> should go in a "debugging" configuration, triggered by setting
> $CASSANDRA_DEBUG?  With a 60+ GB heap, dumping it to a file could be
> very painful.  It's pretty bad with a smaller heap too.
>

You can always override the CASSANDRA_INCLUDE environment variable and point
it at a file with your own options.

-Brandon

Re: 'large' node configuration question

Posted by Ted Zlatanov <tz...@lifelogs.com>.
On Thu, 21 Jan 2010 11:04:58 -0600 Jonathan Ellis <jb...@gmail.com> wrote: 

JE> 2010/1/21 Ted Zlatanov <tz...@lifelogs.com>:
>> Based on that, it seems like a good idea to enable the parallel or
>> concurrent garbage collectors with large heaps.  We're looking at this
>> at our site as well so I'm curious about people's experiences.

JE> Cassandra's default jvm options (bin/cassandra.in.sh) enable the concurrent GC.

On Thu, 21 Jan 2010 11:04:35 -0600 Brandon Williams <dr...@gmail.com> wrote: 

BW> Cassandra already uses the ParNew and CMS GCs by default (in cassandra.in.sh)

Are those the best GC choices for Cassandra on a machine like what the
OP mentioned?  There are many more tuning options:

http://java.sun.com/performance/reference/whitepapers/6_performance.html,
http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html
http://blogs.sun.com/watt/resource/jvm-options-list.html

plus the specific settings Cassandra uses:

        -XX:SurvivorRatio=8 \
        -XX:TargetSurvivorRatio=90 \
        -XX:+AggressiveOpts \
        -XX:+UseParNewGC \
        -XX:+UseConcMarkSweepGC \
        -XX:+CMSParallelRemarkEnabled \
        -XX:SurvivorRatio=128 \
        -XX:MaxTenuringThreshold=0 \

may not be right for a heap 16-64 times larger than the 1 GB heap
specified in cassandra.in.sh.

Also, maybe these options:

        -ea \
        -Xdebug \
        -XX:+HeapDumpOnOutOfMemoryError \
        -Xrunjdwp:transport=dt_socket,server=y,address=8888,suspend=n \

should go in a "debugging" configuration, triggered by setting
$CASSANDRA_DEBUG?  With a 60+ GB heap, dumping it to a file could be
very painful.  It's pretty bad with a smaller heap too.

Finally, is there a reason the -server option is not used?

Thanks
Ted

Re: 'large' node configuration question

Posted by Jonathan Ellis <jb...@gmail.com>.
2010/1/21 Ted Zlatanov <tz...@lifelogs.com>:
> Based on that, it seems like a good idea to enable the parallel or
> concurrent garbage collectors with large heaps.  We're looking at this
> at our site as well so I'm curious about people's experiences.

Cassandra's default jvm options (bin/cassandra.in.sh) enable the concurrent GC.

Re: 'large' node configuration question

Posted by Ted Zlatanov <tz...@lifelogs.com>.
On Wed, 20 Jan 2010 21:14:27 -0600 Jonathan Ellis <jb...@gmail.com> wrote: 

JE> On Wed, Jan 20, 2010 at 9:10 PM, Phillip Michalak
JE> <ph...@digitalreasoning.com> wrote:
>> Does anyone have a recommendation for configuring cassandra on a cluster
>> with 'large' nodes?
>> i.e. multiple nodes, each >64G RAM, 8 cores, >4 disks
>> 
>> Would it be better to run multiple cassandra processes per node (configured
>> on separate IP addresses, separate data directories, etc) or just a single
>> mammoth cassandra process on each node? Is there an upper bound on what
>> you'd want to allocate to a cassandra process?

JE> assuming that you're planning to use most of the ram for read caching
JE> and not for the JVM process, I don't see any advantage to justify the
JE> extra complexity of running multiple processes per machine.

JE> (I only mention that caveat because I don't know how well the JVM
JE> scales to heaps that large.)

Sun mentions garbage collection issues in the HotSpot FAQ:
http://java.sun.com/docs/hotspot/HotSpotFAQ.html#64bit_description

Based on that, it seems like a good idea to enable the parallel or
concurrent garbage collectors with large heaps.  We're looking at this
at our site as well so I'm curious about people's experiences.

Ted

Re: 'large' node configuration question

Posted by Jonathan Ellis <jb...@gmail.com>.
assuming that you're planning to use most of the ram for read caching
and not for the JVM process, I don't see any advantage to justify the
extra complexity of running multiple processes per machine.

(I only mention that caveat because I don't know how well the JVM
scales to heaps that large.)

On Wed, Jan 20, 2010 at 9:10 PM, Phillip Michalak
<ph...@digitalreasoning.com> wrote:
> Does anyone have a recommendation for configuring cassandra on a cluster
> with 'large' nodes?
> i.e. multiple nodes, each >64G RAM, 8 cores, >4 disks
>
> Would it be better to run multiple cassandra processes per node (configured
> on separate IP addresses, separate data directories, etc) or just a single
> mammoth cassandra process on each node? Is there an upper bound on what
> you'd want to allocate to a cassandra process?
>
> Thanks,
> Phil
>