You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Tamar Fraenkel <ta...@tok-media.com> on 2012/08/21 11:13:06 UTC
Heap size question
Hi!
I have a question regarding Cassandra heap size.
Cassandra calculates heap size in cassandra-env.sh according to the
following algorythm
# set max heap size based on the following
# max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
# calculate 1/2 ram and cap to 1024MB
# calculate 1/4 ram and cap to 8192MB
# pick the max
So, for
system_memory_in_mb=7468
half_system_memory_in_mb=3734
quarter_system_memory_in_mb=1867
This will result in
max(min(3734,1024), min(1867,8000)) = max(1024,1867)=*1867MB* or in other
words 1/4 of RAM.
In http://www.datastax.com/docs/1.0/operations/tuning it says: "Cassandra's
default configuration opens the JVM with a heap size of 1/4 of the
available system memory (or a minimum 1GB and maximum of 8GB for systems
with a very low or very high amount of RAM). Heapspace should be a minimum
of 1/2 of your RAM, but a maximum of 8GB. The vast majority of deployments
do not benefit from larger heap sizes because (in most cases) the ability
of Java 6 to gracefully handle garbage collection above 8GB quickly
diminishes."
*If I understand this correctly, this means it is better if my heap size
will be 1/2 of RAM, 3734MB.*
I am running on EC2 m1.large instance (7.5 GB memory, 4 EC2 Compute Units
(2 virtual cores with 2 EC2 Compute Units each)).
My system seems to be suffering from lack of memory, and I should probably
increase heap or (and?) reduce key cache size.
Would you recommend changing the heap to half RAM?
If yes, should I hard-code it in acassandra-env.sh?
Thanks!
*Tamar Fraenkel *
Senior Software Engineer, TOK Media
[image: Inline image 1]
tamar@tok-media.com
Tel: +972 2 6409736
Mob: +972 54 8356490
Fax: +972 2 5612956
Re: Heap size question
Posted by Tamar Fraenkel <ta...@tok-media.com>.
Hi!
I am running 1.0.8.
So if I understand correctly both Memtable and Key cache are stored in the
heap. (I don't have row cache)
SSTables are mapped to operating system's virtual memory system, so if I
increase heap I guess there will be less memory for this?
I have seen the changes in 1.1, but we are not there yet.
Thanks,
*Tamar Fraenkel *
Senior Software Engineer, TOK Media
[image: Inline image 1]
tamar@tok-media.com
Tel: +972 2 6409736
Mob: +972 54 8356490
Fax: +972 2 5612956
On Wed, Aug 22, 2012 at 9:23 AM, Thomas Spengler <
thomas.spengler@toptarif.de> wrote:
> Thats not right
>
> since 1.1.X it will used
>
> my hint:
> take a look at
>
> commitlog_total_space_in_mb
> it seams to be maped to off heap usage
> the fix for taking the whole off-heap-memory is included in 1.1.3
>
> the next parameter you have to take a look
>
> disk_access_mode: mmap_index_only
> #disk_access_mode: auto
> #disk_access_mode: standard
>
> and the third
>
> row_cache_provider:
>
> the default has changed from 1.0.X to 1.1.X
> the new one takes off-heap-cache (with and also without jna)
>
> Regards
> Tom
>
> On 08/22/2012 06:37 AM, aaron morton wrote:
> >> "How do I know if my off-heap memory is not used?"
> >
> >
> > If you are using the default memory mapped file access memory not used
> by the cassandra JVM will be used to cache files.
> >
> > Cheers
> >
> > -----------------
> > Aaron Morton
> > Freelance Developer
> > @aaronmorton
> > http://www.thelastpickle.com
> >
> > On 22/08/2012, at 5:17 AM, Tamar Fraenkel <ta...@tok-media.com> wrote:
> >
> >> Much appreciated.
> >> What you described makes a lot of sense from all my readings :)
> >>
> >> Thanks!
> >> Tamar Fraenkel
> >> Senior Software Engineer, TOK Media
> >>
> >> <tokLogo.png>
> >>
> >> tamar@tok-media.com
> >> Tel: +972 2 6409736
> >> Mob: +972 54 8356490
> >> Fax: +972 2 5612956
> >>
> >>
> >>
> >>
> >>
> >> On Tue, Aug 21, 2012 at 6:43 PM, Alain RODRIGUEZ <ar...@gmail.com>
> wrote:
> >> You're welcome. I'll answer to your new questions but keep in mind that
> I am not a cassandra commiter nor even a cassandra specialist.
> >>
> >> "you mean that key cache is not in heap? I am using cassandra 1.0.8 and
> I was under the expression it was, see
> http://www.datastax.com/docs/1.0/operations/tuning, Tuning Java Heap
> Size."
> >>
> >>
> http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-improved-memory-and-disk-space-management
> >>
> >> If I understood this correctly, It seems that only the row cache is
> off-heap. So it's not an issue for us as far as we don't use row cache.
> >>
> >> "I thought that key-cache-size + 1GB + memtable space should not exceed
> heap size. Am I wrong?"
> >>
> >> I don't know if this is a good formula. Datastax gives it so it
> shouldn't be that bad :). However I would say that "key-cache-size + 1GB +
> memtable space" should not exceed 0.75 * Max Heap (where 0.75 is
> flush_largest_memtables_at). I keep default key-cache (which is 5% of max
> heap if I remember well on 1.1.x) and default memtable space (1/3 of max
> heap). I have enlarged my heap from 2 to 4 GB because I had some memory
> pressure (sometimes the Heap Used was greater than 0.75 * Max Heap)
> >>
> >> "WARN [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line
> 145) Heap is 0.7704251937535934 full. You may need to reduce memtable
> and/or cache sizes. Cassandra will now flush up to the two largest
> memtables to free up memory. Adjust flush_largest_memtables_at threshold
> in cassandra.yaml if you don't want Cassandra to do this automatically"
> >>
> >> This message is the memory pressure I was talking about just above.
> >>
> >> "How do I know if my off-heap memory is not used?"
> >>
> >> Well, if you got no row cache and your server is only used as a
> Cassandra node, I'm quite sure you can tune your heap to get 4GB. I guess a
> htop or any memory monitoring system is able to tell you how much your
> memory is used.
> >>
> >> I hope I didn't tell you too much bullshits :p.
> >>
> >> Alain
> >>
> >> 2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
> >> Thanks for you prompt response. Please see follow up questions below
> >> Thanks!!!
> >>
> >>
> >>
> >> Tamar Fraenkel
> >> Senior Software Engineer, TOK Media
> >>
> >> <tokLogo.png>
> >>
> >> tamar@tok-media.com
> >> Tel: +972 2 6409736
> >> Mob: +972 54 8356490
> >> Fax: +972 2 5612956
> >>
> >>
> >>
> >>
> >>
> >> On Tue, Aug 21, 2012 at 12:57 PM, Alain RODRIGUEZ <ar...@gmail.com>
> wrote:
> >> I have the same configuration and I recently change my
> cassandra-sh.yaml to :
> >>
> >> MAX_HEAP_SIZE="4G"
> >> HEAP_NEWSIZE="200M"
> >>
> >> I guess it depends on how much you use the cache (which is now in the
> off-heap memory).
> >>
> >> you mean that key cache is not in heap? I am using cassandra 1.0.8 and
> I was under the expression it was, see
> http://www.datastax.com/docs/1.0/operations/tuning, Tuning Java Heap Size.
> >> I thought that key-cache-size + 1GB + memtable space should not exceed
> heap size. Am I wrong?
> >>
> >>
> >> I don't use row cache and use the default key cache size.
> >> Me too, I have Key Cache capacity of 200000 for all my CFs. Currently
> if my calculations are correct I have about 1.4GB of key cache.
> >>
> >> I have no more memory pressure nor OOM.
> >> I don't see OOM, but I do see messages like the following in my logs:
> >> INFO [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line
> 122) GC for ParNew: 219 ms for 1 collections, 1491982816 used; max is
> 1937768448
> >> WARN [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line
> 145) Heap is 0.7704251937535934 full. You may need to reduce memtable
> and/or cache sizes. Cassandra will now flush up to the two largest
> memtables to free up memory. Adjust flush_largest_memtables_at threshold
> in cassandra.yaml if you don't want Cassandra to do this automatically
> >>
> >>
> >>
> >> I think that if your off-heap memory is unused, it's better enlarging
> the heap (with a max limit of 8GB)
> >>
> >> How do I know if my off-heap memory is not used?
> >>
> >> Hope this will help.
> >>
> >> Alain
> >>
> >> 2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
> >> Hi!
> >> I have a question regarding Cassandra heap size.
> >> Cassandra calculates heap size in cassandra-env.sh according to the
> following algorythm
> >> # set max heap size based on the following
> >> # max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
> >> # calculate 1/2 ram and cap to 1024MB
> >> # calculate 1/4 ram and cap to 8192MB
> >> # pick the max
> >>
> >> So, for
> >> system_memory_in_mb=7468
> >> half_system_memory_in_mb=3734
> >> quarter_system_memory_in_mb=1867
> >> This will result in
> >> max(min(3734,1024), min(1867,8000)) = max(1024,1867)=1867MB or in other
> words 1/4 of RAM.
> >>
> >> In http://www.datastax.com/docs/1.0/operations/tuning it says:
> "Cassandra's default configuration opens the JVM with a heap size of 1/4 of
> the available system memory (or a minimum 1GB and maximum of 8GB for
> systems with a very low or very high amount of RAM). Heapspace should be a
> minimum of 1/2 of your RAM, but a maximum of 8GB. The vast majority of
> deployments do not benefit from larger heap sizes because (in most cases)
> the ability of Java 6 to gracefully handle garbage collection above 8GB
> quickly diminishes."
> >> If I understand this correctly, this means it is better if my heap size
> will be 1/2 of RAM, 3734MB.
> >> I am running on EC2 m1.large instance (7.5 GB memory, 4 EC2 Compute
> Units (2 virtual cores with 2 EC2 Compute Units each)).
> >> My system seems to be suffering from lack of memory, and I should
> probably increase heap or (and?) reduce key cache size.
> >>
> >> Would you recommend changing the heap to half RAM?
> >>
> >> If yes, should I hard-code it in acassandra-env.sh?
> >>
> >> Thanks!
> >>
> >> Tamar Fraenkel
> >> Senior Software Engineer, TOK Media
> >>
> >> <tokLogo.png>
> >>
> >> tamar@tok-media.com
> >> Tel: +972 2 6409736
> >> Mob: +972 54 8356490
> >> Fax: +972 2 5612956
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
>
>
> --
> Thomas Spengler
> Chief Technology Officer
> ------------------------------------------------------------------------
>
> TopTarif Internet GmbH, Pappelallee 78-79, D-10437 Berlin
> Tel.: (030) 2000912 0 | Fax: (030) 2000912 100
> thomas.spengler@toptarif.de | www.toptarif.de
>
> Amtsgericht Charlottenburg, HRB 113287 B
> Geschäftsführer: Dr. Rainer Brosch, Dr. Carolin Gabor
> -------------------------------------------------------------------------
>
Re: Heap size question
Posted by Thomas Spengler <th...@toptarif.de>.
Thats not right
since 1.1.X it will used
my hint:
take a look at
commitlog_total_space_in_mb
it seams to be maped to off heap usage
the fix for taking the whole off-heap-memory is included in 1.1.3
the next parameter you have to take a look
disk_access_mode: mmap_index_only
#disk_access_mode: auto
#disk_access_mode: standard
and the third
row_cache_provider:
the default has changed from 1.0.X to 1.1.X
the new one takes off-heap-cache (with and also without jna)
Regards
Tom
On 08/22/2012 06:37 AM, aaron morton wrote:
>> "How do I know if my off-heap memory is not used?"
>
>
> If you are using the default memory mapped file access memory not used by the cassandra JVM will be used to cache files.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 22/08/2012, at 5:17 AM, Tamar Fraenkel <ta...@tok-media.com> wrote:
>
>> Much appreciated.
>> What you described makes a lot of sense from all my readings :)
>>
>> Thanks!
>> Tamar Fraenkel
>> Senior Software Engineer, TOK Media
>>
>> <tokLogo.png>
>>
>> tamar@tok-media.com
>> Tel: +972 2 6409736
>> Mob: +972 54 8356490
>> Fax: +972 2 5612956
>>
>>
>>
>>
>>
>> On Tue, Aug 21, 2012 at 6:43 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>> You're welcome. I'll answer to your new questions but keep in mind that I am not a cassandra commiter nor even a cassandra specialist.
>>
>> "you mean that key cache is not in heap? I am using cassandra 1.0.8 and I was under the expression it was, see http://www.datastax.com/docs/1.0/operations/tuning, Tuning Java Heap Size."
>>
>> http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-improved-memory-and-disk-space-management
>>
>> If I understood this correctly, It seems that only the row cache is off-heap. So it's not an issue for us as far as we don't use row cache.
>>
>> "I thought that key-cache-size + 1GB + memtable space should not exceed heap size. Am I wrong?"
>>
>> I don't know if this is a good formula. Datastax gives it so it shouldn't be that bad :). However I would say that "key-cache-size + 1GB + memtable space" should not exceed 0.75 * Max Heap (where 0.75 is flush_largest_memtables_at). I keep default key-cache (which is 5% of max heap if I remember well on 1.1.x) and default memtable space (1/3 of max heap). I have enlarged my heap from 2 to 4 GB because I had some memory pressure (sometimes the Heap Used was greater than 0.75 * Max Heap)
>>
>> "WARN [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line 145) Heap is 0.7704251937535934 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically"
>>
>> This message is the memory pressure I was talking about just above.
>>
>> "How do I know if my off-heap memory is not used?"
>>
>> Well, if you got no row cache and your server is only used as a Cassandra node, I'm quite sure you can tune your heap to get 4GB. I guess a htop or any memory monitoring system is able to tell you how much your memory is used.
>>
>> I hope I didn't tell you too much bullshits :p.
>>
>> Alain
>>
>> 2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
>> Thanks for you prompt response. Please see follow up questions below
>> Thanks!!!
>>
>>
>>
>> Tamar Fraenkel
>> Senior Software Engineer, TOK Media
>>
>> <tokLogo.png>
>>
>> tamar@tok-media.com
>> Tel: +972 2 6409736
>> Mob: +972 54 8356490
>> Fax: +972 2 5612956
>>
>>
>>
>>
>>
>> On Tue, Aug 21, 2012 at 12:57 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>> I have the same configuration and I recently change my cassandra-sh.yaml to :
>>
>> MAX_HEAP_SIZE="4G"
>> HEAP_NEWSIZE="200M"
>>
>> I guess it depends on how much you use the cache (which is now in the off-heap memory).
>>
>> you mean that key cache is not in heap? I am using cassandra 1.0.8 and I was under the expression it was, see http://www.datastax.com/docs/1.0/operations/tuning, Tuning Java Heap Size.
>> I thought that key-cache-size + 1GB + memtable space should not exceed heap size. Am I wrong?
>>
>>
>> I don't use row cache and use the default key cache size.
>> Me too, I have Key Cache capacity of 200000 for all my CFs. Currently if my calculations are correct I have about 1.4GB of key cache.
>>
>> I have no more memory pressure nor OOM.
>> I don't see OOM, but I do see messages like the following in my logs:
>> INFO [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line 122) GC for ParNew: 219 ms for 1 collections, 1491982816 used; max is 1937768448
>> WARN [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line 145) Heap is 0.7704251937535934 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
>>
>>
>>
>> I think that if your off-heap memory is unused, it's better enlarging the heap (with a max limit of 8GB)
>>
>> How do I know if my off-heap memory is not used?
>>
>> Hope this will help.
>>
>> Alain
>>
>> 2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
>> Hi!
>> I have a question regarding Cassandra heap size.
>> Cassandra calculates heap size in cassandra-env.sh according to the following algorythm
>> # set max heap size based on the following
>> # max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
>> # calculate 1/2 ram and cap to 1024MB
>> # calculate 1/4 ram and cap to 8192MB
>> # pick the max
>>
>> So, for
>> system_memory_in_mb=7468
>> half_system_memory_in_mb=3734
>> quarter_system_memory_in_mb=1867
>> This will result in
>> max(min(3734,1024), min(1867,8000)) = max(1024,1867)=1867MB or in other words 1/4 of RAM.
>>
>> In http://www.datastax.com/docs/1.0/operations/tuning it says: "Cassandra's default configuration opens the JVM with a heap size of 1/4 of the available system memory (or a minimum 1GB and maximum of 8GB for systems with a very low or very high amount of RAM). Heapspace should be a minimum of 1/2 of your RAM, but a maximum of 8GB. The vast majority of deployments do not benefit from larger heap sizes because (in most cases) the ability of Java 6 to gracefully handle garbage collection above 8GB quickly diminishes."
>> If I understand this correctly, this means it is better if my heap size will be 1/2 of RAM, 3734MB.
>> I am running on EC2 m1.large instance (7.5 GB memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)).
>> My system seems to be suffering from lack of memory, and I should probably increase heap or (and?) reduce key cache size.
>>
>> Would you recommend changing the heap to half RAM?
>>
>> If yes, should I hard-code it in acassandra-env.sh?
>>
>> Thanks!
>>
>> Tamar Fraenkel
>> Senior Software Engineer, TOK Media
>>
>> <tokLogo.png>
>>
>> tamar@tok-media.com
>> Tel: +972 2 6409736
>> Mob: +972 54 8356490
>> Fax: +972 2 5612956
>>
>>
>>
>>
>>
>>
>>
>
>
--
Thomas Spengler
Chief Technology Officer
------------------------------------------------------------------------
TopTarif Internet GmbH, Pappelallee 78-79, D-10437 Berlin
Tel.: (030) 2000912 0 | Fax: (030) 2000912 100
thomas.spengler@toptarif.de | www.toptarif.de
Amtsgericht Charlottenburg, HRB 113287 B
Geschäftsführer: Dr. Rainer Brosch, Dr. Carolin Gabor
-------------------------------------------------------------------------
Re: Heap size question
Posted by aaron morton <aa...@thelastpickle.com>.
> "How do I know if my off-heap memory is not used?"
If you are using the default memory mapped file access memory not used by the cassandra JVM will be used to cache files.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 22/08/2012, at 5:17 AM, Tamar Fraenkel <ta...@tok-media.com> wrote:
> Much appreciated.
> What you described makes a lot of sense from all my readings :)
>
> Thanks!
> Tamar Fraenkel
> Senior Software Engineer, TOK Media
>
> <tokLogo.png>
>
> tamar@tok-media.com
> Tel: +972 2 6409736
> Mob: +972 54 8356490
> Fax: +972 2 5612956
>
>
>
>
>
> On Tue, Aug 21, 2012 at 6:43 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> You're welcome. I'll answer to your new questions but keep in mind that I am not a cassandra commiter nor even a cassandra specialist.
>
> "you mean that key cache is not in heap? I am using cassandra 1.0.8 and I was under the expression it was, see http://www.datastax.com/docs/1.0/operations/tuning, Tuning Java Heap Size."
>
> http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-improved-memory-and-disk-space-management
>
> If I understood this correctly, It seems that only the row cache is off-heap. So it's not an issue for us as far as we don't use row cache.
>
> "I thought that key-cache-size + 1GB + memtable space should not exceed heap size. Am I wrong?"
>
> I don't know if this is a good formula. Datastax gives it so it shouldn't be that bad :). However I would say that "key-cache-size + 1GB + memtable space" should not exceed 0.75 * Max Heap (where 0.75 is flush_largest_memtables_at). I keep default key-cache (which is 5% of max heap if I remember well on 1.1.x) and default memtable space (1/3 of max heap). I have enlarged my heap from 2 to 4 GB because I had some memory pressure (sometimes the Heap Used was greater than 0.75 * Max Heap)
>
> "WARN [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line 145) Heap is 0.7704251937535934 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically"
>
> This message is the memory pressure I was talking about just above.
>
> "How do I know if my off-heap memory is not used?"
>
> Well, if you got no row cache and your server is only used as a Cassandra node, I'm quite sure you can tune your heap to get 4GB. I guess a htop or any memory monitoring system is able to tell you how much your memory is used.
>
> I hope I didn't tell you too much bullshits :p.
>
> Alain
>
> 2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
> Thanks for you prompt response. Please see follow up questions below
> Thanks!!!
>
>
>
> Tamar Fraenkel
> Senior Software Engineer, TOK Media
>
> <tokLogo.png>
>
> tamar@tok-media.com
> Tel: +972 2 6409736
> Mob: +972 54 8356490
> Fax: +972 2 5612956
>
>
>
>
>
> On Tue, Aug 21, 2012 at 12:57 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> I have the same configuration and I recently change my cassandra-sh.yaml to :
>
> MAX_HEAP_SIZE="4G"
> HEAP_NEWSIZE="200M"
>
> I guess it depends on how much you use the cache (which is now in the off-heap memory).
>
> you mean that key cache is not in heap? I am using cassandra 1.0.8 and I was under the expression it was, see http://www.datastax.com/docs/1.0/operations/tuning, Tuning Java Heap Size.
> I thought that key-cache-size + 1GB + memtable space should not exceed heap size. Am I wrong?
>
>
> I don't use row cache and use the default key cache size.
> Me too, I have Key Cache capacity of 200000 for all my CFs. Currently if my calculations are correct I have about 1.4GB of key cache.
>
> I have no more memory pressure nor OOM.
> I don't see OOM, but I do see messages like the following in my logs:
> INFO [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line 122) GC for ParNew: 219 ms for 1 collections, 1491982816 used; max is 1937768448
> WARN [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line 145) Heap is 0.7704251937535934 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
>
>
>
> I think that if your off-heap memory is unused, it's better enlarging the heap (with a max limit of 8GB)
>
> How do I know if my off-heap memory is not used?
>
> Hope this will help.
>
> Alain
>
> 2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
> Hi!
> I have a question regarding Cassandra heap size.
> Cassandra calculates heap size in cassandra-env.sh according to the following algorythm
> # set max heap size based on the following
> # max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
> # calculate 1/2 ram and cap to 1024MB
> # calculate 1/4 ram and cap to 8192MB
> # pick the max
>
> So, for
> system_memory_in_mb=7468
> half_system_memory_in_mb=3734
> quarter_system_memory_in_mb=1867
> This will result in
> max(min(3734,1024), min(1867,8000)) = max(1024,1867)=1867MB or in other words 1/4 of RAM.
>
> In http://www.datastax.com/docs/1.0/operations/tuning it says: "Cassandra's default configuration opens the JVM with a heap size of 1/4 of the available system memory (or a minimum 1GB and maximum of 8GB for systems with a very low or very high amount of RAM). Heapspace should be a minimum of 1/2 of your RAM, but a maximum of 8GB. The vast majority of deployments do not benefit from larger heap sizes because (in most cases) the ability of Java 6 to gracefully handle garbage collection above 8GB quickly diminishes."
> If I understand this correctly, this means it is better if my heap size will be 1/2 of RAM, 3734MB.
> I am running on EC2 m1.large instance (7.5 GB memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)).
> My system seems to be suffering from lack of memory, and I should probably increase heap or (and?) reduce key cache size.
>
> Would you recommend changing the heap to half RAM?
>
> If yes, should I hard-code it in acassandra-env.sh?
>
> Thanks!
>
> Tamar Fraenkel
> Senior Software Engineer, TOK Media
>
> <tokLogo.png>
>
> tamar@tok-media.com
> Tel: +972 2 6409736
> Mob: +972 54 8356490
> Fax: +972 2 5612956
>
>
>
>
>
>
>
Re: Heap size question
Posted by Tamar Fraenkel <ta...@tok-media.com>.
Much appreciated.
What you described makes a lot of sense from all my readings :)
Thanks!
*Tamar Fraenkel *
Senior Software Engineer, TOK Media
[image: Inline image 1]
tamar@tok-media.com
Tel: +972 2 6409736
Mob: +972 54 8356490
Fax: +972 2 5612956
On Tue, Aug 21, 2012 at 6:43 PM, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> You're welcome. I'll answer to your new questions but keep in mind that I
> am not a cassandra commiter nor even a cassandra specialist.
>
> "you mean that key cache is not in heap? I am using cassandra 1.0.8 and I
> was under the expression it was, see
> http://www.datastax.com/docs/1.0/operations/tuning, Tuning Java Heap
> Size."
>
>
> http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-improved-memory-and-disk-space-management
>
> If I understood this correctly, It seems that only the row cache is
> off-heap. So it's not an issue for us as far as we don't use row cache.
>
> "I thought that key-cache-size + 1GB + memtable space should not exceed
> heap size. Am I wrong?"
>
> I don't know if this is a good formula. Datastax gives it so it shouldn't
> be that bad :). However I would say that "key-cache-size + 1GB + memtable
> space" should not exceed 0.75 * Max Heap (where 0.75 is
> flush_largest_memtables_at). I keep default key-cache (which is 5% of max
> heap if I remember well on 1.1.x) and default memtable space (1/3 of max
> heap). I have enlarged my heap from 2 to 4 GB because I had some memory
> pressure (sometimes the Heap Used was greater than 0.75 * Max Heap)
>
> "WARN [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line
> 145) Heap is 0.7704251937535934 full. You may need to reduce memtable
> and/or cache sizes. Cassandra will now flush up to the two largest
> memtables to free up memory. Adjust flush_largest_memtables_at threshold
> in cassandra.yaml if you don't want Cassandra to do this automatically"
>
> This message is the memory pressure I was talking about just above.
>
> "How do I know if my off-heap memory is not used?"
>
> Well, if you got no row cache and your server is only used as a Cassandra
> node, I'm quite sure you can tune your heap to get 4GB. I guess a htop or
> any memory monitoring system is able to tell you how much your memory is
> used.
>
> I hope I didn't tell you too much bullshits :p.
>
> Alain
>
> 2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
>
>> Thanks for you prompt response. Please see follow up questions below
>> Thanks!!!
>>
>>
>>
>> *Tamar Fraenkel *
>> Senior Software Engineer, TOK Media
>>
>> [image: Inline image 1]
>>
>> tamar@tok-media.com
>> Tel: +972 2 6409736
>> Mob: +972 54 8356490
>> Fax: +972 2 5612956
>>
>>
>>
>>
>>
>> On Tue, Aug 21, 2012 at 12:57 PM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>>
>>> I have the same configuration and I recently change my
>>> cassandra-sh.yaml to :
>>>
>>> MAX_HEAP_SIZE="4G"
>>> HEAP_NEWSIZE="200M"
>>>
>>
>>> I guess it depends on how much you use the cache (which is now in the
>>> off-heap memory).
>>>
>>
>> you mean that key cache is not in heap? I am using cassandra 1.0.8 and
>> I was under the expression it was, see
>> http://www.datastax.com/docs/1.0/operations/tuning, Tuning Java Heap
>> Size.
>> I thought that key-cache-size + 1GB + memtable space should not exceed
>> heap size. Am I wrong?
>>
>>
>>> I don't use row cache and use the default key cache size.
>>>
>> Me too, I have Key Cache capacity of 200000 for all my CFs. Currently if
>> my calculations are correct I have about 1.4GB of key cache.
>>
>>>
>>> I have no more memory pressure nor OOM.
>>>
>> I don't see OOM, but I do see messages like the following in my logs:
>> INFO [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line
>> 122) GC for ParNew: 219 ms for 1 collections, 1491982816 used; max is
>> 1937768448
>> WARN [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line
>> 145) Heap is 0.7704251937535934 full. You may need to reduce memtable
>> and/or cache sizes. Cassandra will now flush up to the two largest
>> memtables to free up memory. Adjust flush_largest_memtables_at threshold
>> in cassandra.yaml if you don't want Cassandra to do this automatically
>>
>>
>>
>>> I think that if your off-heap memory is unused, it's better enlarging
>>> the heap (with a max limit of 8GB)
>>>
>>> How do I know if my off-heap memory is not used?
>>
>>
>>> Hope this will help.
>>>
>>> Alain
>>>
>>> 2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
>>>
>>>> Hi!
>>>> I have a question regarding Cassandra heap size.
>>>> Cassandra calculates heap size in cassandra-env.sh according to the
>>>> following algorythm
>>>> # set max heap size based on the following
>>>> # max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
>>>> # calculate 1/2 ram and cap to 1024MB
>>>> # calculate 1/4 ram and cap to 8192MB
>>>> # pick the max
>>>>
>>>> So, for
>>>> system_memory_in_mb=7468
>>>> half_system_memory_in_mb=3734
>>>> quarter_system_memory_in_mb=1867
>>>> This will result in
>>>> max(min(3734,1024), min(1867,8000)) = max(1024,1867)=*1867MB* or in
>>>> other words 1/4 of RAM.
>>>>
>>>> In http://www.datastax.com/docs/1.0/operations/tuning it says: "Cassandra's
>>>> default configuration opens the JVM with a heap size of 1/4 of the
>>>> available system memory (or a minimum 1GB and maximum of 8GB for systems
>>>> with a very low or very high amount of RAM). Heapspace should be a minimum
>>>> of 1/2 of your RAM, but a maximum of 8GB. The vast majority of deployments
>>>> do not benefit from larger heap sizes because (in most cases) the ability
>>>> of Java 6 to gracefully handle garbage collection above 8GB quickly
>>>> diminishes."
>>>> *If I understand this correctly, this means it is better if my heap
>>>> size will be 1/2 of RAM, 3734MB.*
>>>> I am running on EC2 m1.large instance (7.5 GB memory, 4 EC2 Compute
>>>> Units (2 virtual cores with 2 EC2 Compute Units each)).
>>>> My system seems to be suffering from lack of memory, and I should
>>>> probably increase heap or (and?) reduce key cache size.
>>>>
>>>> Would you recommend changing the heap to half RAM?
>>>>
>>>> If yes, should I hard-code it in acassandra-env.sh?
>>>>
>>>> Thanks!
>>>>
>>>> *Tamar Fraenkel *
>>>> Senior Software Engineer, TOK Media
>>>>
>>>> [image: Inline image 1]
>>>>
>>>> tamar@tok-media.com
>>>> Tel: +972 2 6409736
>>>> Mob: +972 54 8356490
>>>> Fax: +972 2 5612956
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: Heap size question
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
You're welcome. I'll answer to your new questions but keep in mind that I
am not a cassandra commiter nor even a cassandra specialist.
"you mean that key cache is not in heap? I am using cassandra 1.0.8 and I
was under the expression it was, see
http://www.datastax.com/docs/1.0/operations/tuning, Tuning Java Heap Size."
http://www.datastax.com/dev/blog/whats-new-in-cassandra-1-0-improved-memory-and-disk-space-management
If I understood this correctly, It seems that only the row cache is
off-heap. So it's not an issue for us as far as we don't use row cache.
"I thought that key-cache-size + 1GB + memtable space should not exceed
heap size. Am I wrong?"
I don't know if this is a good formula. Datastax gives it so it shouldn't
be that bad :). However I would say that "key-cache-size + 1GB + memtable
space" should not exceed 0.75 * Max Heap (where 0.75 is
flush_largest_memtables_at). I keep default key-cache (which is 5% of max
heap if I remember well on 1.1.x) and default memtable space (1/3 of max
heap). I have enlarged my heap from 2 to 4 GB because I had some memory
pressure (sometimes the Heap Used was greater than 0.75 * Max Heap)
"WARN [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line
145) Heap is 0.7704251937535934 full. You may need to reduce memtable
and/or cache sizes. Cassandra will now flush up to the two largest
memtables to free up memory. Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically"
This message is the memory pressure I was talking about just above.
"How do I know if my off-heap memory is not used?"
Well, if you got no row cache and your server is only used as a Cassandra
node, I'm quite sure you can tune your heap to get 4GB. I guess a htop or
any memory monitoring system is able to tell you how much your memory is
used.
I hope I didn't tell you too much bullshits :p.
Alain
2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
> Thanks for you prompt response. Please see follow up questions below
> Thanks!!!
>
>
>
> *Tamar Fraenkel *
> Senior Software Engineer, TOK Media
>
> [image: Inline image 1]
>
> tamar@tok-media.com
> Tel: +972 2 6409736
> Mob: +972 54 8356490
> Fax: +972 2 5612956
>
>
>
>
>
> On Tue, Aug 21, 2012 at 12:57 PM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
>
>> I have the same configuration and I recently change my cassandra-sh.yaml
>> to :
>>
>> MAX_HEAP_SIZE="4G"
>> HEAP_NEWSIZE="200M"
>>
>
>> I guess it depends on how much you use the cache (which is now in the
>> off-heap memory).
>>
>
> you mean that key cache is not in heap? I am using cassandra 1.0.8 and I
> was under the expression it was, see
> http://www.datastax.com/docs/1.0/operations/tuning, Tuning Java Heap Size.
> I thought that key-cache-size + 1GB + memtable space should not exceed
> heap size. Am I wrong?
>
>
>> I don't use row cache and use the default key cache size.
>>
> Me too, I have Key Cache capacity of 200000 for all my CFs. Currently if
> my calculations are correct I have about 1.4GB of key cache.
>
>>
>> I have no more memory pressure nor OOM.
>>
> I don't see OOM, but I do see messages like the following in my logs:
> INFO [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line
> 122) GC for ParNew: 219 ms for 1 collections, 1491982816 used; max is
> 1937768448
> WARN [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line
> 145) Heap is 0.7704251937535934 full. You may need to reduce memtable
> and/or cache sizes. Cassandra will now flush up to the two largest
> memtables to free up memory. Adjust flush_largest_memtables_at threshold
> in cassandra.yaml if you don't want Cassandra to do this automatically
>
>
>
>> I think that if your off-heap memory is unused, it's better enlarging the
>> heap (with a max limit of 8GB)
>>
>> How do I know if my off-heap memory is not used?
>
>
>> Hope this will help.
>>
>> Alain
>>
>> 2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
>>
>>> Hi!
>>> I have a question regarding Cassandra heap size.
>>> Cassandra calculates heap size in cassandra-env.sh according to the
>>> following algorythm
>>> # set max heap size based on the following
>>> # max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
>>> # calculate 1/2 ram and cap to 1024MB
>>> # calculate 1/4 ram and cap to 8192MB
>>> # pick the max
>>>
>>> So, for
>>> system_memory_in_mb=7468
>>> half_system_memory_in_mb=3734
>>> quarter_system_memory_in_mb=1867
>>> This will result in
>>> max(min(3734,1024), min(1867,8000)) = max(1024,1867)=*1867MB* or in
>>> other words 1/4 of RAM.
>>>
>>> In http://www.datastax.com/docs/1.0/operations/tuning it says: "Cassandra's
>>> default configuration opens the JVM with a heap size of 1/4 of the
>>> available system memory (or a minimum 1GB and maximum of 8GB for systems
>>> with a very low or very high amount of RAM). Heapspace should be a minimum
>>> of 1/2 of your RAM, but a maximum of 8GB. The vast majority of deployments
>>> do not benefit from larger heap sizes because (in most cases) the ability
>>> of Java 6 to gracefully handle garbage collection above 8GB quickly
>>> diminishes."
>>> *If I understand this correctly, this means it is better if my heap
>>> size will be 1/2 of RAM, 3734MB.*
>>> I am running on EC2 m1.large instance (7.5 GB memory, 4 EC2 Compute
>>> Units (2 virtual cores with 2 EC2 Compute Units each)).
>>> My system seems to be suffering from lack of memory, and I should
>>> probably increase heap or (and?) reduce key cache size.
>>>
>>> Would you recommend changing the heap to half RAM?
>>>
>>> If yes, should I hard-code it in acassandra-env.sh?
>>>
>>> Thanks!
>>>
>>> *Tamar Fraenkel *
>>> Senior Software Engineer, TOK Media
>>>
>>> [image: Inline image 1]
>>>
>>> tamar@tok-media.com
>>> Tel: +972 2 6409736
>>> Mob: +972 54 8356490
>>> Fax: +972 2 5612956
>>>
>>>
>>>
>>>
>>
>
Re: Heap size question
Posted by Tamar Fraenkel <ta...@tok-media.com>.
Thanks for you prompt response. Please see follow up questions below
Thanks!!!
*Tamar Fraenkel *
Senior Software Engineer, TOK Media
[image: Inline image 1]
tamar@tok-media.com
Tel: +972 2 6409736
Mob: +972 54 8356490
Fax: +972 2 5612956
On Tue, Aug 21, 2012 at 12:57 PM, Alain RODRIGUEZ <ar...@gmail.com>wrote:
> I have the same configuration and I recently change my cassandra-sh.yaml
> to :
>
> MAX_HEAP_SIZE="4G"
> HEAP_NEWSIZE="200M"
>
> I guess it depends on how much you use the cache (which is now in the
> off-heap memory).
>
you mean that key cache is not in heap? I am using cassandra 1.0.8 and I
was under the expression it was, see
http://www.datastax.com/docs/1.0/operations/tuning, Tuning Java Heap Size.
I thought that key-cache-size + 1GB + memtable space should not exceed
heap size. Am I wrong?
> I don't use row cache and use the default key cache size.
>
Me too, I have Key Cache capacity of 200000 for all my CFs. Currently if my
calculations are correct I have about 1.4GB of key cache.
>
> I have no more memory pressure nor OOM.
>
I don't see OOM, but I do see messages like the following in my logs:
INFO [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line 122)
GC for ParNew: 219 ms for 1 collections, 1491982816 used; max is 1937768448
WARN [ScheduledTasks:1] 2012-08-20 12:31:46,506 GCInspector.java (line
145) Heap is 0.7704251937535934 full. You may need to reduce memtable
and/or cache sizes. Cassandra will now flush up to the two largest
memtables to free up memory. Adjust flush_largest_memtables_at threshold
in cassandra.yaml if you don't want Cassandra to do this automatically
> I think that if your off-heap memory is unused, it's better enlarging the
> heap (with a max limit of 8GB)
>
> How do I know if my off-heap memory is not used?
> Hope this will help.
>
> Alain
>
> 2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
>
>> Hi!
>> I have a question regarding Cassandra heap size.
>> Cassandra calculates heap size in cassandra-env.sh according to the
>> following algorythm
>> # set max heap size based on the following
>> # max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
>> # calculate 1/2 ram and cap to 1024MB
>> # calculate 1/4 ram and cap to 8192MB
>> # pick the max
>>
>> So, for
>> system_memory_in_mb=7468
>> half_system_memory_in_mb=3734
>> quarter_system_memory_in_mb=1867
>> This will result in
>> max(min(3734,1024), min(1867,8000)) = max(1024,1867)=*1867MB* or in
>> other words 1/4 of RAM.
>>
>> In http://www.datastax.com/docs/1.0/operations/tuning it says: "Cassandra's
>> default configuration opens the JVM with a heap size of 1/4 of the
>> available system memory (or a minimum 1GB and maximum of 8GB for systems
>> with a very low or very high amount of RAM). Heapspace should be a minimum
>> of 1/2 of your RAM, but a maximum of 8GB. The vast majority of deployments
>> do not benefit from larger heap sizes because (in most cases) the ability
>> of Java 6 to gracefully handle garbage collection above 8GB quickly
>> diminishes."
>> *If I understand this correctly, this means it is better if my heap size
>> will be 1/2 of RAM, 3734MB.*
>> I am running on EC2 m1.large instance (7.5 GB memory, 4 EC2 Compute
>> Units (2 virtual cores with 2 EC2 Compute Units each)).
>> My system seems to be suffering from lack of memory, and I should
>> probably increase heap or (and?) reduce key cache size.
>>
>> Would you recommend changing the heap to half RAM?
>>
>> If yes, should I hard-code it in acassandra-env.sh?
>>
>> Thanks!
>>
>> *Tamar Fraenkel *
>> Senior Software Engineer, TOK Media
>>
>> [image: Inline image 1]
>>
>> tamar@tok-media.com
>> Tel: +972 2 6409736
>> Mob: +972 54 8356490
>> Fax: +972 2 5612956
>>
>>
>>
>>
>
Re: Heap size question
Posted by Alain RODRIGUEZ <ar...@gmail.com>.
I have the same configuration and I recently change my cassandra-sh.yaml
to :
MAX_HEAP_SIZE="4G"
HEAP_NEWSIZE="200M"
I guess it depends on how much you use the cache (which is now in the
off-heap memory).
I don't use row cache and use the default key cache size.
I have no more memory pressure nor OOM.
I think that if your off-heap memory is unused, it's better enlarging the
heap (with a max limit of 8GB)
Hope this will help.
Alain
2012/8/21 Tamar Fraenkel <ta...@tok-media.com>
> Hi!
> I have a question regarding Cassandra heap size.
> Cassandra calculates heap size in cassandra-env.sh according to the
> following algorythm
> # set max heap size based on the following
> # max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
> # calculate 1/2 ram and cap to 1024MB
> # calculate 1/4 ram and cap to 8192MB
> # pick the max
>
> So, for
> system_memory_in_mb=7468
> half_system_memory_in_mb=3734
> quarter_system_memory_in_mb=1867
> This will result in
> max(min(3734,1024), min(1867,8000)) = max(1024,1867)=*1867MB* or in other
> words 1/4 of RAM.
>
> In http://www.datastax.com/docs/1.0/operations/tuning it says: "Cassandra's
> default configuration opens the JVM with a heap size of 1/4 of the
> available system memory (or a minimum 1GB and maximum of 8GB for systems
> with a very low or very high amount of RAM). Heapspace should be a minimum
> of 1/2 of your RAM, but a maximum of 8GB. The vast majority of deployments
> do not benefit from larger heap sizes because (in most cases) the ability
> of Java 6 to gracefully handle garbage collection above 8GB quickly
> diminishes."
> *If I understand this correctly, this means it is better if my heap size
> will be 1/2 of RAM, 3734MB.*
> I am running on EC2 m1.large instance (7.5 GB memory, 4 EC2 Compute Units
> (2 virtual cores with 2 EC2 Compute Units each)).
> My system seems to be suffering from lack of memory, and I should probably
> increase heap or (and?) reduce key cache size.
>
> Would you recommend changing the heap to half RAM?
>
> If yes, should I hard-code it in acassandra-env.sh?
>
> Thanks!
>
> *Tamar Fraenkel *
> Senior Software Engineer, TOK Media
>
> [image: Inline image 1]
>
> tamar@tok-media.com
> Tel: +972 2 6409736
> Mob: +972 54 8356490
> Fax: +972 2 5612956
>
>
>
>