You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Arindam Barua <ab...@247-inc.com> on 2013/11/01 02:59:49 UTC
RE: Heap almost full
Thank you for your responses. In another recent test, the heap actually got full, and we got an out of memory error. When we analyzed the heap, almost all of it was memtables. Is there any known issue with 1.1.5 which causes memtable_total_space_in_mb not to be respected, or not defaulting to 1/3rd of the heap size? Or is it possible that the load in the test is that high that Cassandra is not able to keep flushing even though it starts the process when memtable_total_space_in_mb is 1/3rd of the heap?
We recently switched to LeveledCompaction, however, when we got the earlier heap warning, that was running on SizeTiered.
The latest test was running on high performance 32-core, 128 GB RAM, 7 RAID-0 1TB disks (regular). Earlier tests were run on lesser hardware with the same load, but there was no memory problem. We are running more tests to check if this is always reproducible.
Answering some of the earlier questions if it helps:
We have Cassandra 1.1.5 running in production. Upgrading to the latest 1.2.x release is on the roadmap, but till then this needs to be figured out.
> - How many data do you got per node ?
We are running into these errors while running tests in QA starting with 0 load. These are around 4 hr tests which end up adding under 1 GB of data on each node of a 4-node ring, or a 2-node ring.
> - What is the value of the "index_intval" (cassandra.yaml) ?
It's the default value of 128.
Thanks,
Arindam
-----Original Message-----
From: Aaron Morton [mailto:aaron@thelastpickle.com]
Sent: Monday, October 28, 2013 12:09 AM
To: Cassandra User
Subject: Re: Heap almost full
> 1] [14/10/2013:19:15:08 PDT] ScheduledTasks:1: WARN GCInspector.java (line 145) Heap is 0.8287082580489245 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
This means that the CMS GC was unable to free memory quickly, you've not run out but may do under heavy load.
CMS uses CPU resources to do it's job, how much CPU do you have available ?
To check the behaviour of the CMS collector using JConsole or another tool to watch the heap size, you should see a nice saw tooth graph. It should gradually grow then drop quickly to below 3ish GB. If the size of CMS is not low enough you will spend more time in GC.
You may also want to adjust flush_largest_memtables_at to be .8 to give CMS a chance to do it's work. It starts at .75
> In 1.2+ bloomfilters are off-heap, you can use vnodes...
+1 for 1.2 with off heap bloom filters.
> - increasing the heap to 10GB.
-1
Unless you have a node under heavy memory problems, pre 1.2 with 1+billion rows and lots of bloom filters, increasing the heap is not the answer. It will increase the time taken for ParNew CMS and in kicks the problem down the road.
Cheers
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 26/10/2013, at 8:32 am, Alain RODRIGUEZ <ar...@gmail.com> wrote:
> If you are starting with Cassandra I really advice you to start with 1.2.11
>
> In 1.2+ bloomfilters are off-heap, you can use vnodes...
>
> "I summed up the bloom filter usage reported by nodetool cfstats in all the CFs and it was under 50 MB."
>
> This is quite a small value. Is there no error in your conversion from Bytes read in cfstats ?
>
> If you are trying to understand this could you tell us :
>
> - How many data do you got per node ?
> - What is the value of the "index_intval" (cassandra.yaml) ?
>
> If you are trying to fix this, you can try :
>
> - changing the "memtable_total_space_in_mb" to 1024
> - increasing the heap to 10GB.
>
> Hope this will help somehow :).
>
> Good luck
>
>
> 2013/10/16 Arindam Barua <ab...@247-inc.com>
>
>
> During performance testing being run on our 4 node Cassandra 1.1.5 cluster, we are seeing warning logs about the heap being almost full - [1]. I'm trying to figure out why, and how to prevent it.
>
>
>
> The tests are being run on a Cassandra ring consisting of 4 dedicated boxes with 32 GB of RAM each.
>
> The heap size is set to 8 GB as recommended.
>
> All the other relevant settings I know off are the default ones:
>
> - memtable_total_space_in_mb is not set in the yaml, so should default to 1/3rd the heap size.
>
> - They key cache should be 100 MB at the most. I checked the key cache the day after the tests were run via nodetool info, and it reported 4.5 MB being used.
>
> - row cache is not being used
>
> - I summed up the bloom filter usage reported by nodetool cfstats in all the CFs and it was under 50 MB.
>
>
>
> The resident size of the cassandra process accd to top is 8.4g even now. Did a heap histogram using jmap, but not sure how to interpret those results usefully - [2].
>
>
>
> Performance test details:
>
> - The test is write only, and is writing relatively large amount of data to one CF.
>
> - There is some other traffic that is constantly on that writes smaller amounts of data to many CFs, and does some reads.
>
>
>
> The total number of CFs are 114, but quite a few of them are not used.
>
>
>
> Thanks,
>
> Arindam
>
>
>
> [1] [14/10/2013:19:15:08 PDT] ScheduledTasks:1: WARN GCInspector.java (line 145) Heap is 0.8287082580489245 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
>
>
>
> [2] Object Histogram:
>
>
>
> num #instances #bytes Class description
>
> --------------------------------------------------------------------------
>
> 1: 152855 86035312 int[]
>
> 2: 13395 45388008 long[]
>
> 3: 49517 9712000 java.lang.Object[]
>
> 4: 120094 8415560 char[]
>
> 5: 145106 6965088 java.nio.HeapByteBuffer
>
> 6: 40525 5891040 * ConstMethodKlass
>
> 7: 231258 5550192 java.lang.Long
>
> 8: 40525 5521592 * MethodKlass
>
> 9: 134574 5382960 java.math.BigInteger
>
> 10: 36692 4403040 java.net.SocksSocketImpl
>
> 11: 3741 4385048 * ConstantPoolKlass
>
> 12: 63875 3538128 * SymbolKlass
>
> 13: 104048 3329536 java.lang.String
>
> 14: 132636 3183264 org.apache.cassandra.db.DecoratedKey
>
> 15: 97466 3118912 java.util.concurrent.ConcurrentHashMap$HashEntry
>
> 16: 97216 3110912 com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
>
>
>
>
>
Re: Heap almost full
Posted by Aaron Morton <aa...@thelastpickle.com>.
> When we analyzed the heap, almost all of it was memtables.
What were the top classes ?
I would normally expect an OOM in pre 1.2 days to be the result of bloom filters, compaction meta data and index samples.
> Is there any known issue with 1.1.5 which causes memtable_total_space_in_mb not to be respected, or not defaulting to 1/3rd of the heap size?
Nothing I can remember.
We estimate the in memory size of the memtables using the live ratio. That’s been pretty good for a while now, but you may want to check the change log for changes there.
> The latest test was running on high performance 32-core, 128 GB RAM, 7 RAID-0 1TB disks (regular).
With all those cores grab the TLAB setting from the 1.2 cassandra-env.sh file.
Cheers
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 1/11/2013, at 2:59 pm, Arindam Barua <ab...@247-inc.com> wrote:
>
> Thank you for your responses. In another recent test, the heap actually got full, and we got an out of memory error. When we analyzed the heap, almost all of it was memtables. Is there any known issue with 1.1.5 which causes memtable_total_space_in_mb not to be respected, or not defaulting to 1/3rd of the heap size? Or is it possible that the load in the test is that high that Cassandra is not able to keep flushing even though it starts the process when memtable_total_space_in_mb is 1/3rd of the heap?
>
> We recently switched to LeveledCompaction, however, when we got the earlier heap warning, that was running on SizeTiered.
> The latest test was running on high performance 32-core, 128 GB RAM, 7 RAID-0 1TB disks (regular). Earlier tests were run on lesser hardware with the same load, but there was no memory problem. We are running more tests to check if this is always reproducible.
>
> Answering some of the earlier questions if it helps:
>
> We have Cassandra 1.1.5 running in production. Upgrading to the latest 1.2.x release is on the roadmap, but till then this needs to be figured out.
>
>> - How many data do you got per node ?
> We are running into these errors while running tests in QA starting with 0 load. These are around 4 hr tests which end up adding under 1 GB of data on each node of a 4-node ring, or a 2-node ring.
>
>> - What is the value of the "index_intval" (cassandra.yaml) ?
> It's the default value of 128.
>
> Thanks,
> Arindam
>
> -----Original Message-----
> From: Aaron Morton [mailto:aaron@thelastpickle.com]
> Sent: Monday, October 28, 2013 12:09 AM
> To: Cassandra User
> Subject: Re: Heap almost full
>
>> 1] [14/10/2013:19:15:08 PDT] ScheduledTasks:1: WARN GCInspector.java (line 145) Heap is 0.8287082580489245 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
> This means that the CMS GC was unable to free memory quickly, you've not run out but may do under heavy load.
>
> CMS uses CPU resources to do it's job, how much CPU do you have available ?
> To check the behaviour of the CMS collector using JConsole or another tool to watch the heap size, you should see a nice saw tooth graph. It should gradually grow then drop quickly to below 3ish GB. If the size of CMS is not low enough you will spend more time in GC.
>
> You may also want to adjust flush_largest_memtables_at to be .8 to give CMS a chance to do it's work. It starts at .75
>
>> In 1.2+ bloomfilters are off-heap, you can use vnodes...
> +1 for 1.2 with off heap bloom filters.
>
>> - increasing the heap to 10GB.
>
> -1
> Unless you have a node under heavy memory problems, pre 1.2 with 1+billion rows and lots of bloom filters, increasing the heap is not the answer. It will increase the time taken for ParNew CMS and in kicks the problem down the road.
>
> Cheers
>
> -----------------
> Aaron Morton
> New Zealand
> @aaronmorton
>
> Co-Founder & Principal Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> On 26/10/2013, at 8:32 am, Alain RODRIGUEZ <ar...@gmail.com> wrote:
>
>> If you are starting with Cassandra I really advice you to start with 1.2.11
>>
>> In 1.2+ bloomfilters are off-heap, you can use vnodes...
>>
>> "I summed up the bloom filter usage reported by nodetool cfstats in all the CFs and it was under 50 MB."
>>
>> This is quite a small value. Is there no error in your conversion from Bytes read in cfstats ?
>>
>> If you are trying to understand this could you tell us :
>>
>> - How many data do you got per node ?
>> - What is the value of the "index_intval" (cassandra.yaml) ?
>>
>> If you are trying to fix this, you can try :
>>
>> - changing the "memtable_total_space_in_mb" to 1024
>> - increasing the heap to 10GB.
>>
>> Hope this will help somehow :).
>>
>> Good luck
>>
>>
>> 2013/10/16 Arindam Barua <ab...@247-inc.com>
>>
>>
>> During performance testing being run on our 4 node Cassandra 1.1.5 cluster, we are seeing warning logs about the heap being almost full - [1]. I'm trying to figure out why, and how to prevent it.
>>
>>
>>
>> The tests are being run on a Cassandra ring consisting of 4 dedicated boxes with 32 GB of RAM each.
>>
>> The heap size is set to 8 GB as recommended.
>>
>> All the other relevant settings I know off are the default ones:
>>
>> - memtable_total_space_in_mb is not set in the yaml, so should default to 1/3rd the heap size.
>>
>> - They key cache should be 100 MB at the most. I checked the key cache the day after the tests were run via nodetool info, and it reported 4.5 MB being used.
>>
>> - row cache is not being used
>>
>> - I summed up the bloom filter usage reported by nodetool cfstats in all the CFs and it was under 50 MB.
>>
>>
>>
>> The resident size of the cassandra process accd to top is 8.4g even now. Did a heap histogram using jmap, but not sure how to interpret those results usefully - [2].
>>
>>
>>
>> Performance test details:
>>
>> - The test is write only, and is writing relatively large amount of data to one CF.
>>
>> - There is some other traffic that is constantly on that writes smaller amounts of data to many CFs, and does some reads.
>>
>>
>>
>> The total number of CFs are 114, but quite a few of them are not used.
>>
>>
>>
>> Thanks,
>>
>> Arindam
>>
>>
>>
>> [1] [14/10/2013:19:15:08 PDT] ScheduledTasks:1: WARN GCInspector.java (line 145) Heap is 0.8287082580489245 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
>>
>>
>>
>> [2] Object Histogram:
>>
>>
>>
>> num #instances #bytes Class description
>>
>> --------------------------------------------------------------------------
>>
>> 1: 152855 86035312 int[]
>>
>> 2: 13395 45388008 long[]
>>
>> 3: 49517 9712000 java.lang.Object[]
>>
>> 4: 120094 8415560 char[]
>>
>> 5: 145106 6965088 java.nio.HeapByteBuffer
>>
>> 6: 40525 5891040 * ConstMethodKlass
>>
>> 7: 231258 5550192 java.lang.Long
>>
>> 8: 40525 5521592 * MethodKlass
>>
>> 9: 134574 5382960 java.math.BigInteger
>>
>> 10: 36692 4403040 java.net.SocksSocketImpl
>>
>> 11: 3741 4385048 * ConstantPoolKlass
>>
>> 12: 63875 3538128 * SymbolKlass
>>
>> 13: 104048 3329536 java.lang.String
>>
>> 14: 132636 3183264 org.apache.cassandra.db.DecoratedKey
>>
>> 15: 97466 3118912 java.util.concurrent.ConcurrentHashMap$HashEntry
>>
>> 16: 97216 3110912 com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap$Node
>>
>>
>>
>>
>>
>