You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Maciej Miklas <ma...@gmail.com> on 2012/12/03 10:14:11 UTC
Cassandra 1.1.5 - SerializingCacheProvider - possible memory leak?
Hi,
I have following Cassandra setup on server with 24GB RAM:
*cassandra-env.sh*
MAX_HEAP_SIZE="6G"
HEAP_NEWSIZE="500M"
*cassandra.yaml*
key_cache_save_period: 0
row_cache_save_period: 0
key_cache_size_in_mb: 512
row_cache_size_in_mb: 10240
row_cache_provider: SerializingCacheProvider
I'am getting OutOfMemory errors, and VisulalVM shows that Old Gen takes
nearly whole heap.
Those are the Cassandra log messages:
INFO CLibrary JNA mlockall successful
INFO DatabaseDescriptor DiskAccessMode 'auto' determined to be mmap,
indexAccessMode is mmap
INFO DatabaseDescriptor Global memtable threshold is enabled at 1981 MB
INFO CacheService Initializing key cache with capacity of 512 MBs.
INFO CacheService Scheduling key cache save to each 0 seconds (going to
save all keys).
INFO CacheService Initializing row cache with capacity of 10240 MBs and
provider org.apache.cassandra.cache.SerializingCacheProvider
INFO CacheService Scheduling row cache save to each 0 seconds (going to
save all keys).
.....
INFO GCInspector GC for ConcurrentMarkSweep: 1106 ms for 1 collections,
5445489440 used; max is 6232735744
....
INFO StatusLogger Cache Type Size
Capacity KeysToSave
Provider
INFO StatusLogger KeyCache 831 782
831 782 all
INFO StatusLogger RowCache 196404489
196404688 all
org.apache.cassandra.cache.SerializingCacheProvider
.....
INFO StatusLogger ColumnFamily Memtable ops, data
INFO StatusLogger MyCF1 192828,66056113
INFO StatusLogger MyCF2 59913,19535021
INFO StatusLogger MyCF3 124953,59082091
....
WARN [ScheduledTasks:1] GCInspector.java Heap is 0.8623632454134093 full.
You may need to reduce memtable and/or cache sizes. Cassandra will now
flush up to the two largest memtables to free up memory. Adjust
flush_largest_memtables_at threshold in cassandra.yaml if you don't want
Cassandra to do this automatically
1) I've set row cache size to 10GB. Single row needs in serialized form
between 300-500 bytes, this would allow maximum 20 millions row key
entries.
SerializingCacheProvider reports size of 196 millions, how can I
interpret this number?
2) I am using default settings besides changes described above. Since key
cache is small, and off heap cache is active, what is taking space in Old
Gen?
Thanks,
Maciej
Re: Cassandra 1.1.5 - SerializingCacheProvider - possible memory leak?
Posted by aaron morton <aa...@thelastpickle.com>.
> SerializingCacheProvider reports size of 196 millions, how can I interpret this number?
Can you include the output from nodetoo info ?
> 2) I am using default settings besides changes described above. Since key cache is small, and off heap cache is active, what is taking space in Old Gen?
Objects are promoted to the old gen when there is not enough space in the new gen. You have a small new gen, so I would be guessing there is some premature tenuring. Try increasing the new heap to 800M or 1200M depending on the number of cores.
GC activity is a result of the configuration and the work load. Look for very big rows, very long lived rows with a lot of deletes, or very big select / get calls.
Hope that helps.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 4/12/2012, at 2:10 AM, Maciej Miklas <ma...@gmail.com> wrote:
> Size and Capacity are in bytes. The RAM is consumed right after Cassandra start (3GB heap) - the reason for this could be 400.000.000 rows on single node, serialized bloom filters take 1,2 GB HDD space.
>
>
> On Mon, Dec 3, 2012 at 10:14 AM, Maciej Miklas <ma...@gmail.com> wrote:
> Hi,
>
> I have following Cassandra setup on server with 24GB RAM:
>
> cassandra-env.sh
> MAX_HEAP_SIZE="6G"
> HEAP_NEWSIZE="500M"
>
> cassandra.yaml
> key_cache_save_period: 0
> row_cache_save_period: 0
> key_cache_size_in_mb: 512
> row_cache_size_in_mb: 10240
> row_cache_provider: SerializingCacheProvider
>
>
> I'am getting OutOfMemory errors, and VisulalVM shows that Old Gen takes nearly whole heap.
>
> Those are the Cassandra log messages:
>
> INFO CLibrary JNA mlockall successful
> INFO DatabaseDescriptor DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
> INFO DatabaseDescriptor Global memtable threshold is enabled at 1981 MB
> INFO CacheService Initializing key cache with capacity of 512 MBs.
> INFO CacheService Scheduling key cache save to each 0 seconds (going to save all keys).
> INFO CacheService Initializing row cache with capacity of 10240 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
> INFO CacheService Scheduling row cache save to each 0 seconds (going to save all keys).
> .....
> INFO GCInspector GC for ConcurrentMarkSweep: 1106 ms for 1 collections, 5445489440 used; max is 6232735744
> ....
> INFO StatusLogger Cache Type Size Capacity KeysToSave Provider
> INFO StatusLogger KeyCache 831 782 831 782 all
> INFO StatusLogger RowCache 196404489 196404688 all org.apache.cassandra.cache.SerializingCacheProvider
> .....
> INFO StatusLogger ColumnFamily Memtable ops, data
> INFO StatusLogger MyCF1 192828,66056113
> INFO StatusLogger MyCF2 59913,19535021
> INFO StatusLogger MyCF3 124953,59082091
> ....
> WARN [ScheduledTasks:1] GCInspector.java Heap is 0.8623632454134093 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
>
>
>
> 1) I've set row cache size to 10GB. Single row needs in serialized form between 300-500 bytes, this would allow maximum 20 millions row key entries.
> SerializingCacheProvider reports size of 196 millions, how can I interpret this number?
>
> 2) I am using default settings besides changes described above. Since key cache is small, and off heap cache is active, what is taking space in Old Gen?
>
>
> Thanks,
> Maciej
>
>
>
>
>
>
Re: Cassandra 1.1.5 - SerializingCacheProvider - possible memory leak?
Posted by Maciej Miklas <ma...@gmail.com>.
Size and Capacity are in bytes. The RAM is consumed right after Cassandra
start (3GB heap) - the reason for this could be 400.000.000 rows on single
node, serialized bloom filters take 1,2 GB HDD space.
On Mon, Dec 3, 2012 at 10:14 AM, Maciej Miklas <ma...@gmail.com> wrote:
> Hi,
>
> I have following Cassandra setup on server with 24GB RAM:
>
> *cassandra-env.sh*
> MAX_HEAP_SIZE="6G"
> HEAP_NEWSIZE="500M"
>
> *cassandra.yaml*
> key_cache_save_period: 0
> row_cache_save_period: 0
> key_cache_size_in_mb: 512
> row_cache_size_in_mb: 10240
> row_cache_provider: SerializingCacheProvider
>
>
> I'am getting OutOfMemory errors, and VisulalVM shows that Old Gen takes
> nearly whole heap.
>
> Those are the Cassandra log messages:
>
> INFO CLibrary JNA mlockall successful
> INFO DatabaseDescriptor DiskAccessMode 'auto' determined to be mmap,
> indexAccessMode is mmap
> INFO DatabaseDescriptor Global memtable threshold is enabled at 1981 MB
> INFO CacheService Initializing key cache with capacity of 512 MBs.
> INFO CacheService Scheduling key cache save to each 0 seconds (going to
> save all keys).
> INFO CacheService Initializing row cache with capacity of 10240 MBs and
> provider org.apache.cassandra.cache.SerializingCacheProvider
> INFO CacheService Scheduling row cache save to each 0 seconds (going to
> save all keys).
> .....
> INFO GCInspector GC for ConcurrentMarkSweep: 1106 ms for 1 collections,
> 5445489440 used; max is 6232735744
> ....
> INFO StatusLogger Cache Type Size
> Capacity KeysToSave
> Provider
> INFO StatusLogger KeyCache 831 782
> 831 782 all
>
> INFO StatusLogger RowCache 196404489
> 196404688 all
> org.apache.cassandra.cache.SerializingCacheProvider
> .....
> INFO StatusLogger ColumnFamily Memtable ops, data
> INFO StatusLogger MyCF1 192828,66056113
> INFO StatusLogger MyCF2 59913,19535021
> INFO StatusLogger MyCF3 124953,59082091
> ....
> WARN [ScheduledTasks:1] GCInspector.java Heap is 0.8623632454134093 full.
> You may need to reduce memtable and/or cache sizes. Cassandra will now
> flush up to the two largest memtables to free up memory. Adjust
> flush_largest_memtables_at threshold in cassandra.yaml if you don't want
> Cassandra to do this automatically
>
>
>
> 1) I've set row cache size to 10GB. Single row needs in serialized form
> between 300-500 bytes, this would allow maximum 20 millions row key
> entries.
> SerializingCacheProvider reports size of 196 millions, how can I
> interpret this number?
>
> 2) I am using default settings besides changes described above. Since key
> cache is small, and off heap cache is active, what is taking space in Old
> Gen?
>
>
> Thanks,
> Maciej
>
>
>
>
>
>