You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Nikolay Kоvshov <nk...@yandex.ru> on 2011/04/22 15:00:53 UTC
OOM on heavy write load
I am using Cassandra 0.7.0 with following settings
binary_memtable_throughput_in_mb: 64
in_memory_compaction_limit_in_mb: 64
keys_cached 1 million
rows_cached 0
RAM for Cassandra 2 GB
I run very simple test
1 Node with 4 HDDs (1 HDD - commitlog and caches, 3 HDDs - data)
1 KS => 1 CF => 1 Column
I insert data (random key 64 bytes + value 64 bytes) at a maximum possible speed, trying to hit disk i/o, calculate speed and make sure Cassandra stays alive. It doesn't, unfortunately.
After several hundreds millions of inserts Cassandra always does down by OOM. Getting it up again doesn't help - after inserting some new data it goes down again. By this time Cassandra goes to swap and has a lot of tasks pending. I am not inserting anything now and tasks sloooowly disappear, but it will take her weeks to do all of them.
compaction type: Minor
column family: Standard1
bytes compacted: 3661003227
bytes total in progress: 4176296448
pending tasks: 630
So, what am I (or Cassandra) doing wrong ? I dont want to get Cassandra crashed without means of repair on heavy load circumstances.
Re: OOM on heavy write load
Posted by Peter Schuller <pe...@infidyne.com>.
> My gut feel: Maybe, if the slowness/timeouts reported by the OP are
> intermixed with periods of activity to indicate compacting full gc.
But even then, after taking a single full GC the behavior should
disappear since there should be no left-overs from the smaller columns
causing fragmentation issues. *Maybe* after two full gc:s tops if the
first happens while there's a mix still active in memtables.
--
/ Peter Schuller
Re: OOM on heavy write load
Posted by Peter Schuller <pe...@infidyne.com>.
> Could this be related as well to
> https://issues.apache.org/jira/browse/CASSANDRA-2463?
My gut feel: Maybe, if the slowness/timeouts reported by the OP are
intermixed with periods of activity to indicate compacting full gc.
OP: Check if cassandra is going into 100% (not less, not more) CPU
usage during periods of timeouts. If the huge allocations fail due to
fragmentation and fallback to Full GC that might be an expected
result. Else -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps.
--
/ Peter Schuller
Re: OOM on heavy write load
Posted by Thibaut Britz <th...@trendiction.com>.
Could this be related as well to
https://issues.apache.org/jira/browse/CASSANDRA-2463?
Thibaut
On Wed, Apr 27, 2011 at 11:35 PM, Aaron Morton <aa...@thelastpickle.com>wrote:
> I'm a bit confused by the two different cases you described, so cannot
> comment specially on your case.
>
> In general if Cassandra is slowing down take a look at the thread pool
> stats, using nodetool tpstats to see where it is backing up and take at look
> at the logs to check for excessive GC. If node stats shows the read or
> mutation stage backing up, check the iostats.
>
> Hope that helps.
> Aaron
>
> On 28/04/2011, at 12:32 AM, Nikolay Kоvshov <nk...@yandex.ru> wrote:
>
> > I have set quite low memory consumption (see my configuration in first
> message) and give Cassandra 2.7 Gb of memory.
> > I cache 1M of 64-bytes keys + 64 Mb memtables. I believe overhead can't
> be 500% or so ?
> >
> > memtable operations in millions = default 0.3
> >
> > I see now very strange behaviour
> >
> > If i fill Cassandra with, say, 100 millions of 64B key + 64B data and
> after that I start inserting 64B key + 64 KB data, compaction queue
> immediately grows to hundreds and cassandra extremely slows down (it makes
> aroung 30-50 operations/sec), then starts to give timeout errors.
> >
> > But if I insert 64B key + 64 KB data from the very beginning, cassandra
> works fine and makes around 300 operations/sec even when the database
> exceeds 2-3 TB
> >
> > The behaviour is quite complex and i cannot predict the effect of my
> actions. What consequences I will have if I turn off compaction ? Can i read
> somewhere about what is compaction and why it so heavily depends on what and
> in which order i write into cassandra ?
> >
> > 26.04.2011, 00:08, "Shu Zhang" <sz...@mediosystems.com>:
> >> the way I measure actual memtable row sizes is this
> >>
> >> write X rows into a cassandra node
> >> trigger GC
> >> record heap usage
> >> trigger compaction and GC
> >> record heap savings and divide by X for actual cassandra memtable row
> size in memory
> >>
> >> Similar process to measure per-key/per-row cache sizes for your data. To
> understand your memtable row overhead size, you can do the above exercise
> with very different data sizes.
> >>
> >> Also, you probably know this, but when setting your memory usage ceiling
> or heap size, make sure to leave a few hundred MBs for GC.
> >> ________________________________________
> >> From: Shu Zhang [szhang@mediosystems.com]
> >> Sent: Monday, April 25, 2011 12:55 PM
> >> To: user@cassandra.apache.org
> >> Subject: RE: OOM on heavy write load
> >>
> >> How large are your rows? binary_memtable_throughput_in_
> >> mb only tracks size of data, but there is an overhead associated with
> each row on the order of magnitude of a few KBs. If your row data sizes are
> really small then the overhead dominates the memory usage and
> binary_memtable_throughput_in_
> >> mb end up not limiting your memory usage the way you'd expect. It's a
> good idea to specify memtable_operations_in_millions in that case. If you're
> not sure how big your data is compared to memtable overhead, it's a good
> idea to specify both parameters to effectively put in a memory ceiling no
> matter which dominates your actual memory usage.
> >>
> >> It could also be that your key cache is too big, you should measure your
> key sizes and make sure you have enough memory to cache 1m keys (along with
> your memtables). Finally if you have multiple keyspaces (for multiple
> applications) on your cluster, they all share the total available heap, so
> you have to account for that.
> >>
> >> There's no measure in cassandra to guard against OOM, you must configure
> nodes such that the max memory usage on each node, that is max size all your
> caches and memtables can potentially grow to, is less than your heap size.
> >> ________________________________________
> >> From: Nikolay Kоvshov [nkovshov@yandex.ru]
> >> Sent: Monday, April 25, 2011 5:21 AM
> >> To: user@cassandra.apache.org
> >> Subject: Re: OOM on heavy write load
> >>
> >> I assume if I turn off swap it will just die earlier, no ? What is the
> mechanism of dying ?
> >>
> >> From the link you provided
> >>
> >> # Row cache is too large, or is caching large rows
> >> my row_cache is 0
> >>
> >> # The memtable sizes are too large for the amount of heap allocated to
> the JVM
> >> Is my memtable size too large ? I have made it less to surely fit the
> "magical formula"
> >>
> >> Trying to analyze heap dumps gives me the following:
> >>
> >> In one case diagram has 3 Memtables about 64 Mb each + 72 Mb "Thread" +
> 700 Mb "Unreachable objects"
> >>
> >> suspected threats:
> >> 7 instances of "org.apache.cassandra.db.Memtable", loaded by
> "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy 456,292,912
> (48.36%) bytes.
> >> 25,211 instances of "org.apache.cassandra.io.sstable.SSTableReader",
> loaded by "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy
> 294,908,984 (31.26%) byte
> >> 72 instances of "java.lang.Thread", loaded by "<system class loader>"
> occupy 143,632,624 (15.22%) bytes.
> >>
> >> In other cases memory analyzer hangs trying to parse 2Gb dump
> >>
> >> 22.04.2011, 17:26, "Jonathan Ellis" <jb...@gmail.com>;;:
> >>
> >>> (0) turn off swap
> >>> (1)
> http://www.datastax.com/docs/0.7/troubleshooting/index#nodes-are-dying-with-oom-errors
> >>>
> >>> On Fri, Apr 22, 2011 at 8:00 AM, Nikolay Kоvshov <nk...@yandex.ru>;;;
> wrote:
> >>>> I am using Cassandra 0.7.0 with following settings
> >>>>
> >>>> binary_memtable_throughput_in_mb: 64
> >>>> in_memory_compaction_limit_in_mb: 64
> >>>> keys_cached 1 million
> >>>> rows_cached 0
> >>>>
> >>>> RAM for Cassandra 2 GB
> >>>>
> >>>> I run very simple test
> >>>>
> >>>> 1 Node with 4 HDDs (1 HDD - commitlog and caches, 3 HDDs - data)
> >>>> 1 KS => 1 CF => 1 Column
> >>>>
> >>>> I insert data (random key 64 bytes + value 64 bytes) at a maximum
> possible speed, trying to hit disk i/o, calculate speed and make sure
> Cassandra stays alive. It doesn't, unfortunately.
> >>>> After several hundreds millions of inserts Cassandra always does
> down by OOM. Getting it up again doesn't help - after inserting some new
> data it goes down again. By this time Cassandra goes to swap and has a lot
> of tasks pending. I am not inserting anything now and tasks sloooowly
> disappear, but it will take her weeks to do all of them.
> >>>>
> >>>> compaction type: Minor
> >>>> column family: Standard1
> >>>> bytes compacted: 3661003227
> >>>> bytes total in progress: 4176296448
> >>>> pending tasks: 630
> >>>>
> >>>> So, what am I (or Cassandra) doing wrong ? I dont want to get
> Cassandra crashed without means of repair on heavy load circumstances.
> >>> --
> >>> Jonathan Ellis
> >>> Project Chair, Apache Cassandra
> >>> co-founder of DataStax, the source for professional Cassandra support
> >>> http://www.datastax.com
>
Re: OOM on heavy write load
Posted by Aaron Morton <aa...@thelastpickle.com>.
I'm a bit confused by the two different cases you described, so cannot comment specially on your case.
In general if Cassandra is slowing down take a look at the thread pool stats, using nodetool tpstats to see where it is backing up and take at look at the logs to check for excessive GC. If node stats shows the read or mutation stage backing up, check the iostats.
Hope that helps.
Aaron
On 28/04/2011, at 12:32 AM, Nikolay Kоvshov <nk...@yandex.ru> wrote:
> I have set quite low memory consumption (see my configuration in first message) and give Cassandra 2.7 Gb of memory.
> I cache 1M of 64-bytes keys + 64 Mb memtables. I believe overhead can't be 500% or so ?
>
> memtable operations in millions = default 0.3
>
> I see now very strange behaviour
>
> If i fill Cassandra with, say, 100 millions of 64B key + 64B data and after that I start inserting 64B key + 64 KB data, compaction queue immediately grows to hundreds and cassandra extremely slows down (it makes aroung 30-50 operations/sec), then starts to give timeout errors.
>
> But if I insert 64B key + 64 KB data from the very beginning, cassandra works fine and makes around 300 operations/sec even when the database exceeds 2-3 TB
>
> The behaviour is quite complex and i cannot predict the effect of my actions. What consequences I will have if I turn off compaction ? Can i read somewhere about what is compaction and why it so heavily depends on what and in which order i write into cassandra ?
>
> 26.04.2011, 00:08, "Shu Zhang" <sz...@mediosystems.com>:
>> the way I measure actual memtable row sizes is this
>>
>> write X rows into a cassandra node
>> trigger GC
>> record heap usage
>> trigger compaction and GC
>> record heap savings and divide by X for actual cassandra memtable row size in memory
>>
>> Similar process to measure per-key/per-row cache sizes for your data. To understand your memtable row overhead size, you can do the above exercise with very different data sizes.
>>
>> Also, you probably know this, but when setting your memory usage ceiling or heap size, make sure to leave a few hundred MBs for GC.
>> ________________________________________
>> From: Shu Zhang [szhang@mediosystems.com]
>> Sent: Monday, April 25, 2011 12:55 PM
>> To: user@cassandra.apache.org
>> Subject: RE: OOM on heavy write load
>>
>> How large are your rows? binary_memtable_throughput_in_
>> mb only tracks size of data, but there is an overhead associated with each row on the order of magnitude of a few KBs. If your row data sizes are really small then the overhead dominates the memory usage and binary_memtable_throughput_in_
>> mb end up not limiting your memory usage the way you'd expect. It's a good idea to specify memtable_operations_in_millions in that case. If you're not sure how big your data is compared to memtable overhead, it's a good idea to specify both parameters to effectively put in a memory ceiling no matter which dominates your actual memory usage.
>>
>> It could also be that your key cache is too big, you should measure your key sizes and make sure you have enough memory to cache 1m keys (along with your memtables). Finally if you have multiple keyspaces (for multiple applications) on your cluster, they all share the total available heap, so you have to account for that.
>>
>> There's no measure in cassandra to guard against OOM, you must configure nodes such that the max memory usage on each node, that is max size all your caches and memtables can potentially grow to, is less than your heap size.
>> ________________________________________
>> From: Nikolay Kоvshov [nkovshov@yandex.ru]
>> Sent: Monday, April 25, 2011 5:21 AM
>> To: user@cassandra.apache.org
>> Subject: Re: OOM on heavy write load
>>
>> I assume if I turn off swap it will just die earlier, no ? What is the mechanism of dying ?
>>
>> From the link you provided
>>
>> # Row cache is too large, or is caching large rows
>> my row_cache is 0
>>
>> # The memtable sizes are too large for the amount of heap allocated to the JVM
>> Is my memtable size too large ? I have made it less to surely fit the "magical formula"
>>
>> Trying to analyze heap dumps gives me the following:
>>
>> In one case diagram has 3 Memtables about 64 Mb each + 72 Mb "Thread" + 700 Mb "Unreachable objects"
>>
>> suspected threats:
>> 7 instances of "org.apache.cassandra.db.Memtable", loaded by "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy 456,292,912 (48.36%) bytes.
>> 25,211 instances of "org.apache.cassandra.io.sstable.SSTableReader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy 294,908,984 (31.26%) byte
>> 72 instances of "java.lang.Thread", loaded by "<system class loader>" occupy 143,632,624 (15.22%) bytes.
>>
>> In other cases memory analyzer hangs trying to parse 2Gb dump
>>
>> 22.04.2011, 17:26, "Jonathan Ellis" <jb...@gmail.com>;;:
>>
>>> (0) turn off swap
>>> (1) http://www.datastax.com/docs/0.7/troubleshooting/index#nodes-are-dying-with-oom-errors
>>>
>>> On Fri, Apr 22, 2011 at 8:00 AM, Nikolay Kоvshov <nk...@yandex.ru>;;; wrote:
>>>> I am using Cassandra 0.7.0 with following settings
>>>>
>>>> binary_memtable_throughput_in_mb: 64
>>>> in_memory_compaction_limit_in_mb: 64
>>>> keys_cached 1 million
>>>> rows_cached 0
>>>>
>>>> RAM for Cassandra 2 GB
>>>>
>>>> I run very simple test
>>>>
>>>> 1 Node with 4 HDDs (1 HDD - commitlog and caches, 3 HDDs - data)
>>>> 1 KS => 1 CF => 1 Column
>>>>
>>>> I insert data (random key 64 bytes + value 64 bytes) at a maximum possible speed, trying to hit disk i/o, calculate speed and make sure Cassandra stays alive. It doesn't, unfortunately.
>>>> After several hundreds millions of inserts Cassandra always does down by OOM. Getting it up again doesn't help - after inserting some new data it goes down again. By this time Cassandra goes to swap and has a lot of tasks pending. I am not inserting anything now and tasks sloooowly disappear, but it will take her weeks to do all of them.
>>>>
>>>> compaction type: Minor
>>>> column family: Standard1
>>>> bytes compacted: 3661003227
>>>> bytes total in progress: 4176296448
>>>> pending tasks: 630
>>>>
>>>> So, what am I (or Cassandra) doing wrong ? I dont want to get Cassandra crashed without means of repair on heavy load circumstances.
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of DataStax, the source for professional Cassandra support
>>> http://www.datastax.com
Re: OOM on heavy write load
Posted by Nikolay Kоvshov <nk...@yandex.ru>.
I have set quite low memory consumption (see my configuration in first message) and give Cassandra 2.7 Gb of memory.
I cache 1M of 64-bytes keys + 64 Mb memtables. I believe overhead can't be 500% or so ?
memtable operations in millions = default 0.3
I see now very strange behaviour
If i fill Cassandra with, say, 100 millions of 64B key + 64B data and after that I start inserting 64B key + 64 KB data, compaction queue immediately grows to hundreds and cassandra extremely slows down (it makes aroung 30-50 operations/sec), then starts to give timeout errors.
But if I insert 64B key + 64 KB data from the very beginning, cassandra works fine and makes around 300 operations/sec even when the database exceeds 2-3 TB
The behaviour is quite complex and i cannot predict the effect of my actions. What consequences I will have if I turn off compaction ? Can i read somewhere about what is compaction and why it so heavily depends on what and in which order i write into cassandra ?
26.04.2011, 00:08, "Shu Zhang" <sz...@mediosystems.com>:
> the way I measure actual memtable row sizes is this
>
> write X rows into a cassandra node
> trigger GC
> record heap usage
> trigger compaction and GC
> record heap savings and divide by X for actual cassandra memtable row size in memory
>
> Similar process to measure per-key/per-row cache sizes for your data. To understand your memtable row overhead size, you can do the above exercise with very different data sizes.
>
> Also, you probably know this, but when setting your memory usage ceiling or heap size, make sure to leave a few hundred MBs for GC.
> ________________________________________
> From: Shu Zhang [szhang@mediosystems.com]
> Sent: Monday, April 25, 2011 12:55 PM
> To: user@cassandra.apache.org
> Subject: RE: OOM on heavy write load
>
> How large are your rows? binary_memtable_throughput_in_
> mb only tracks size of data, but there is an overhead associated with each row on the order of magnitude of a few KBs. If your row data sizes are really small then the overhead dominates the memory usage and binary_memtable_throughput_in_
> mb end up not limiting your memory usage the way you'd expect. It's a good idea to specify memtable_operations_in_millions in that case. If you're not sure how big your data is compared to memtable overhead, it's a good idea to specify both parameters to effectively put in a memory ceiling no matter which dominates your actual memory usage.
>
> It could also be that your key cache is too big, you should measure your key sizes and make sure you have enough memory to cache 1m keys (along with your memtables). Finally if you have multiple keyspaces (for multiple applications) on your cluster, they all share the total available heap, so you have to account for that.
>
> There's no measure in cassandra to guard against OOM, you must configure nodes such that the max memory usage on each node, that is max size all your caches and memtables can potentially grow to, is less than your heap size.
> ________________________________________
> From: Nikolay Kоvshov [nkovshov@yandex.ru]
> Sent: Monday, April 25, 2011 5:21 AM
> To: user@cassandra.apache.org
> Subject: Re: OOM on heavy write load
>
> I assume if I turn off swap it will just die earlier, no ? What is the mechanism of dying ?
>
> From the link you provided
>
> # Row cache is too large, or is caching large rows
> my row_cache is 0
>
> # The memtable sizes are too large for the amount of heap allocated to the JVM
> Is my memtable size too large ? I have made it less to surely fit the "magical formula"
>
> Trying to analyze heap dumps gives me the following:
>
> In one case diagram has 3 Memtables about 64 Mb each + 72 Mb "Thread" + 700 Mb "Unreachable objects"
>
> suspected threats:
> 7 instances of "org.apache.cassandra.db.Memtable", loaded by "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy 456,292,912 (48.36%) bytes.
> 25,211 instances of "org.apache.cassandra.io.sstable.SSTableReader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy 294,908,984 (31.26%) byte
> 72 instances of "java.lang.Thread", loaded by "<system class loader>" occupy 143,632,624 (15.22%) bytes.
>
> In other cases memory analyzer hangs trying to parse 2Gb dump
>
> 22.04.2011, 17:26, "Jonathan Ellis" <jb...@gmail.com>;;:
>
>> (0) turn off swap
>> (1) http://www.datastax.com/docs/0.7/troubleshooting/index#nodes-are-dying-with-oom-errors
>>
>> On Fri, Apr 22, 2011 at 8:00 AM, Nikolay Kоvshov <nk...@yandex.ru>;;; wrote:
>>> I am using Cassandra 0.7.0 with following settings
>>>
>>> binary_memtable_throughput_in_mb: 64
>>> in_memory_compaction_limit_in_mb: 64
>>> keys_cached 1 million
>>> rows_cached 0
>>>
>>> RAM for Cassandra 2 GB
>>>
>>> I run very simple test
>>>
>>> 1 Node with 4 HDDs (1 HDD - commitlog and caches, 3 HDDs - data)
>>> 1 KS => 1 CF => 1 Column
>>>
>>> I insert data (random key 64 bytes + value 64 bytes) at a maximum possible speed, trying to hit disk i/o, calculate speed and make sure Cassandra stays alive. It doesn't, unfortunately.
>>> After several hundreds millions of inserts Cassandra always does down by OOM. Getting it up again doesn't help - after inserting some new data it goes down again. By this time Cassandra goes to swap and has a lot of tasks pending. I am not inserting anything now and tasks sloooowly disappear, but it will take her weeks to do all of them.
>>>
>>> compaction type: Minor
>>> column family: Standard1
>>> bytes compacted: 3661003227
>>> bytes total in progress: 4176296448
>>> pending tasks: 630
>>>
>>> So, what am I (or Cassandra) doing wrong ? I dont want to get Cassandra crashed without means of repair on heavy load circumstances.
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
RE: OOM on heavy write load
Posted by Shu Zhang <sz...@mediosystems.com>.
the way I measure actual memtable row sizes is this
write X rows into a cassandra node
trigger GC
record heap usage
trigger compaction and GC
record heap savings and divide by X for actual cassandra memtable row size in memory
Similar process to measure per-key/per-row cache sizes for your data. To understand your memtable row overhead size, you can do the above exercise with very different data sizes.
Also, you probably know this, but when setting your memory usage ceiling or heap size, make sure to leave a few hundred MBs for GC.
________________________________________
From: Shu Zhang [szhang@mediosystems.com]
Sent: Monday, April 25, 2011 12:55 PM
To: user@cassandra.apache.org
Subject: RE: OOM on heavy write load
How large are your rows? binary_memtable_throughput_in_
mb only tracks size of data, but there is an overhead associated with each row on the order of magnitude of a few KBs. If your row data sizes are really small then the overhead dominates the memory usage and binary_memtable_throughput_in_
mb end up not limiting your memory usage the way you'd expect. It's a good idea to specify memtable_operations_in_millions in that case. If you're not sure how big your data is compared to memtable overhead, it's a good idea to specify both parameters to effectively put in a memory ceiling no matter which dominates your actual memory usage.
It could also be that your key cache is too big, you should measure your key sizes and make sure you have enough memory to cache 1m keys (along with your memtables). Finally if you have multiple keyspaces (for multiple applications) on your cluster, they all share the total available heap, so you have to account for that.
There's no measure in cassandra to guard against OOM, you must configure nodes such that the max memory usage on each node, that is max size all your caches and memtables can potentially grow to, is less than your heap size.
________________________________________
From: Nikolay Kоvshov [nkovshov@yandex.ru]
Sent: Monday, April 25, 2011 5:21 AM
To: user@cassandra.apache.org
Subject: Re: OOM on heavy write load
I assume if I turn off swap it will just die earlier, no ? What is the mechanism of dying ?
>From the link you provided
# Row cache is too large, or is caching large rows
my row_cache is 0
# The memtable sizes are too large for the amount of heap allocated to the JVM
Is my memtable size too large ? I have made it less to surely fit the "magical formula"
Trying to analyze heap dumps gives me the following:
In one case diagram has 3 Memtables about 64 Mb each + 72 Mb "Thread" + 700 Mb "Unreachable objects"
suspected threats:
7 instances of "org.apache.cassandra.db.Memtable", loaded by "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy 456,292,912 (48.36%) bytes.
25,211 instances of "org.apache.cassandra.io.sstable.SSTableReader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy 294,908,984 (31.26%) byte
72 instances of "java.lang.Thread", loaded by "<system class loader>" occupy 143,632,624 (15.22%) bytes.
In other cases memory analyzer hangs trying to parse 2Gb dump
22.04.2011, 17:26, "Jonathan Ellis" <jb...@gmail.com>;:
> (0) turn off swap
> (1) http://www.datastax.com/docs/0.7/troubleshooting/index#nodes-are-dying-with-oom-errors
>
> On Fri, Apr 22, 2011 at 8:00 AM, Nikolay Kоvshov <nk...@yandex.ru>;; wrote:
>> I am using Cassandra 0.7.0 with following settings
>>
>> binary_memtable_throughput_in_mb: 64
>> in_memory_compaction_limit_in_mb: 64
>> keys_cached 1 million
>> rows_cached 0
>>
>> RAM for Cassandra 2 GB
>>
>> I run very simple test
>>
>> 1 Node with 4 HDDs (1 HDD - commitlog and caches, 3 HDDs - data)
>> 1 KS => 1 CF => 1 Column
>>
>> I insert data (random key 64 bytes + value 64 bytes) at a maximum possible speed, trying to hit disk i/o, calculate speed and make sure Cassandra stays alive. It doesn't, unfortunately.
>> After several hundreds millions of inserts Cassandra always does down by OOM. Getting it up again doesn't help - after inserting some new data it goes down again. By this time Cassandra goes to swap and has a lot of tasks pending. I am not inserting anything now and tasks sloooowly disappear, but it will take her weeks to do all of them.
>>
>> compaction type: Minor
>> column family: Standard1
>> bytes compacted: 3661003227
>> bytes total in progress: 4176296448
>> pending tasks: 630
>>
>> So, what am I (or Cassandra) doing wrong ? I dont want to get Cassandra crashed without means of repair on heavy load circumstances.
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
RE: OOM on heavy write load
Posted by Shu Zhang <sz...@mediosystems.com>.
How large are your rows? binary_memtable_throughput_in_
mb only tracks size of data, but there is an overhead associated with each row on the order of magnitude of a few KBs. If your row data sizes are really small then the overhead dominates the memory usage and binary_memtable_throughput_in_
mb end up not limiting your memory usage the way you'd expect. It's a good idea to specify memtable_operations_in_millions in that case. If you're not sure how big your data is compared to memtable overhead, it's a good idea to specify both parameters to effectively put in a memory ceiling no matter which dominates your actual memory usage.
It could also be that your key cache is too big, you should measure your key sizes and make sure you have enough memory to cache 1m keys (along with your memtables). Finally if you have multiple keyspaces (for multiple applications) on your cluster, they all share the total available heap, so you have to account for that.
There's no measure in cassandra to guard against OOM, you must configure nodes such that the max memory usage on each node, that is max size all your caches and memtables can potentially grow to, is less than your heap size.
________________________________________
From: Nikolay Kоvshov [nkovshov@yandex.ru]
Sent: Monday, April 25, 2011 5:21 AM
To: user@cassandra.apache.org
Subject: Re: OOM on heavy write load
I assume if I turn off swap it will just die earlier, no ? What is the mechanism of dying ?
>From the link you provided
# Row cache is too large, or is caching large rows
my row_cache is 0
# The memtable sizes are too large for the amount of heap allocated to the JVM
Is my memtable size too large ? I have made it less to surely fit the "magical formula"
Trying to analyze heap dumps gives me the following:
In one case diagram has 3 Memtables about 64 Mb each + 72 Mb "Thread" + 700 Mb "Unreachable objects"
suspected threats:
7 instances of "org.apache.cassandra.db.Memtable", loaded by "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy 456,292,912 (48.36%) bytes.
25,211 instances of "org.apache.cassandra.io.sstable.SSTableReader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy 294,908,984 (31.26%) byte
72 instances of "java.lang.Thread", loaded by "<system class loader>" occupy 143,632,624 (15.22%) bytes.
In other cases memory analyzer hangs trying to parse 2Gb dump
22.04.2011, 17:26, "Jonathan Ellis" <jb...@gmail.com>;:
> (0) turn off swap
> (1) http://www.datastax.com/docs/0.7/troubleshooting/index#nodes-are-dying-with-oom-errors
>
> On Fri, Apr 22, 2011 at 8:00 AM, Nikolay Kоvshov <nk...@yandex.ru>;; wrote:
>> I am using Cassandra 0.7.0 with following settings
>>
>> binary_memtable_throughput_in_mb: 64
>> in_memory_compaction_limit_in_mb: 64
>> keys_cached 1 million
>> rows_cached 0
>>
>> RAM for Cassandra 2 GB
>>
>> I run very simple test
>>
>> 1 Node with 4 HDDs (1 HDD - commitlog and caches, 3 HDDs - data)
>> 1 KS => 1 CF => 1 Column
>>
>> I insert data (random key 64 bytes + value 64 bytes) at a maximum possible speed, trying to hit disk i/o, calculate speed and make sure Cassandra stays alive. It doesn't, unfortunately.
>> After several hundreds millions of inserts Cassandra always does down by OOM. Getting it up again doesn't help - after inserting some new data it goes down again. By this time Cassandra goes to swap and has a lot of tasks pending. I am not inserting anything now and tasks sloooowly disappear, but it will take her weeks to do all of them.
>>
>> compaction type: Minor
>> column family: Standard1
>> bytes compacted: 3661003227
>> bytes total in progress: 4176296448
>> pending tasks: 630
>>
>> So, what am I (or Cassandra) doing wrong ? I dont want to get Cassandra crashed without means of repair on heavy load circumstances.
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: OOM on heavy write load
Posted by Nikolay Kоvshov <nk...@yandex.ru>.
I assume if I turn off swap it will just die earlier, no ? What is the mechanism of dying ?
>From the link you provided
# Row cache is too large, or is caching large rows
my row_cache is 0
# The memtable sizes are too large for the amount of heap allocated to the JVM
Is my memtable size too large ? I have made it less to surely fit the "magical formula"
Trying to analyze heap dumps gives me the following:
In one case diagram has 3 Memtables about 64 Mb each + 72 Mb "Thread" + 700 Mb "Unreachable objects"
suspected threats:
7 instances of "org.apache.cassandra.db.Memtable", loaded by "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy 456,292,912 (48.36%) bytes.
25,211 instances of "org.apache.cassandra.io.sstable.SSTableReader", loaded by "sun.misc.Launcher$AppClassLoader @ 0x7f29f4992d68" occupy 294,908,984 (31.26%) byte
72 instances of "java.lang.Thread", loaded by "<system class loader>" occupy 143,632,624 (15.22%) bytes.
In other cases memory analyzer hangs trying to parse 2Gb dump
22.04.2011, 17:26, "Jonathan Ellis" <jb...@gmail.com>;:
> (0) turn off swap
> (1) http://www.datastax.com/docs/0.7/troubleshooting/index#nodes-are-dying-with-oom-errors
>
> On Fri, Apr 22, 2011 at 8:00 AM, Nikolay Kоvshov <nk...@yandex.ru>;; wrote:
>> I am using Cassandra 0.7.0 with following settings
>>
>> binary_memtable_throughput_in_mb: 64
>> in_memory_compaction_limit_in_mb: 64
>> keys_cached 1 million
>> rows_cached 0
>>
>> RAM for Cassandra 2 GB
>>
>> I run very simple test
>>
>> 1 Node with 4 HDDs (1 HDD - commitlog and caches, 3 HDDs - data)
>> 1 KS => 1 CF => 1 Column
>>
>> I insert data (random key 64 bytes + value 64 bytes) at a maximum possible speed, trying to hit disk i/o, calculate speed and make sure Cassandra stays alive. It doesn't, unfortunately.
>> After several hundreds millions of inserts Cassandra always does down by OOM. Getting it up again doesn't help - after inserting some new data it goes down again. By this time Cassandra goes to swap and has a lot of tasks pending. I am not inserting anything now and tasks sloooowly disappear, but it will take her weeks to do all of them.
>>
>> compaction type: Minor
>> column family: Standard1
>> bytes compacted: 3661003227
>> bytes total in progress: 4176296448
>> pending tasks: 630
>>
>> So, what am I (or Cassandra) doing wrong ? I dont want to get Cassandra crashed without means of repair on heavy load circumstances.
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: OOM on heavy write load
Posted by Jonathan Ellis <jb...@gmail.com>.
(0) turn off swap
(1) http://www.datastax.com/docs/0.7/troubleshooting/index#nodes-are-dying-with-oom-errors
On Fri, Apr 22, 2011 at 8:00 AM, Nikolay Kоvshov <nk...@yandex.ru> wrote:
> I am using Cassandra 0.7.0 with following settings
>
> binary_memtable_throughput_in_mb: 64
> in_memory_compaction_limit_in_mb: 64
> keys_cached 1 million
> rows_cached 0
>
> RAM for Cassandra 2 GB
>
> I run very simple test
>
> 1 Node with 4 HDDs (1 HDD - commitlog and caches, 3 HDDs - data)
> 1 KS => 1 CF => 1 Column
>
> I insert data (random key 64 bytes + value 64 bytes) at a maximum possible speed, trying to hit disk i/o, calculate speed and make sure Cassandra stays alive. It doesn't, unfortunately.
> After several hundreds millions of inserts Cassandra always does down by OOM. Getting it up again doesn't help - after inserting some new data it goes down again. By this time Cassandra goes to swap and has a lot of tasks pending. I am not inserting anything now and tasks sloooowly disappear, but it will take her weeks to do all of them.
>
> compaction type: Minor
> column family: Standard1
> bytes compacted: 3661003227
> bytes total in progress: 4176296448
> pending tasks: 630
>
> So, what am I (or Cassandra) doing wrong ? I dont want to get Cassandra crashed without means of repair on heavy load circumstances.
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com