You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Radim Kolar <hs...@filez.com> on 2012/05/10 14:07:14 UTC
live ratio counting
liveratio calc should do nothing if memtable has 0 columns. I did manual
flush before this.
WARN [MemoryMeter:1] 2012-05-10 13:21:19,430 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of Infinity
INFO [MemoryMeter:1] 2012-05-10 13:21:19,431 Memtable.java (line 186)
CFS(Keyspace='rapidshare', ColumnFamily='resultcache') liveRatio is 64.0
(just-counted was 64.0). calculation took 1ms for 0 columns
liveratio calculation logic also needs to be changed because it is based
on assumption that workloads do not change. I propose to get
configuration option (per node is enough) for maximum interval between
liveratio calculation and raise maximum liveratio from 64 to at least 80.
Re: live ratio counting
Posted by Radim Kolar <hs...@filez.com>.
Are you experiencing memory pressure you think may be attributed to
memtables not being flushed frequently enough ?
> yes
especially delete workload is really good for OOM cassandra for some reason.
Re: live ratio counting
Posted by Radim Kolar <hs...@filez.com>.
> Try reducing memtable_total_space_in_mb config setting. If the problem
> is incorrect memory metering that should help.
it does not helps much because difference in correct and cassandra
assumed calculation is way too high. It would require me to shrink
memtables to about 10% of their correct size leading to too much
compactions.
>
>> i have 3 workload types running in batch. Delete only workload,
>> insert only and heavy update (lot of overwrites)
> Are you saying you do a lot of deletes, followed by a lot of inserts
> and then updates all for the same CF ?
no. most common workload type is insert only. from time to time there
are batch job doing lot of overwrites in memtables, and ocassionaly
cleanup jobs doing only deletes. This breaks liveratio calculation too
because cassandra assumes not only that average column size stored in
memtable is constant but also that overwrite ratio in memtable is
constant. If you overwrite too much cassandra starts to make very tiny
sstables, if you delete too much there is risk of OOM.
>
>> yes. Record is about 120, but it is rare. 80 should be good enough.
>> Default 10 (if not jusing jamm) is way too low.
> Can you provide some information on what is stored in the CF and what
> sort of workload. It would be interesting to understand why the real
> memory usage is 120 times the serialised size.
super column family:
and column_metadata = [
{column_name : 'crc32',
validation_class : LongType},
{column_name : 'id',
validation_class : LongType},
{column_name : 'name',
validation_class : AsciiType},
{column_name : 'size',
validation_class : LongType}];
]
but this is not important, problem is that you do not calculate live
ration frequently enough, if workload changes ratio looks like:
INFO [MemoryMeter:1] 2012-05-12 21:11:51,649 Memtable.java (line 186)
CFS(Keyspace='dedup', ColumnFamily='resultcache') liveRatio is 64.0
(just-counted was 4.633391051722882). calculation took 111ms for 4465
columns
why not recalculate it every 5 or 10 minutes. calculation takes just few
seconds.
Re: live ratio counting
Posted by Radim Kolar <hs...@filez.com>.
here is part of log. actually record is 419.
ponto:(admin)log/cassandra>grep "to maximum of 64" system.log.1
WARN [MemoryMeter:1] 2012-02-03 00:00:19,444 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of 64.9096047648211
WARN [MemoryMeter:1] 2012-02-08 00:00:17,379 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of 68.81016452376322
WARN [MemoryMeter:1] 2012-02-08 00:00:32,358 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of 88.49747308025415
WARN [MemoryMeter:1] 2012-02-09 00:00:08,448 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of 76.2444888765154
WARN [MemoryMeter:1] 2012-02-10 18:18:52,677 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of 142.22477982642255
WARN [MemoryMeter:1] 2012-02-20 00:00:53,753 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of 88.19832386767173
WARN [MemoryMeter:1] 2012-03-02 10:41:00,232 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of 419.9607495592804
WARN [MemoryMeter:1] 2012-03-07 14:13:15,141 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of Infinity
WARN [MemoryMeter:1] 2012-03-08 00:01:12,766 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of 94.20215772717702
WARN [MemoryMeter:1] 2012-03-09 00:00:38,633 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of 98.54003447121715
WARN [MemoryMeter:1] 2012-03-11 00:00:13,243 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of 193.14262214179965
WARN [MemoryMeter:1] 2012-03-14 00:00:26,709 Memtable.java (line 181)
setting live ratio to maximum of 64 instead of 103.88360138951437
Re: live ratio counting
Posted by aaron morton <aa...@thelastpickle.com>.
>> Are you experiencing memory pressure you think may be attributed to memtables not being flushed frequently enough ?
>
Try reducing memtable_total_space_in_mb config setting. If the problem is incorrect memory metering that should help.
> i have 3 workload types running in batch. Delete only workload, insert only and heavy update (lot of overwrites)
Are you saying you do a lot of deletes, followed by a lot of inserts and then updates all for the same CF ?
> yes. Record is about 120, but it is rare. 80 should be good enough. Default 10 (if not jusing jamm) is way too low.
Can you provide some information on what is stored in the CF and what sort of workload. It would be interesting to understand why the real memory usage is 120 times the serialised size.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 15/05/2012, at 1:50 AM, Radim Kolar wrote:
>
> liveratio calculation logic also needs to be changed because it is based on assumption that workloads do not change.
>> Can you give an example of the sort of workload change you are thinking of ?
> i have 3 workload types running in batch. Delete only workload, insert only and heavy update (lot of overwrites)
>
>>> raise maximum liveratio from 64 to at least 80.
>> Have you seen examples in your logs where the calculated live ratio was between 64 and 80 ?
> yes. Record is about 120, but it is rare. 80 should be good enough. Default 10 (if not jusing jamm) is way too low.
>> Are you experiencing memory pressure you think may be attributed to memtables not being flushed frequently enough ?
> yes
Re: live ratio counting
Posted by Radim Kolar <hs...@filez.com>.
liveratio calculation logic also needs to be changed because it is based
on assumption that workloads do not change.
> Can you give an example of the sort of workload change you are
> thinking of ?
i have 3 workload types running in batch. Delete only workload, insert
only and heavy update (lot of overwrites)
>> raise maximum liveratio from 64 to at least 80.
> Have you seen examples in your logs where the calculated live ratio
> was between 64 and 80 ?
yes. Record is about 120, but it is rare. 80 should be good enough.
Default 10 (if not jusing jamm) is way too low.
> Are you experiencing memory pressure you think may be attributed to
> memtables not being flushed frequently enough ?
yes
Re: live ratio counting
Posted by aaron morton <aa...@thelastpickle.com>.
> liveratio calc should do nothing if memtable has 0 columns.
Sounds reasonable, as counting with zero columns may dramatically change the live ratio and it may take some time to be counted again.
Please create a ticket on https://issues.apache.org/jira/browse/CASSANDRA
> liveratio calculation logic also needs to be changed because it is based on assumption that workloads do not change.
Can you give an example of the sort of workload change you are thinking of ?
> raise maximum liveratio from 64 to at least 80.
Have you seen examples in your logs where the calculated live ratio was between 64 and 80 ?
Are you experiencing memory pressure you think may be attributed to memtables not being flushed frequently enough ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 11/05/2012, at 12:07 AM, Radim Kolar wrote:
> liveratio calc should do nothing if memtable has 0 columns. I did manual flush before this.
>
> WARN [MemoryMeter:1] 2012-05-10 13:21:19,430 Memtable.java (line 181) setting live ratio to maximum of 64 instead of Infinity
> INFO [MemoryMeter:1] 2012-05-10 13:21:19,431 Memtable.java (line 186) CFS(Keyspace='rapidshare', ColumnFamily='resultcache') liveRatio is 64.0 (just-counted was 64.0). calculation took 1ms for 0 columns
>
> liveratio calculation logic also needs to be changed because it is based on assumption that workloads do not change. I propose to get configuration option (per node is enough) for maximum interval between liveratio calculation and raise maximum liveratio from 64 to at least 80.