You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Mohit Anchlia <mo...@gmail.com> on 2012/01/31 01:58:46 UTC
Re: WARN [Memtable] live ratio
I have the same experience. Wondering what's causing this? One thing I
noticed is that this happens if server is idle for some time and then
load starts going high is when I start to see these messages.
On Mon, Jan 30, 2012 at 4:54 PM, Roshan <co...@gmail.com> wrote:
> Hi All
>
> Time to time I am seen this below warning in Cassandra logs.
> WARN [Memtable] setting live ratio to minimum of 1.0 instead of
> 0.21084217381985554
>
> Not sure what the exact cause for this and the solution to eliminate this.
> Any help is appreciated. Thanks.
>
> --
> View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/WARN-Memtable-live-ratio-tp7238582p7238582.html
> Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: WARN [Memtable] live ratio
Posted by Roshan <co...@gmail.com>.
It happens to me like this.
I have 2 node Cassandra cluster with one column space. No super columns.
Start server freshly (no commit logs, no SSTables, no saves caches,
basically nothing). Then write process starts (not much write load). See
what my log says:
2012-02-06 13:06:13,598 INFO [Gossiper] InetAddress /10.1.161.68 is now UP
2012-02-06 13:06:13,598 INFO [StorageService] Node /10.1.161.68 state jump
to normal
2012-02-06 13:06:13,601 INFO [ColumnFamilyStore] Enqueuing flush of
Memtable-LocationInfo@403316091(35/43 serialized/live bytes, 1 ops)
2012-02-06 13:06:13,601 INFO [Memtable] Writing
Memtable-LocationInfo@403316091(35/43 serialized/live bytes, 1 ops)
2012-02-06 13:06:13,603 INFO [CassandraDaemon] Listening for thrift
clients...
2012-02-06 13:06:13,714 INFO [Memtable] Completed flushing
/data/cassandradb/data/system/LocationInfo-hc-64-Data.db (89 bytes)
2012-02-06 13:06:58,985 INFO [HintedHandOffManager] Started hinted handoff
for token: 113427455640312814857969558651062452224 with IP: /10.1.161.63
2012-02-06 13:06:58,986 INFO [HintedHandOffManager] Finished hinted handoff
of 0 rows to endpoint /10.1.161.63
2012-02-06 13:07:26,324 INFO [HintedHandOffManager] Started hinted handoff
for token: 56713727820156407428984779325531226112 with IP: /10.1.161.68
2012-02-06 13:07:26,325 INFO [HintedHandOffManager] Finished hinted handoff
of 0 rows to endpoint /10.1.161.68
111111111112012-02-06 14:06:13,083 INFO [AutoSavingCache] Saved
WhispirCache-WhispirStandard-KeyCache (186 items) in 28 ms
11112012-02-06 14:24:45,579 WARN [Memtable] setting live ratio to minimum
of 1.0 instead of 0.13675048407845158
2012-02-06 14:24:45,581 INFO [Memtable] CFS(Keyspace='WhispirCache',
ColumnFamily='WhispirStandard') liveRatio is 1.0 (just-counted was 1.0).
calculation took 137ms for 57 columns
2012-02-06 14:24:48,016 WARN [Memtable] setting live ratio to minimum of
1.0 instead of 0.10823538589289985
2012-02-06 14:24:48,016 INFO [Memtable] CFS(Keyspace='WhispirCache',
ColumnFamily='WhispirStandard') liveRatio is 1.0 (just-counted was 1.0).
calculation took 46ms for 119 columns
2012-02-06 14:24:52,279 WARN [Memtable] setting live ratio to minimum of
1.0 instead of 0.11414583341420956
2012-02-06 14:24:52,279 INFO [Memtable] CFS(Keyspace='WhispirCache',
ColumnFamily='WhispirStandard') liveRatio is 1.0 (just-counted was 1.0).
calculation took 78ms for 233 columns
2012-02-06 14:24:58,808 WARN [Memtable] setting live ratio to minimum of
1.0 instead of 0.12002818405606677
2012-02-06 14:24:58,808 INFO [Memtable] CFS(Keyspace='WhispirCache',
ColumnFamily='WhispirStandard') liveRatio is 1.0 (just-counted was 1.0).
calculation took 24ms for 490 columns
2012-02-06 14:25:08,477 WARN [Memtable] setting live ratio to minimum of
1.0 instead of 0.12033609444170106
2012-02-06 14:25:08,477 INFO [Memtable] CFS(Keyspace='WhispirCache',
ColumnFamily='WhispirStandard') liveRatio is 1.0 (just-counted was 1.0).
calculation took 75ms for 1001 columns
12012-02-06 14:28:28,284 WARN [Memtable] setting live ratio to minimum of
1.0 instead of 0.11954297494496129
2012-02-06 14:28:28,284 INFO [Memtable] CFS(Keyspace='WhispirCache',
ColumnFamily='WhispirStandard') liveRatio is 1.0 (just-counted was 1.0).
calculation took 69ms for 1544 columns
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/WARN-Memtable-live-ratio-tp7238582p7264583.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: WARN [Memtable] live ratio
Posted by Mohit Anchlia <mo...@gmail.com>.
On Fri, Feb 3, 2012 at 8:46 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> You should come up with a way to reproduce so we can fix it. :)
This happens when server is idle for a while with not so much data and
then you start a load like 50 write and 50 read requests per sec. We
have 4 CF that we write to and then read from.
>
> On Fri, Feb 3, 2012 at 10:31 AM, Mohit Anchlia <mo...@gmail.com> wrote:
>> On Fri, Feb 3, 2012 at 7:32 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>>> It's a warn because it's nonsense for the JVM to report that an column
>>> + overhead, takes less space than just the column data itself.
>>>
>>
>> But is there any action we need to take, or worry about? We trigger
>> alerts based on WARN and ERROR. But if there is nothing to do then it
>> probably is just an INFO.
>>> On Tue, Jan 31, 2012 at 9:41 PM, Mohit Anchlia <mo...@gmail.com> wrote:
>>>> I guess this is not really a WARN in that case.
>>>>
>>>> On Tue, Jan 31, 2012 at 4:29 PM, aaron morton <aa...@thelastpickle.com> wrote:
>>>>> The ratio is the ratio of serialised bytes for a memtable to actual JVM
>>>>> allocated memory. Using a ratio below 1 would imply the JVM is using less
>>>>> bytes to store the memtable in memory than it takes to store it on disk
>>>>> (without compression).
>>>>>
>>>>> The ceiling for the ratio is 64.
>>>>>
>>>>> The ratio is calculated periodically so if the workload changes, such as
>>>>> system start up, the number will lag behind. I would guess numbers less than
>>>>> 1 mean the memtable does not have any data.
>>>>>
>>>>> Cheers
>>>>>
>>>>> -----------------
>>>>> Aaron Morton
>>>>> Freelance Developer
>>>>> @aaronmorton
>>>>> http://www.thelastpickle.com
>>>>>
>>>>> On 1/02/2012, at 8:27 AM, Radim Kolar wrote:
>>>>>
>>>>>
>>>>> but a ration of< 1 may occur
>>>>>
>>>>> for column families with a very high update to insert ratio.
>>>>>
>>>>> better to ask why minimum ratio is 1.0. What harm can be done with using <
>>>>> 1.0 ratio?
>>>>>
>>>>>
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of DataStax, the source for professional Cassandra support
>>> http://www.datastax.com
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: WARN [Memtable] live ratio
Posted by Radim Kolar <hs...@sendmail.cz>.
Dne 3.2.2012 17:46, Jonathan Ellis napsal(a):
> You should come up with a way to reproduce so we can fix it. :)
it happens after HH delivery when memtable contains lot of deletes
Re: WARN [Memtable] live ratio
Posted by Jonathan Ellis <jb...@gmail.com>.
You should come up with a way to reproduce so we can fix it. :)
On Fri, Feb 3, 2012 at 10:31 AM, Mohit Anchlia <mo...@gmail.com> wrote:
> On Fri, Feb 3, 2012 at 7:32 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>> It's a warn because it's nonsense for the JVM to report that an column
>> + overhead, takes less space than just the column data itself.
>>
>
> But is there any action we need to take, or worry about? We trigger
> alerts based on WARN and ERROR. But if there is nothing to do then it
> probably is just an INFO.
>> On Tue, Jan 31, 2012 at 9:41 PM, Mohit Anchlia <mo...@gmail.com> wrote:
>>> I guess this is not really a WARN in that case.
>>>
>>> On Tue, Jan 31, 2012 at 4:29 PM, aaron morton <aa...@thelastpickle.com> wrote:
>>>> The ratio is the ratio of serialised bytes for a memtable to actual JVM
>>>> allocated memory. Using a ratio below 1 would imply the JVM is using less
>>>> bytes to store the memtable in memory than it takes to store it on disk
>>>> (without compression).
>>>>
>>>> The ceiling for the ratio is 64.
>>>>
>>>> The ratio is calculated periodically so if the workload changes, such as
>>>> system start up, the number will lag behind. I would guess numbers less than
>>>> 1 mean the memtable does not have any data.
>>>>
>>>> Cheers
>>>>
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>>
>>>> On 1/02/2012, at 8:27 AM, Radim Kolar wrote:
>>>>
>>>>
>>>> but a ration of< 1 may occur
>>>>
>>>> for column families with a very high update to insert ratio.
>>>>
>>>> better to ask why minimum ratio is 1.0. What harm can be done with using <
>>>> 1.0 ratio?
>>>>
>>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: WARN [Memtable] live ratio
Posted by Mohit Anchlia <mo...@gmail.com>.
On Fri, Feb 3, 2012 at 7:32 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> It's a warn because it's nonsense for the JVM to report that an column
> + overhead, takes less space than just the column data itself.
>
But is there any action we need to take, or worry about? We trigger
alerts based on WARN and ERROR. But if there is nothing to do then it
probably is just an INFO.
> On Tue, Jan 31, 2012 at 9:41 PM, Mohit Anchlia <mo...@gmail.com> wrote:
>> I guess this is not really a WARN in that case.
>>
>> On Tue, Jan 31, 2012 at 4:29 PM, aaron morton <aa...@thelastpickle.com> wrote:
>>> The ratio is the ratio of serialised bytes for a memtable to actual JVM
>>> allocated memory. Using a ratio below 1 would imply the JVM is using less
>>> bytes to store the memtable in memory than it takes to store it on disk
>>> (without compression).
>>>
>>> The ceiling for the ratio is 64.
>>>
>>> The ratio is calculated periodically so if the workload changes, such as
>>> system start up, the number will lag behind. I would guess numbers less than
>>> 1 mean the memtable does not have any data.
>>>
>>> Cheers
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 1/02/2012, at 8:27 AM, Radim Kolar wrote:
>>>
>>>
>>> but a ration of< 1 may occur
>>>
>>> for column families with a very high update to insert ratio.
>>>
>>> better to ask why minimum ratio is 1.0. What harm can be done with using <
>>> 1.0 ratio?
>>>
>>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
Re: WARN [Memtable] live ratio
Posted by Jonathan Ellis <jb...@gmail.com>.
It's a warn because it's nonsense for the JVM to report that an column
+ overhead, takes less space than just the column data itself.
On Tue, Jan 31, 2012 at 9:41 PM, Mohit Anchlia <mo...@gmail.com> wrote:
> I guess this is not really a WARN in that case.
>
> On Tue, Jan 31, 2012 at 4:29 PM, aaron morton <aa...@thelastpickle.com> wrote:
>> The ratio is the ratio of serialised bytes for a memtable to actual JVM
>> allocated memory. Using a ratio below 1 would imply the JVM is using less
>> bytes to store the memtable in memory than it takes to store it on disk
>> (without compression).
>>
>> The ceiling for the ratio is 64.
>>
>> The ratio is calculated periodically so if the workload changes, such as
>> system start up, the number will lag behind. I would guess numbers less than
>> 1 mean the memtable does not have any data.
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 1/02/2012, at 8:27 AM, Radim Kolar wrote:
>>
>>
>> but a ration of< 1 may occur
>>
>> for column families with a very high update to insert ratio.
>>
>> better to ask why minimum ratio is 1.0. What harm can be done with using <
>> 1.0 ratio?
>>
>>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: WARN [Memtable] live ratio
Posted by Mohit Anchlia <mo...@gmail.com>.
I guess this is not really a WARN in that case.
On Tue, Jan 31, 2012 at 4:29 PM, aaron morton <aa...@thelastpickle.com> wrote:
> The ratio is the ratio of serialised bytes for a memtable to actual JVM
> allocated memory. Using a ratio below 1 would imply the JVM is using less
> bytes to store the memtable in memory than it takes to store it on disk
> (without compression).
>
> The ceiling for the ratio is 64.
>
> The ratio is calculated periodically so if the workload changes, such as
> system start up, the number will lag behind. I would guess numbers less than
> 1 mean the memtable does not have any data.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 1/02/2012, at 8:27 AM, Radim Kolar wrote:
>
>
> but a ration of< 1 may occur
>
> for column families with a very high update to insert ratio.
>
> better to ask why minimum ratio is 1.0. What harm can be done with using <
> 1.0 ratio?
>
>
Re: WARN [Memtable] live ratio
Posted by Roshan <co...@gmail.com>.
Thanks for the explanation.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/WARN-Memtable-live-ratio-tp7238582p7242021.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: WARN [Memtable] live ratio
Posted by aaron morton <aa...@thelastpickle.com>.
The ratio is the ratio of serialised bytes for a memtable to actual JVM allocated memory. Using a ratio below 1 would imply the JVM is using less bytes to store the memtable in memory than it takes to store it on disk (without compression).
The ceiling for the ratio is 64.
The ratio is calculated periodically so if the workload changes, such as system start up, the number will lag behind. I would guess numbers less than 1 mean the memtable does not have any data.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 1/02/2012, at 8:27 AM, Radim Kolar wrote:
>
>> but a ration of< 1 may occur
>> for column families with a very high update to insert ratio.
> better to ask why minimum ratio is 1.0. What harm can be done with using < 1.0 ratio?
Re: WARN [Memtable] live ratio
Posted by Radim Kolar <hs...@sendmail.cz>.
> but a ration of< 1 may occur
> for column families with a very high update to insert ratio.
better to ask why minimum ratio is 1.0. What harm can be done with using
< 1.0 ratio?
RE: WARN [Memtable] live ratio
Posted by Dan Hendry <da...@gmail.com>.
http://thelastpickle.com/2011/05/04/How-are-Memtables-measured/ - Gives some
background information (specific to 0.8 but still valid for 1.0 I believe).
Not quite sure why a warning message is logged but a ration of < 1 may occur
for column families with a very high update to insert ratio.
Dan
-----Original Message-----
From: Roshan [mailto:codevally@gmail.com]
Sent: January-30-12 20:04
To: cassandra-user@incubator.apache.org
Subject: Re: WARN [Memtable] live ratio
Exactly, I am also getting this when server moving idle to high load. May be
Cassandra Experts can help to us.
--
View this message in context:
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/WARN-Memtab
le-live-ratio-tp7238582p7238603.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at
Nabble.com.
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.927 / Virus Database: 271.1.1/4171 - Release Date: 01/30/12
14:34:00
Re: WARN [Memtable] live ratio
Posted by Roshan <co...@gmail.com>.
Exactly, I am also getting this when server moving idle to high load. May be
Cassandra Experts can help to us.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/WARN-Memtable-live-ratio-tp7238582p7238603.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.