You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Eran Chinthaka Withana <er...@gmail.com> on 2012/02/14 17:54:50 UTC
Key cache hit rate issue
Hi,
I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80% (using
the numerical values). This is visible in cfstats too. But I'm getting less
than 20% (or sometimes even 0%) key cache hit rate. Well, the data access
pattern is not the issue here as I know they are retrieving the same row
multiple times. I'm using hector client with dynamic load balancing policy
with consistency ONE for both reads and writes. Any ideas on how to find
the issue and fix this?
Here is what I see on cfstats.
Key cache capacity: 16637958
Key cache size: 16637958
Key cache hit rate: 0.045454545454545456
Also, is there a way to warm start the key cache, meaning pre-load the
amount of keys I set as keys_cached?
Thanks,
Eran
Re: Re: Key cache hit rate issue
Posted by zhangcheng <zh...@jike.com>.
Thanks, Jonathan. I got it.
2012-02-17
zhangcheng
发件人： Jonathan Ellis
发送时间： 2012-02-17 10:15:05
收件人： user
抄送：
主题： Re: Key cache hit rate issue
Look for this code in SSTableReader.getPosition:
Pair<Descriptor, DecoratedKey> unifiedKey = new
Pair<Descriptor, DecoratedKey>(descriptor, decoratedKey);
Long cachedPosition = getCachedPosition(unifiedKey, true);
On Thu, Feb 16, 2012 at 4:23 PM, Todd Burruss <bb...@expedia.com> wrote:
> jonathan, you said the key to the cache is key + sstable? looking at the
> code it looks like a DecoratedKey is the "row key". how does sstable come
> into play?
>
> On 2/16/12 1:20 PM, "Jonathan Ellis" <jb...@gmail.com> wrote:
>
>>So, you have roughly 1/6 of your (physical) row keys cached and about
>>1/4 cache hit rate, which doesn't sound unreasonable to me. Remember,
>>each logical key may be spread across multiple physical sstables --
>>each (key, sstable) pair is one entry in the key cache.
>>
>>On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana
>><er...@gmail.com> wrote:
>>> Hi Aaron,
>>>
>>> Here it is.
>>>
>>> Keyspace: XXXX
>>> Read Count: 1123637972
>>> Read Latency: 5.757938114343114 ms.
>>> Write Count: 128201833
>>> Write Latency: 0.0682576607387509 ms.
>>> Pending Tasks: 0
>>> Column Family: YY
>>> SSTable count: 18
>>> Space used (live): 103318720685
>>> Space used (total): 103318720685
>>> Number of Keys (estimate): 92404992
>>> Memtable Columns Count: 1425580
>>> Memtable Data Size: 359655747
>>> Memtable Switch Count: 2522
>>> Read Count: 1123637972
>>> Read Latency: 14.731 ms.
>>> Write Count: 128201833
>>> Write Latency: NaN ms.
>>> Pending Tasks: 0
>>> Bloom Filter False Postives: 1488
>>> Bloom Filter False Ratio: 0.00000
>>> Bloom Filter Space Used: 331522920
>>> Key cache capacity: 16637958
>>> Key cache size: 16637958
>>> Key cache hit rate: 0.2708333333333333
>>> Row cache: disabled
>>> Compacted row minimum size: 51
>>> Compacted row maximum size: 6866
>>> Compacted row mean size: 2560
>>>
>>> Thanks,
>>> Eran Chinthaka Withana
>>>
>>>
>>>
>>> On Thu, Feb 16, 2012 at 12:30 AM, aaron morton <aa...@thelastpickle.com>
>>> wrote:
>>>>
>>>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>>>>8000 is
>>>> bit high. Is there a way to fix/improve it?
>>>>
>>>> Sorry I don't understand what you mean. But if the ratio is 0.0 all is
>>>> good.
>>>>
>>>> Could you include the full output from cfstats for the CF you are
>>>>looking
>>>> at ?
>>>>
>>>> Cheers
>>>>
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>>
>>>> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
>>>>
>>>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>>>>8000 is
>>>> bit high. Is there a way to fix/improve it?
>>>>
>>>> Thanks,
>>>> Eran Chinthaka Withana
>>>>
>>>>
>>>> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aa...@thelastpickle.com>
>>>> wrote:
>>>>>
>>>>> Out of interest what does cfstats say about the bloom filter stats ? A
>>>>> high false positive could lead to a low key cache hit rate.
>>>>>
>>>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>>>> amount of keys I set as keys_cached?
>>>>>
>>>>> See key_cache_save_period when creating the CF.
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> -----------------
>>>>> Aaron Morton
>>>>> Freelance Developer
>>>>> @aaronmorton
>>>>> http://www.thelastpickle.com
>>>>>
>>>>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80%
>>>>> (using the numerical values). This is visible in cfstats too. But I'm
>>>>> getting less than 20% (or sometimes even 0%) key cache hit rate.
>>>>>Well, the
>>>>> data access pattern is not the issue here as I know they are
>>>>>retrieving the
>>>>> same row multiple times. I'm using hector client with dynamic load
>>>>>balancing
>>>>> policy with consistency ONE for both reads and writes. Any ideas on
>>>>>how to
>>>>> find the issue and fix this?
>>>>>
>>>>> Here is what I see on cfstats.
>>>>>
>>>>> Key cache capacity: 16637958
>>>>> Key cache size: 16637958
>>>>> Key cache hit rate: 0.045454545454545456
>>>>>
>>>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>>>> amount of keys I set as keys_cached?
>>>>>
>>>>> Thanks,
>>>>> Eran
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>>--
>>Jonathan Ellis
>>Project Chair, Apache Cassandra
>>co-founder of DataStax, the source for professional Cassandra support
>>http://www.datastax.com
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Key cache hit rate issue
Posted by Todd Burruss <bb...@expedia.com>.
ah, I missed the part about "key" cache .. I read "row" cache. thx
On 2/16/12 6:14 PM, "Jonathan Ellis" <jb...@gmail.com> wrote:
>Look for this code in SSTableReader.getPosition:
>
> Pair<Descriptor, DecoratedKey> unifiedKey = new
>Pair<Descriptor, DecoratedKey>(descriptor, decoratedKey);
> Long cachedPosition = getCachedPosition(unifiedKey, true);
>
>On Thu, Feb 16, 2012 at 4:23 PM, Todd Burruss <bb...@expedia.com>
>wrote:
>> jonathan, you said the key to the cache is key + sstable? looking at
>>the
>> code it looks like a DecoratedKey is the "row key". how does sstable
>>come
>> into play?
>>
>> On 2/16/12 1:20 PM, "Jonathan Ellis" <jb...@gmail.com> wrote:
>>
>>>So, you have roughly 1/6 of your (physical) row keys cached and about
>>>1/4 cache hit rate, which doesn't sound unreasonable to me. Remember,
>>>each logical key may be spread across multiple physical sstables --
>>>each (key, sstable) pair is one entry in the key cache.
>>>
>>>On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana
>>><er...@gmail.com> wrote:
>>>> Hi Aaron,
>>>>
>>>> Here it is.
>>>>
>>>> Keyspace: XXXX
>>>> Read Count: 1123637972
>>>> Read Latency: 5.757938114343114 ms.
>>>> Write Count: 128201833
>>>> Write Latency: 0.0682576607387509 ms.
>>>> Pending Tasks: 0
>>>> Column Family: YY
>>>> SSTable count: 18
>>>> Space used (live): 103318720685
>>>> Space used (total): 103318720685
>>>> Number of Keys (estimate): 92404992
>>>> Memtable Columns Count: 1425580
>>>> Memtable Data Size: 359655747
>>>> Memtable Switch Count: 2522
>>>> Read Count: 1123637972
>>>> Read Latency: 14.731 ms.
>>>> Write Count: 128201833
>>>> Write Latency: NaN ms.
>>>> Pending Tasks: 0
>>>> Bloom Filter False Postives: 1488
>>>> Bloom Filter False Ratio: 0.00000
>>>> Bloom Filter Space Used: 331522920
>>>> Key cache capacity: 16637958
>>>> Key cache size: 16637958
>>>> Key cache hit rate: 0.2708333333333333
>>>> Row cache: disabled
>>>> Compacted row minimum size: 51
>>>> Compacted row maximum size: 6866
>>>> Compacted row mean size: 2560
>>>>
>>>> Thanks,
>>>> Eran Chinthaka Withana
>>>>
>>>>
>>>>
>>>> On Thu, Feb 16, 2012 at 12:30 AM, aaron morton
>>>><aa...@thelastpickle.com>
>>>> wrote:
>>>>>
>>>>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>>>>>8000 is
>>>>> bit high. Is there a way to fix/improve it?
>>>>>
>>>>> Sorry I don't understand what you mean. But if the ratio is 0.0 all
>>>>>is
>>>>> good.
>>>>>
>>>>> Could you include the full output from cfstats for the CF you are
>>>>>looking
>>>>> at ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> -----------------
>>>>> Aaron Morton
>>>>> Freelance Developer
>>>>> @aaronmorton
>>>>> http://www.thelastpickle.com
>>>>>
>>>>> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
>>>>>
>>>>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>>>>>8000 is
>>>>> bit high. Is there a way to fix/improve it?
>>>>>
>>>>> Thanks,
>>>>> Eran Chinthaka Withana
>>>>>
>>>>>
>>>>> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton
>>>>><aa...@thelastpickle.com>
>>>>> wrote:
>>>>>>
>>>>>> Out of interest what does cfstats say about the bloom filter stats
>>>>>>? A
>>>>>> high false positive could lead to a low key cache hit rate.
>>>>>>
>>>>>> Also, is there a way to warm start the key cache, meaning pre-load
>>>>>>the
>>>>>> amount of keys I set as keys_cached?
>>>>>>
>>>>>> See key_cache_save_period when creating the CF.
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>>>> -----------------
>>>>>> Aaron Morton
>>>>>> Freelance Developer
>>>>>> @aaronmorton
>>>>>> http://www.thelastpickle.com
>>>>>>
>>>>>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80%
>>>>>> (using the numerical values). This is visible in cfstats too. But
>>>>>>I'm
>>>>>> getting less than 20% (or sometimes even 0%) key cache hit rate.
>>>>>>Well, the
>>>>>> data access pattern is not the issue here as I know they are
>>>>>>retrieving the
>>>>>> same row multiple times. I'm using hector client with dynamic load
>>>>>>balancing
>>>>>> policy with consistency ONE for both reads and writes. Any ideas on
>>>>>>how to
>>>>>> find the issue and fix this?
>>>>>>
>>>>>> Here is what I see on cfstats.
>>>>>>
>>>>>> Key cache capacity: 16637958
>>>>>> Key cache size: 16637958
>>>>>> Key cache hit rate: 0.045454545454545456
>>>>>>
>>>>>> Also, is there a way to warm start the key cache, meaning pre-load
>>>>>>the
>>>>>> amount of keys I set as keys_cached?
>>>>>>
>>>>>> Thanks,
>>>>>> Eran
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>>--
>>>Jonathan Ellis
>>>Project Chair, Apache Cassandra
>>>co-founder of DataStax, the source for professional Cassandra support
>>>http://www.datastax.com
>>
>
>
>
>--
>Jonathan Ellis
>Project Chair, Apache Cassandra
>co-founder of DataStax, the source for professional Cassandra support
>http://www.datastax.com
Re: Key cache hit rate issue
Posted by Jonathan Ellis <jb...@gmail.com>.
Look for this code in SSTableReader.getPosition:
Pair<Descriptor, DecoratedKey> unifiedKey = new
Pair<Descriptor, DecoratedKey>(descriptor, decoratedKey);
Long cachedPosition = getCachedPosition(unifiedKey, true);
On Thu, Feb 16, 2012 at 4:23 PM, Todd Burruss <bb...@expedia.com> wrote:
> jonathan, you said the key to the cache is key + sstable? looking at the
> code it looks like a DecoratedKey is the "row key". how does sstable come
> into play?
>
> On 2/16/12 1:20 PM, "Jonathan Ellis" <jb...@gmail.com> wrote:
>
>>So, you have roughly 1/6 of your (physical) row keys cached and about
>>1/4 cache hit rate, which doesn't sound unreasonable to me. Remember,
>>each logical key may be spread across multiple physical sstables --
>>each (key, sstable) pair is one entry in the key cache.
>>
>>On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana
>><er...@gmail.com> wrote:
>>> Hi Aaron,
>>>
>>> Here it is.
>>>
>>> Keyspace: XXXX
>>> Read Count: 1123637972
>>> Read Latency: 5.757938114343114 ms.
>>> Write Count: 128201833
>>> Write Latency: 0.0682576607387509 ms.
>>> Pending Tasks: 0
>>> Column Family: YY
>>> SSTable count: 18
>>> Space used (live): 103318720685
>>> Space used (total): 103318720685
>>> Number of Keys (estimate): 92404992
>>> Memtable Columns Count: 1425580
>>> Memtable Data Size: 359655747
>>> Memtable Switch Count: 2522
>>> Read Count: 1123637972
>>> Read Latency: 14.731 ms.
>>> Write Count: 128201833
>>> Write Latency: NaN ms.
>>> Pending Tasks: 0
>>> Bloom Filter False Postives: 1488
>>> Bloom Filter False Ratio: 0.00000
>>> Bloom Filter Space Used: 331522920
>>> Key cache capacity: 16637958
>>> Key cache size: 16637958
>>> Key cache hit rate: 0.2708333333333333
>>> Row cache: disabled
>>> Compacted row minimum size: 51
>>> Compacted row maximum size: 6866
>>> Compacted row mean size: 2560
>>>
>>> Thanks,
>>> Eran Chinthaka Withana
>>>
>>>
>>>
>>> On Thu, Feb 16, 2012 at 12:30 AM, aaron morton <aa...@thelastpickle.com>
>>> wrote:
>>>>
>>>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>>>>8000 is
>>>> bit high. Is there a way to fix/improve it?
>>>>
>>>> Sorry I don't understand what you mean. But if the ratio is 0.0 all is
>>>> good.
>>>>
>>>> Could you include the full output from cfstats for the CF you are
>>>>looking
>>>> at ?
>>>>
>>>> Cheers
>>>>
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>>
>>>> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
>>>>
>>>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>>>>8000 is
>>>> bit high. Is there a way to fix/improve it?
>>>>
>>>> Thanks,
>>>> Eran Chinthaka Withana
>>>>
>>>>
>>>> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aa...@thelastpickle.com>
>>>> wrote:
>>>>>
>>>>> Out of interest what does cfstats say about the bloom filter stats ? A
>>>>> high false positive could lead to a low key cache hit rate.
>>>>>
>>>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>>>> amount of keys I set as keys_cached?
>>>>>
>>>>> See key_cache_save_period when creating the CF.
>>>>>
>>>>> Cheers
>>>>>
>>>>>
>>>>> -----------------
>>>>> Aaron Morton
>>>>> Freelance Developer
>>>>> @aaronmorton
>>>>> http://www.thelastpickle.com
>>>>>
>>>>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80%
>>>>> (using the numerical values). This is visible in cfstats too. But I'm
>>>>> getting less than 20% (or sometimes even 0%) key cache hit rate.
>>>>>Well, the
>>>>> data access pattern is not the issue here as I know they are
>>>>>retrieving the
>>>>> same row multiple times. I'm using hector client with dynamic load
>>>>>balancing
>>>>> policy with consistency ONE for both reads and writes. Any ideas on
>>>>>how to
>>>>> find the issue and fix this?
>>>>>
>>>>> Here is what I see on cfstats.
>>>>>
>>>>> Key cache capacity: 16637958
>>>>> Key cache size: 16637958
>>>>> Key cache hit rate: 0.045454545454545456
>>>>>
>>>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>>>> amount of keys I set as keys_cached?
>>>>>
>>>>> Thanks,
>>>>> Eran
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>>--
>>Jonathan Ellis
>>Project Chair, Apache Cassandra
>>co-founder of DataStax, the source for professional Cassandra support
>>http://www.datastax.com
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Key cache hit rate issue
Posted by Eran Chinthaka Withana <er...@gmail.com>.
Hi Todd,
Thanks for the reply. But I don't think the settings that you mentioned is
not playing any role as those are set to 0.85 and 0.6 in my cassandra.yaml
and the proportion between the number I see as space used and the amount I
set is much less than those numbers.
Thanks,
Eran Chinthaka Withana
On Thu, Feb 16, 2012 at 2:18 PM, Todd Burruss <bb...@expedia.com> wrote:
> there is a setting in the yaml file that helps relieve memory pressure by
> reducing the row cache. it is based on the percent of memory used by the
> JVM
>
> the setting are, reduce_cache_sizes_at and reduce_cache_capacity_to. see
> how much free memory you have and if the numbers suggest that you have hit
> that limit, therefore reducing row cache size
>
>
>
> From: Eran Chinthaka Withana <er...@gmail.com>
> Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Date: Thu, 16 Feb 2012 13:52:38 -0800
> To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
> Subject: Re: Key cache hit rate issue
>
> Hi Jonathan,
>
> Thanks for the reply. Yes there is a possibility that the keys can be
> distributed in multiple SSTables, but my data access patterns are such that
> I always read/write the whole row. So I expect all the data to be in the
> same SSTable (please correct me if I'm wrong).
>
> For some reason 16637958 (the keys cached) has become a golden number and
> I don't see key cache increasing beyond that. I also checked memory and I
> have about 4GB left in JVM memory and didn't see any issues on logs.
>
> Thanks,
> Eran Chinthaka Withana
>
>
> On Thu, Feb 16, 2012 at 1:20 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>
>> So, you have roughly 1/6 of your (physical) row keys cached and about
>> 1/4 cache hit rate, which doesn't sound unreasonable to me. Remember,
>> each logical key may be spread across multiple physical sstables --
>> each (key, sstable) pair is one entry in the key cache.
>>
>> On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana
>> <er...@gmail.com> wrote:
>> > Hi Aaron,
>> >
>> > Here it is.
>> >
>> > Keyspace: XXXX
>> > Read Count: 1123637972
>> > Read Latency: 5.757938114343114 ms.
>> > Write Count: 128201833
>> > Write Latency: 0.0682576607387509 ms.
>> > Pending Tasks: 0
>> > Column Family: YY
>> > SSTable count: 18
>> > Space used (live): 103318720685
>> > Space used (total): 103318720685
>> > Number of Keys (estimate): 92404992
>> > Memtable Columns Count: 1425580
>> > Memtable Data Size: 359655747
>> > Memtable Switch Count: 2522
>> > Read Count: 1123637972
>> > Read Latency: 14.731 ms.
>> > Write Count: 128201833
>> > Write Latency: NaN ms.
>> > Pending Tasks: 0
>> > Bloom Filter False Postives: 1488
>> > Bloom Filter False Ratio: 0.00000
>> > Bloom Filter Space Used: 331522920
>> > Key cache capacity: 16637958
>> > Key cache size: 16637958
>> > Key cache hit rate: 0.2708333333333333
>> > Row cache: disabled
>> > Compacted row minimum size: 51
>> > Compacted row maximum size: 6866
>> > Compacted row mean size: 2560
>> >
>> > Thanks,
>> > Eran Chinthaka Withana
>> >
>> >
>> >
>> > On Thu, Feb 16, 2012 at 12:30 AM, aaron morton <aaron@thelastpickle.com
>> >
>> > wrote:
>> >>
>> >> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>> 8000 is
>> >> bit high. Is there a way to fix/improve it?
>> >>
>> >> Sorry I don't understand what you mean. But if the ratio is 0.0 all is
>> >> good.
>> >>
>> >> Could you include the full output from cfstats for the CF you are
>> looking
>> >> at ?
>> >>
>> >> Cheers
>> >>
>> >> -----------------
>> >> Aaron Morton
>> >> Freelance Developer
>> >> @aaronmorton
>> >> http://www.thelastpickle.com
>> >>
>> >> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
>> >>
>> >> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>> 8000 is
>> >> bit high. Is there a way to fix/improve it?
>> >>
>> >> Thanks,
>> >> Eran Chinthaka Withana
>> >>
>> >>
>> >> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aaron@thelastpickle.com
>> >
>> >> wrote:
>> >>>
>> >>> Out of interest what does cfstats say about the bloom filter stats ? A
>> >>> high false positive could lead to a low key cache hit rate.
>> >>>
>> >>> Also, is there a way to warm start the key cache, meaning pre-load the
>> >>> amount of keys I set as keys_cached?
>> >>>
>> >>> See key_cache_save_period when creating the CF.
>> >>>
>> >>> Cheers
>> >>>
>> >>>
>> >>> -----------------
>> >>> Aaron Morton
>> >>> Freelance Developer
>> >>> @aaronmorton
>> >>> http://www.thelastpickle.com
>> >>>
>> >>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80%
>> >>> (using the numerical values). This is visible in cfstats too. But I'm
>> >>> getting less than 20% (or sometimes even 0%) key cache hit rate.
>> Well, the
>> >>> data access pattern is not the issue here as I know they are
>> retrieving the
>> >>> same row multiple times. I'm using hector client with dynamic load
>> balancing
>> >>> policy with consistency ONE for both reads and writes. Any ideas on
>> how to
>> >>> find the issue and fix this?
>> >>>
>> >>> Here is what I see on cfstats.
>> >>>
>> >>> Key cache capacity: 16637958
>> >>> Key cache size: 16637958
>> >>> Key cache hit rate: 0.045454545454545456
>> >>>
>> >>> Also, is there a way to warm start the key cache, meaning pre-load the
>> >>> amount of keys I set as keys_cached?
>> >>>
>> >>> Thanks,
>> >>> Eran
>> >>>
>> >>>
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>>
>
>
Re: Key cache hit rate issue
Posted by Franc Carter <fr...@sirca.org.au>.
On 17/02/2012 8:53 AM, "Eran Chinthaka Withana" <er...@gmail.com>
wrote:
>
> Hi Jonathan,
>
> Thanks for the reply. Yes there is a possibility that the keys can be
distributed in multiple SSTables, but my data access patterns are such that
I always read/write the whole row. So I expect all the data to be in the
same SSTable (please correct me if I'm wrong).
>
> For some reason 16637958 (the keys cached) has become a golden number and
I don't see key cache increasing beyond that. I also checked memory and I
have about 4GB left in JVM memory and didn't see any issues on logs.
I have seen the same thing with the keycache size becoming static
cheers
>
> Thanks,
> Eran Chinthaka Withana
>
>
>
> On Thu, Feb 16, 2012 at 1:20 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> So, you have roughly 1/6 of your (physical) row keys cached and about
>> 1/4 cache hit rate, which doesn't sound unreasonable to me. Remember,
>> each logical key may be spread across multiple physical sstables --
>> each (key, sstable) pair is one entry in the key cache.
>>
>> On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana
>> <er...@gmail.com> wrote:
>> > Hi Aaron,
>> >
>> > Here it is.
>> >
>> > Keyspace: XXXX
>> > Read Count: 1123637972
>> > Read Latency: 5.757938114343114 ms.
>> > Write Count: 128201833
>> > Write Latency: 0.0682576607387509 ms.
>> > Pending Tasks: 0
>> > Column Family: YY
>> > SSTable count: 18
>> > Space used (live): 103318720685
>> > Space used (total): 103318720685
>> > Number of Keys (estimate): 92404992
>> > Memtable Columns Count: 1425580
>> > Memtable Data Size: 359655747
>> > Memtable Switch Count: 2522
>> > Read Count: 1123637972
>> > Read Latency: 14.731 ms.
>> > Write Count: 128201833
>> > Write Latency: NaN ms.
>> > Pending Tasks: 0
>> > Bloom Filter False Postives: 1488
>> > Bloom Filter False Ratio: 0.00000
>> > Bloom Filter Space Used: 331522920
>> > Key cache capacity: 16637958
>> > Key cache size: 16637958
>> > Key cache hit rate: 0.2708333333333333
>> > Row cache: disabled
>> > Compacted row minimum size: 51
>> > Compacted row maximum size: 6866
>> > Compacted row mean size: 2560
>> >
>> > Thanks,
>> > Eran Chinthaka Withana
>> >
>> >
>> >
>> > On Thu, Feb 16, 2012 at 12:30 AM, aaron morton <aaron@thelastpickle.com
>
>> > wrote:
>> >>
>> >> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
8000 is
>> >> bit high. Is there a way to fix/improve it?
>> >>
>> >> Sorry I don't understand what you mean. But if the ratio is 0.0 all is
>> >> good.
>> >>
>> >> Could you include the full output from cfstats for the CF you are
looking
>> >> at ?
>> >>
>> >> Cheers
>> >>
>> >> -----------------
>> >> Aaron Morton
>> >> Freelance Developer
>> >> @aaronmorton
>> >> http://www.thelastpickle.com
>> >>
>> >> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
>> >>
>> >> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
8000 is
>> >> bit high. Is there a way to fix/improve it?
>> >>
>> >> Thanks,
>> >> Eran Chinthaka Withana
>> >>
>> >>
>> >> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aaron@thelastpickle.com
>
>> >> wrote:
>> >>>
>> >>> Out of interest what does cfstats say about the bloom filter stats ?
A
>> >>> high false positive could lead to a low key cache hit rate.
>> >>>
>> >>> Also, is there a way to warm start the key cache, meaning pre-load
the
>> >>> amount of keys I set as keys_cached?
>> >>>
>> >>> See key_cache_save_period when creating the CF.
>> >>>
>> >>> Cheers
>> >>>
>> >>>
>> >>> -----------------
>> >>> Aaron Morton
>> >>> Freelance Developer
>> >>> @aaronmorton
>> >>> http://www.thelastpickle.com
>> >>>
>> >>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80%
>> >>> (using the numerical values). This is visible in cfstats too. But I'm
>> >>> getting less than 20% (or sometimes even 0%) key cache hit rate.
Well, the
>> >>> data access pattern is not the issue here as I know they are
retrieving the
>> >>> same row multiple times. I'm using hector client with dynamic load
balancing
>> >>> policy with consistency ONE for both reads and writes. Any ideas on
how to
>> >>> find the issue and fix this?
>> >>>
>> >>> Here is what I see on cfstats.
>> >>>
>> >>> Key cache capacity: 16637958
>> >>> Key cache size: 16637958
>> >>> Key cache hit rate: 0.045454545454545456
>> >>>
>> >>> Also, is there a way to warm start the key cache, meaning pre-load
the
>> >>> amount of keys I set as keys_cached?
>> >>>
>> >>> Thanks,
>> >>> Eran
>> >>>
>> >>>
>> >>
>> >>
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
Re: Re: Key cache hit rate issue
Posted by zhangcheng <zh...@jike.com>.
Yes, Cassandra has no way of knowing that all the data is in the most recent sstable, and will have to check the others too, and this bring a lot of difficulty to data compaction.
I have a question that if I want a high performance data compaction, how can I implement that all the columns are all in the most recent sstable, by this, I think the read performace will be better.
2012-02-17
zhangcheng
发件人： Jonathan Ellis
发送时间： 2012-02-17 10:13:20
收件人： user
抄送：
主题： Re: Key cache hit rate issue
On Thu, Feb 16, 2012 at 3:52 PM, Eran Chinthaka Withana
<er...@gmail.com> wrote:
> Thanks for the reply. Yes there is a possibility that the keys can be
> distributed in multiple SSTables, but my data access patterns are such that
> I always read/write the whole row. So I expect all the data to be in the
> same SSTable (please correct me if I'm wrong).
If you're doing slice reads, Cassandra has no way of knowing that all
the data is in the most recent sstable, and will have to check the
others too.
> For some reason 16637958 (the keys cached) has become a golden number and I
> don't see key cache increasing beyond that.
16637958 is your configured cache capacity according to the cfstats you pasted.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Key cache hit rate issue
Posted by Eran Chinthaka Withana <er...@gmail.com>.
True, the high hit rate has translated to low read latency. But the
question is how can I debug the reason for low hit rate now assuming read
patterns haven't changed.
Thanks,
Eran Chinthaka Withana
On Fri, Feb 17, 2012 at 3:07 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> I suspect the main difference is that 2-3 weeks ago almost none of
> your reads had to hit disk.
>
> On Fri, Feb 17, 2012 at 1:53 PM, Eran Chinthaka Withana
> <er...@gmail.com> wrote:
> > I never used JMX for any changes and use JMX only for monitoring. All my
> > updates goes through schema updates.
> >
> > To give you little bit more context (not sure whether this will help but
> > anyway), about 2-3 weeks back the read latency was 4-8ms with about
> 90-95%
> > key cache hit rate. But after that point, I stopped all the reads and
> kept
> > only writes into the system. When I enabled reads this week, then only I
> saw
> > this read latency degradation. I didn't run nodetool repair for sometime
> (gc
> > grace is 10 days), since I stopped readers. Can this be connected to the
> > read degradation? I started running repairs last night but didn't see any
> > read latency improvements, yet.
> >
> > Thanks,
> > Eran Chinthaka Withana
> >
> >
> >
> > On Fri, Feb 17, 2012 at 11:30 AM, Jonathan Ellis <jb...@gmail.com>
> wrote:
> >>
> >> Only thing I can think of is that if you've set the cache size
> >> manually over JMX it will preserve that size if you change it via a
> >> schema update.
> >>
> >> On Fri, Feb 17, 2012 at 12:10 AM, Eran Chinthaka Withana
> >> <er...@gmail.com> wrote:
> >> > Hi Jonathan,
> >> >>
> >> >>
> >> >> > For some reason 16637958 (the keys cached) has become a golden
> number
> >> >> > and I
> >> >> > don't see key cache increasing beyond that.
> >> >>
> >> >> 16637958 is your configured cache capacity according to the cfstats
> you
> >> >> pasted.
> >> >
> >> >
> >> > this is another weird part. If you look at the schema[1] (pasted here,
> >> > extracted through cli), the keys_cached is set to 40million. And
> cfstats
> >> > shows that cassandra is only using 16 million.
> >> >
> >> > Please note that I enabled row cache since I posted the cfstats. And
> >> > pasting
> >> > the latest cfstats[2] herewith.
> >> >
> >> > [1]
> >> > create column family YY
> >> > with column_type = 'Standard'
> >> > and comparator = 'UTF8Type'
> >> > and default_validation_class = 'UTF8Type'
> >> > and key_validation_class = 'LongType'
> >> > and rows_cached = 10000.0
> >> > and row_cache_save_period = 0
> >> > and row_cache_keys_to_save = 2147483647
> >> > and keys_cached = 4.0E7
> >> > and key_cache_save_period = 14400
> >> > and read_repair_chance = 0.1
> >> > and gc_grace = 864000
> >> > and min_compaction_threshold = 4
> >> > and max_compaction_threshold = 32
> >> > and replicate_on_write = true
> >> > and row_cache_provider = 'SerializingCacheProvider'
> >> > and compaction_strategy =
> >> > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> >> > and compression_options = {'chunk_length_kb' : '64',
> >> > 'sstable_compression'
> >> > : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> >> >
> >> > [2]
> >> > Keyspace: XXXXX
> >> > Read Count: 590667
> >> > Read Latency: 11.644107261790484 ms.
> >> > Write Count: 12720068
> >> > Write Latency: 0.06049896832312532 ms.
> >> > Pending Tasks: 0
> >> > Column Family: YY
> >> > SSTable count: 12
> >> > Space used (live): 66300297338
> >> > Space used (total): 66300297338
> >> > Number of Keys (estimate): 61152896
> >> > Memtable Columns Count: 881972
> >> > Memtable Data Size: 211749842
> >> > Memtable Switch Count: 267
> >> > Read Count: 590667
> >> > Read Latency: 16.859 ms.
> >> > Write Count: 12720068
> >> > Write Latency: 0.060 ms.
> >> > Pending Tasks: 0
> >> > Bloom Filter False Postives: 507
> >> > Bloom Filter False Ratio: 0.00099
> >> > Bloom Filter Space Used: 168118968
> >> > Key cache capacity: 40000000
> >> > Key cache size: 3838755
> >> > Key cache hit rate: 0.2072072072072072
> >> > Row cache capacity: 10000
> >> > Row cache size: 10000
> >> > Row cache hit rate: 0.022222222222222223
> >> > Compacted row minimum size: 51
> >> > Compacted row maximum size: 6866
> >> > Compacted row mean size: 2577
> >> >
> >> > Thanks,
> >> > Eran
> >>
> >>
> >>
> >> --
> >> Jonathan Ellis
> >> Project Chair, Apache Cassandra
> >> co-founder of DataStax, the source for professional Cassandra support
> >> http://www.datastax.com
> >
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: Key cache hit rate issue
Posted by Jonathan Ellis <jb...@gmail.com>.
I suspect the main difference is that 2-3 weeks ago almost none of
your reads had to hit disk.
On Fri, Feb 17, 2012 at 1:53 PM, Eran Chinthaka Withana
<er...@gmail.com> wrote:
> I never used JMX for any changes and use JMX only for monitoring. All my
> updates goes through schema updates.
>
> To give you little bit more context (not sure whether this will help but
> anyway), about 2-3 weeks back the read latency was 4-8ms with about 90-95%
> key cache hit rate. But after that point, I stopped all the reads and kept
> only writes into the system. When I enabled reads this week, then only I saw
> this read latency degradation. I didn't run nodetool repair for sometime (gc
> grace is 10 days), since I stopped readers. Can this be connected to the
> read degradation? I started running repairs last night but didn't see any
> read latency improvements, yet.
>
> Thanks,
> Eran Chinthaka Withana
>
>
>
> On Fri, Feb 17, 2012 at 11:30 AM, Jonathan Ellis <jb...@gmail.com> wrote:
>>
>> Only thing I can think of is that if you've set the cache size
>> manually over JMX it will preserve that size if you change it via a
>> schema update.
>>
>> On Fri, Feb 17, 2012 at 12:10 AM, Eran Chinthaka Withana
>> <er...@gmail.com> wrote:
>> > Hi Jonathan,
>> >>
>> >>
>> >> > For some reason 16637958 (the keys cached) has become a golden number
>> >> > and I
>> >> > don't see key cache increasing beyond that.
>> >>
>> >> 16637958 is your configured cache capacity according to the cfstats you
>> >> pasted.
>> >
>> >
>> > this is another weird part. If you look at the schema[1] (pasted here,
>> > extracted through cli), the keys_cached is set to 40million. And cfstats
>> > shows that cassandra is only using 16 million.
>> >
>> > Please note that I enabled row cache since I posted the cfstats. And
>> > pasting
>> > the latest cfstats[2] herewith.
>> >
>> > [1]
>> > create column family YY
>> > with column_type = 'Standard'
>> > and comparator = 'UTF8Type'
>> > and default_validation_class = 'UTF8Type'
>> > and key_validation_class = 'LongType'
>> > and rows_cached = 10000.0
>> > and row_cache_save_period = 0
>> > and row_cache_keys_to_save = 2147483647
>> > and keys_cached = 4.0E7
>> > and key_cache_save_period = 14400
>> > and read_repair_chance = 0.1
>> > and gc_grace = 864000
>> > and min_compaction_threshold = 4
>> > and max_compaction_threshold = 32
>> > and replicate_on_write = true
>> > and row_cache_provider = 'SerializingCacheProvider'
>> > and compaction_strategy =
>> > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
>> > and compression_options = {'chunk_length_kb' : '64',
>> > 'sstable_compression'
>> > : 'org.apache.cassandra.io.compress.SnappyCompressor'};
>> >
>> > [2]
>> > Keyspace: XXXXX
>> > Read Count: 590667
>> > Read Latency: 11.644107261790484 ms.
>> > Write Count: 12720068
>> > Write Latency: 0.06049896832312532 ms.
>> > Pending Tasks: 0
>> > Column Family: YY
>> > SSTable count: 12
>> > Space used (live): 66300297338
>> > Space used (total): 66300297338
>> > Number of Keys (estimate): 61152896
>> > Memtable Columns Count: 881972
>> > Memtable Data Size: 211749842
>> > Memtable Switch Count: 267
>> > Read Count: 590667
>> > Read Latency: 16.859 ms.
>> > Write Count: 12720068
>> > Write Latency: 0.060 ms.
>> > Pending Tasks: 0
>> > Bloom Filter False Postives: 507
>> > Bloom Filter False Ratio: 0.00099
>> > Bloom Filter Space Used: 168118968
>> > Key cache capacity: 40000000
>> > Key cache size: 3838755
>> > Key cache hit rate: 0.2072072072072072
>> > Row cache capacity: 10000
>> > Row cache size: 10000
>> > Row cache hit rate: 0.022222222222222223
>> > Compacted row minimum size: 51
>> > Compacted row maximum size: 6866
>> > Compacted row mean size: 2577
>> >
>> > Thanks,
>> > Eran
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of DataStax, the source for professional Cassandra support
>> http://www.datastax.com
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Key cache hit rate issue
Posted by Eran Chinthaka Withana <er...@gmail.com>.
I never used JMX for any changes and use JMX only for monitoring. All my
updates goes through schema updates.
To give you little bit more context (not sure whether this will help but
anyway), about 2-3 weeks back the read latency was 4-8ms with about 90-95%
key cache hit rate. But after that point, I stopped all the reads and kept
only writes into the system. When I enabled reads this week, then only I
saw this read latency degradation. I didn't run nodetool repair for
sometime (gc grace is 10 days), since I stopped readers. Can this be
connected to the read degradation? I started running repairs last night but
didn't see any read latency improvements, yet.
Thanks,
Eran Chinthaka Withana
On Fri, Feb 17, 2012 at 11:30 AM, Jonathan Ellis <jb...@gmail.com> wrote:
> Only thing I can think of is that if you've set the cache size
> manually over JMX it will preserve that size if you change it via a
> schema update.
>
> On Fri, Feb 17, 2012 at 12:10 AM, Eran Chinthaka Withana
> <er...@gmail.com> wrote:
> > Hi Jonathan,
> >>
> >>
> >> > For some reason 16637958 (the keys cached) has become a golden number
> >> > and I
> >> > don't see key cache increasing beyond that.
> >>
> >> 16637958 is your configured cache capacity according to the cfstats you
> >> pasted.
> >
> >
> > this is another weird part. If you look at the schema[1] (pasted here,
> > extracted through cli), the keys_cached is set to 40million. And cfstats
> > shows that cassandra is only using 16 million.
> >
> > Please note that I enabled row cache since I posted the cfstats. And
> pasting
> > the latest cfstats[2] herewith.
> >
> > [1]
> > create column family YY
> > with column_type = 'Standard'
> > and comparator = 'UTF8Type'
> > and default_validation_class = 'UTF8Type'
> > and key_validation_class = 'LongType'
> > and rows_cached = 10000.0
> > and row_cache_save_period = 0
> > and row_cache_keys_to_save = 2147483647
> > and keys_cached = 4.0E7
> > and key_cache_save_period = 14400
> > and read_repair_chance = 0.1
> > and gc_grace = 864000
> > and min_compaction_threshold = 4
> > and max_compaction_threshold = 32
> > and replicate_on_write = true
> > and row_cache_provider = 'SerializingCacheProvider'
> > and compaction_strategy =
> > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> > and compression_options = {'chunk_length_kb' : '64',
> 'sstable_compression'
> > : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> >
> > [2]
> > Keyspace: XXXXX
> > Read Count: 590667
> > Read Latency: 11.644107261790484 ms.
> > Write Count: 12720068
> > Write Latency: 0.06049896832312532 ms.
> > Pending Tasks: 0
> > Column Family: YY
> > SSTable count: 12
> > Space used (live): 66300297338
> > Space used (total): 66300297338
> > Number of Keys (estimate): 61152896
> > Memtable Columns Count: 881972
> > Memtable Data Size: 211749842
> > Memtable Switch Count: 267
> > Read Count: 590667
> > Read Latency: 16.859 ms.
> > Write Count: 12720068
> > Write Latency: 0.060 ms.
> > Pending Tasks: 0
> > Bloom Filter False Postives: 507
> > Bloom Filter False Ratio: 0.00099
> > Bloom Filter Space Used: 168118968
> > Key cache capacity: 40000000
> > Key cache size: 3838755
> > Key cache hit rate: 0.2072072072072072
> > Row cache capacity: 10000
> > Row cache size: 10000
> > Row cache hit rate: 0.022222222222222223
> > Compacted row minimum size: 51
> > Compacted row maximum size: 6866
> > Compacted row mean size: 2577
> >
> > Thanks,
> > Eran
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: Key cache hit rate issue
Posted by Jonathan Ellis <jb...@gmail.com>.
Only thing I can think of is that if you've set the cache size
manually over JMX it will preserve that size if you change it via a
schema update.
On Fri, Feb 17, 2012 at 12:10 AM, Eran Chinthaka Withana
<er...@gmail.com> wrote:
> Hi Jonathan,
>>
>>
>> > For some reason 16637958 (the keys cached) has become a golden number
>> > and I
>> > don't see key cache increasing beyond that.
>>
>> 16637958 is your configured cache capacity according to the cfstats you
>> pasted.
>
>
> this is another weird part. If you look at the schema[1] (pasted here,
> extracted through cli), the keys_cached is set to 40million. And cfstats
> shows that cassandra is only using 16 million.
>
> Please note that I enabled row cache since I posted the cfstats. And pasting
> the latest cfstats[2] herewith.
>
> [1]
> create column family YY
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'UTF8Type'
> and key_validation_class = 'LongType'
> and rows_cached = 10000.0
> and row_cache_save_period = 0
> and row_cache_keys_to_save = 2147483647
> and keys_cached = 4.0E7
> and key_cache_save_period = 14400
> and read_repair_chance = 0.1
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and row_cache_provider = 'SerializingCacheProvider'
> and compaction_strategy =
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and compression_options = {'chunk_length_kb' : '64', 'sstable_compression'
> : 'org.apache.cassandra.io.compress.SnappyCompressor'};
>
> [2]
> Keyspace: XXXXX
> Read Count: 590667
> Read Latency: 11.644107261790484 ms.
> Write Count: 12720068
> Write Latency: 0.06049896832312532 ms.
> Pending Tasks: 0
> Column Family: YY
> SSTable count: 12
> Space used (live): 66300297338
> Space used (total): 66300297338
> Number of Keys (estimate): 61152896
> Memtable Columns Count: 881972
> Memtable Data Size: 211749842
> Memtable Switch Count: 267
> Read Count: 590667
> Read Latency: 16.859 ms.
> Write Count: 12720068
> Write Latency: 0.060 ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 507
> Bloom Filter False Ratio: 0.00099
> Bloom Filter Space Used: 168118968
> Key cache capacity: 40000000
> Key cache size: 3838755
> Key cache hit rate: 0.2072072072072072
> Row cache capacity: 10000
> Row cache size: 10000
> Row cache hit rate: 0.022222222222222223
> Compacted row minimum size: 51
> Compacted row maximum size: 6866
> Compacted row mean size: 2577
>
> Thanks,
> Eran
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Key cache hit rate issue
Posted by Eran Chinthaka Withana <er...@gmail.com>.
Hi Jonathan,
>
> > For some reason 16637958 (the keys cached) has become a golden number
> and I
> > don't see key cache increasing beyond that.
>
> 16637958 is your configured cache capacity according to the cfstats you
> pasted.
this is another weird part. If you look at the schema[1] (pasted here,
extracted through cli), the keys_cached is set to 40million. And cfstats
shows that cassandra is only using 16 million.
Please note that I enabled row cache since I posted the cfstats. And
pasting the latest cfstats[2] herewith.
[1]
create column family YY
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'UTF8Type'
and key_validation_class = 'LongType'
and rows_cached = 10000.0
and row_cache_save_period = 0
and row_cache_keys_to_save = 2147483647
and keys_cached = 4.0E7
and key_cache_save_period = 14400
and read_repair_chance = 0.1
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and row_cache_provider = 'SerializingCacheProvider'
and compaction_strategy =
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and compression_options = {'chunk_length_kb' : '64',
'sstable_compression' :
'org.apache.cassandra.io.compress.SnappyCompressor'};
[2]
Keyspace: XXXXX
Read Count: 590667
Read Latency: 11.644107261790484 ms.
Write Count: 12720068
Write Latency: 0.06049896832312532 ms.
Pending Tasks: 0
Column Family: YY
SSTable count: 12
Space used (live): 66300297338
Space used (total): 66300297338
Number of Keys (estimate): 61152896
Memtable Columns Count: 881972
Memtable Data Size: 211749842
Memtable Switch Count: 267
Read Count: 590667
Read Latency: 16.859 ms.
Write Count: 12720068
Write Latency: 0.060 ms.
Pending Tasks: 0
Bloom Filter False Postives: 507
Bloom Filter False Ratio: 0.00099
Bloom Filter Space Used: 168118968
Key cache capacity: 40000000
Key cache size: 3838755
Key cache hit rate: 0.2072072072072072
Row cache capacity: 10000
Row cache size: 10000
Row cache hit rate: 0.022222222222222223
Compacted row minimum size: 51
Compacted row maximum size: 6866
Compacted row mean size: 2577
Thanks,
Eran
Re: Key cache hit rate issue
Posted by Jonathan Ellis <jb...@gmail.com>.
On Thu, Feb 16, 2012 at 3:52 PM, Eran Chinthaka Withana
<er...@gmail.com> wrote:
> Thanks for the reply. Yes there is a possibility that the keys can be
> distributed in multiple SSTables, but my data access patterns are such that
> I always read/write the whole row. So I expect all the data to be in the
> same SSTable (please correct me if I'm wrong).
If you're doing slice reads, Cassandra has no way of knowing that all
the data is in the most recent sstable, and will have to check the
others too.
> For some reason 16637958 (the keys cached) has become a golden number and I
> don't see key cache increasing beyond that.
16637958 is your configured cache capacity according to the cfstats you pasted.
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Key cache hit rate issue
Posted by Todd Burruss <bb...@expedia.com>.
there is a setting in the yaml file that helps relieve memory pressure by reducing the row cache. it is based on the percent of memory used by the JVM
the setting are, reduce_cache_sizes_at and reduce_cache_capacity_to. see how much free memory you have and if the numbers suggest that you have hit that limit, therefore reducing row cache size
From: Eran Chinthaka Withana <er...@gmail.com>>
Reply-To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Date: Thu, 16 Feb 2012 13:52:38 -0800
To: "user@cassandra.apache.org<ma...@cassandra.apache.org>" <us...@cassandra.apache.org>>
Subject: Re: Key cache hit rate issue
Hi Jonathan,
Thanks for the reply. Yes there is a possibility that the keys can be distributed in multiple SSTables, but my data access patterns are such that I always read/write the whole row. So I expect all the data to be in the same SSTable (please correct me if I'm wrong).
For some reason 16637958 (the keys cached) has become a golden number and I don't see key cache increasing beyond that. I also checked memory and I have about 4GB left in JVM memory and didn't see any issues on logs.
Thanks,
Eran Chinthaka Withana
On Thu, Feb 16, 2012 at 1:20 PM, Jonathan Ellis <jb...@gmail.com>> wrote:
So, you have roughly 1/6 of your (physical) row keys cached and about
1/4 cache hit rate, which doesn't sound unreasonable to me. Remember,
each logical key may be spread across multiple physical sstables --
each (key, sstable) pair is one entry in the key cache.
On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana
<er...@gmail.com>> wrote:
> Hi Aaron,
>
> Here it is.
>
> Keyspace: XXXX
> Read Count: 1123637972
> Read Latency: 5.757938114343114 ms.
> Write Count: 128201833
> Write Latency: 0.0682576607387509 ms.
> Pending Tasks: 0
> Column Family: YY
> SSTable count: 18
> Space used (live): 103318720685
> Space used (total): 103318720685
> Number of Keys (estimate): 92404992
> Memtable Columns Count: 1425580
> Memtable Data Size: 359655747
> Memtable Switch Count: 2522
> Read Count: 1123637972
> Read Latency: 14.731 ms.
> Write Count: 128201833
> Write Latency: NaN ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 1488
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 331522920
> Key cache capacity: 16637958
> Key cache size: 16637958
> Key cache hit rate: 0.2708333333333333
> Row cache: disabled
> Compacted row minimum size: 51
> Compacted row maximum size: 6866
> Compacted row mean size: 2560
>
> Thanks,
> Eran Chinthaka Withana
>
>
>
> On Thu, Feb 16, 2012 at 12:30 AM, aaron morton <aa...@thelastpickle.com>>
> wrote:
>>
>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 is
>> bit high. Is there a way to fix/improve it?
>>
>> Sorry I don't understand what you mean. But if the ratio is 0.0 all is
>> good.
>>
>> Could you include the full output from cfstats for the CF you are looking
>> at ?
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
>>
>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 is
>> bit high. Is there a way to fix/improve it?
>>
>> Thanks,
>> Eran Chinthaka Withana
>>
>>
>> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aa...@thelastpickle.com>>
>> wrote:
>>>
>>> Out of interest what does cfstats say about the bloom filter stats ? A
>>> high false positive could lead to a low key cache hit rate.
>>>
>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>> amount of keys I set as keys_cached?
>>>
>>> See key_cache_save_period when creating the CF.
>>>
>>> Cheers
>>>
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>>>
>>> Hi,
>>>
>>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80%
>>> (using the numerical values). This is visible in cfstats too. But I'm
>>> getting less than 20% (or sometimes even 0%) key cache hit rate. Well, the
>>> data access pattern is not the issue here as I know they are retrieving the
>>> same row multiple times. I'm using hector client with dynamic load balancing
>>> policy with consistency ONE for both reads and writes. Any ideas on how to
>>> find the issue and fix this?
>>>
>>> Here is what I see on cfstats.
>>>
>>> Key cache capacity: 16637958
>>> Key cache size: 16637958
>>> Key cache hit rate: 0.045454545454545456
>>>
>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>> amount of keys I set as keys_cached?
>>>
>>> Thanks,
>>> Eran
>>>
>>>
>>
>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Key cache hit rate issue
Posted by Eran Chinthaka Withana <er...@gmail.com>.
Hi Jonathan,
Thanks for the reply. Yes there is a possibility that the keys can be
distributed in multiple SSTables, but my data access patterns are such that
I always read/write the whole row. So I expect all the data to be in the
same SSTable (please correct me if I'm wrong).
For some reason 16637958 (the keys cached) has become a golden number and I
don't see key cache increasing beyond that. I also checked memory and I
have about 4GB left in JVM memory and didn't see any issues on logs.
Thanks,
Eran Chinthaka Withana
On Thu, Feb 16, 2012 at 1:20 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> So, you have roughly 1/6 of your (physical) row keys cached and about
> 1/4 cache hit rate, which doesn't sound unreasonable to me. Remember,
> each logical key may be spread across multiple physical sstables --
> each (key, sstable) pair is one entry in the key cache.
>
> On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana
> <er...@gmail.com> wrote:
> > Hi Aaron,
> >
> > Here it is.
> >
> > Keyspace: XXXX
> > Read Count: 1123637972
> > Read Latency: 5.757938114343114 ms.
> > Write Count: 128201833
> > Write Latency: 0.0682576607387509 ms.
> > Pending Tasks: 0
> > Column Family: YY
> > SSTable count: 18
> > Space used (live): 103318720685
> > Space used (total): 103318720685
> > Number of Keys (estimate): 92404992
> > Memtable Columns Count: 1425580
> > Memtable Data Size: 359655747
> > Memtable Switch Count: 2522
> > Read Count: 1123637972
> > Read Latency: 14.731 ms.
> > Write Count: 128201833
> > Write Latency: NaN ms.
> > Pending Tasks: 0
> > Bloom Filter False Postives: 1488
> > Bloom Filter False Ratio: 0.00000
> > Bloom Filter Space Used: 331522920
> > Key cache capacity: 16637958
> > Key cache size: 16637958
> > Key cache hit rate: 0.2708333333333333
> > Row cache: disabled
> > Compacted row minimum size: 51
> > Compacted row maximum size: 6866
> > Compacted row mean size: 2560
> >
> > Thanks,
> > Eran Chinthaka Withana
> >
> >
> >
> > On Thu, Feb 16, 2012 at 12:30 AM, aaron morton <aa...@thelastpickle.com>
> > wrote:
> >>
> >> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000
> is
> >> bit high. Is there a way to fix/improve it?
> >>
> >> Sorry I don't understand what you mean. But if the ratio is 0.0 all is
> >> good.
> >>
> >> Could you include the full output from cfstats for the CF you are
> looking
> >> at ?
> >>
> >> Cheers
> >>
> >> -----------------
> >> Aaron Morton
> >> Freelance Developer
> >> @aaronmorton
> >> http://www.thelastpickle.com
> >>
> >> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
> >>
> >> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000
> is
> >> bit high. Is there a way to fix/improve it?
> >>
> >> Thanks,
> >> Eran Chinthaka Withana
> >>
> >>
> >> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aa...@thelastpickle.com>
> >> wrote:
> >>>
> >>> Out of interest what does cfstats say about the bloom filter stats ? A
> >>> high false positive could lead to a low key cache hit rate.
> >>>
> >>> Also, is there a way to warm start the key cache, meaning pre-load the
> >>> amount of keys I set as keys_cached?
> >>>
> >>> See key_cache_save_period when creating the CF.
> >>>
> >>> Cheers
> >>>
> >>>
> >>> -----------------
> >>> Aaron Morton
> >>> Freelance Developer
> >>> @aaronmorton
> >>> http://www.thelastpickle.com
> >>>
> >>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
> >>>
> >>> Hi,
> >>>
> >>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80%
> >>> (using the numerical values). This is visible in cfstats too. But I'm
> >>> getting less than 20% (or sometimes even 0%) key cache hit rate. Well,
> the
> >>> data access pattern is not the issue here as I know they are
> retrieving the
> >>> same row multiple times. I'm using hector client with dynamic load
> balancing
> >>> policy with consistency ONE for both reads and writes. Any ideas on
> how to
> >>> find the issue and fix this?
> >>>
> >>> Here is what I see on cfstats.
> >>>
> >>> Key cache capacity: 16637958
> >>> Key cache size: 16637958
> >>> Key cache hit rate: 0.045454545454545456
> >>>
> >>> Also, is there a way to warm start the key cache, meaning pre-load the
> >>> amount of keys I set as keys_cached?
> >>>
> >>> Thanks,
> >>> Eran
> >>>
> >>>
> >>
> >>
> >
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of DataStax, the source for professional Cassandra support
> http://www.datastax.com
>
Re: Re: Key cache hit rate issue
Posted by zhangcheng <zh...@jike.com>.
according to the read process, the key of the keycache should be the row key.
2012-02-17
zhangcheng
发件人： Todd Burruss
发送时间： 2012-02-17 06:23:47
收件人： user@cassandra.apache.org
抄送：
主题： Re: Key cache hit rate issue
jonathan, you said the key to the cache is key + sstable? looking at the
code it looks like a DecoratedKey is the "row key". how does sstable come
into play?
On 2/16/12 1:20 PM, "Jonathan Ellis" <jb...@gmail.com> wrote:
>So, you have roughly 1/6 of your (physical) row keys cached and about
>1/4 cache hit rate, which doesn't sound unreasonable to me. Remember,
>each logical key may be spread across multiple physical sstables --
>each (key, sstable) pair is one entry in the key cache.
>
>On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana
><er...@gmail.com> wrote:
>> Hi Aaron,
>>
>> Here it is.
>>
>> Keyspace: XXXX
>> Read Count: 1123637972
>> Read Latency: 5.757938114343114 ms.
>> Write Count: 128201833
>> Write Latency: 0.0682576607387509 ms.
>> Pending Tasks: 0
>> Column Family: YY
>> SSTable count: 18
>> Space used (live): 103318720685
>> Space used (total): 103318720685
>> Number of Keys (estimate): 92404992
>> Memtable Columns Count: 1425580
>> Memtable Data Size: 359655747
>> Memtable Switch Count: 2522
>> Read Count: 1123637972
>> Read Latency: 14.731 ms.
>> Write Count: 128201833
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Bloom Filter False Postives: 1488
>> Bloom Filter False Ratio: 0.00000
>> Bloom Filter Space Used: 331522920
>> Key cache capacity: 16637958
>> Key cache size: 16637958
>> Key cache hit rate: 0.2708333333333333
>> Row cache: disabled
>> Compacted row minimum size: 51
>> Compacted row maximum size: 6866
>> Compacted row mean size: 2560
>>
>> Thanks,
>> Eran Chinthaka Withana
>>
>>
>>
>> On Thu, Feb 16, 2012 at 12:30 AM, aaron morton <aa...@thelastpickle.com>
>> wrote:
>>>
>>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>>>8000 is
>>> bit high. Is there a way to fix/improve it?
>>>
>>> Sorry I don't understand what you mean. But if the ratio is 0.0 all is
>>> good.
>>>
>>> Could you include the full output from cfstats for the CF you are
>>>looking
>>> at ?
>>>
>>> Cheers
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
>>>
>>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>>>8000 is
>>> bit high. Is there a way to fix/improve it?
>>>
>>> Thanks,
>>> Eran Chinthaka Withana
>>>
>>>
>>> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aa...@thelastpickle.com>
>>> wrote:
>>>>
>>>> Out of interest what does cfstats say about the bloom filter stats ? A
>>>> high false positive could lead to a low key cache hit rate.
>>>>
>>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>>> amount of keys I set as keys_cached?
>>>>
>>>> See key_cache_save_period when creating the CF.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>>
>>>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80%
>>>> (using the numerical values). This is visible in cfstats too. But I'm
>>>> getting less than 20% (or sometimes even 0%) key cache hit rate.
>>>>Well, the
>>>> data access pattern is not the issue here as I know they are
>>>>retrieving the
>>>> same row multiple times. I'm using hector client with dynamic load
>>>>balancing
>>>> policy with consistency ONE for both reads and writes. Any ideas on
>>>>how to
>>>> find the issue and fix this?
>>>>
>>>> Here is what I see on cfstats.
>>>>
>>>> Key cache capacity: 16637958
>>>> Key cache size: 16637958
>>>> Key cache hit rate: 0.045454545454545456
>>>>
>>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>>> amount of keys I set as keys_cached?
>>>>
>>>> Thanks,
>>>> Eran
>>>>
>>>>
>>>
>>>
>>
>
>
>
>--
>Jonathan Ellis
>Project Chair, Apache Cassandra
>co-founder of DataStax, the source for professional Cassandra support
>http://www.datastax.com
Re: Key cache hit rate issue
Posted by Todd Burruss <bb...@expedia.com>.
jonathan, you said the key to the cache is key + sstable? looking at the
code it looks like a DecoratedKey is the "row key". how does sstable come
into play?
On 2/16/12 1:20 PM, "Jonathan Ellis" <jb...@gmail.com> wrote:
>So, you have roughly 1/6 of your (physical) row keys cached and about
>1/4 cache hit rate, which doesn't sound unreasonable to me. Remember,
>each logical key may be spread across multiple physical sstables --
>each (key, sstable) pair is one entry in the key cache.
>
>On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana
><er...@gmail.com> wrote:
>> Hi Aaron,
>>
>> Here it is.
>>
>> Keyspace: XXXX
>> Read Count: 1123637972
>> Read Latency: 5.757938114343114 ms.
>> Write Count: 128201833
>> Write Latency: 0.0682576607387509 ms.
>> Pending Tasks: 0
>> Column Family: YY
>> SSTable count: 18
>> Space used (live): 103318720685
>> Space used (total): 103318720685
>> Number of Keys (estimate): 92404992
>> Memtable Columns Count: 1425580
>> Memtable Data Size: 359655747
>> Memtable Switch Count: 2522
>> Read Count: 1123637972
>> Read Latency: 14.731 ms.
>> Write Count: 128201833
>> Write Latency: NaN ms.
>> Pending Tasks: 0
>> Bloom Filter False Postives: 1488
>> Bloom Filter False Ratio: 0.00000
>> Bloom Filter Space Used: 331522920
>> Key cache capacity: 16637958
>> Key cache size: 16637958
>> Key cache hit rate: 0.2708333333333333
>> Row cache: disabled
>> Compacted row minimum size: 51
>> Compacted row maximum size: 6866
>> Compacted row mean size: 2560
>>
>> Thanks,
>> Eran Chinthaka Withana
>>
>>
>>
>> On Thu, Feb 16, 2012 at 12:30 AM, aaron morton <aa...@thelastpickle.com>
>> wrote:
>>>
>>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>>>8000 is
>>> bit high. Is there a way to fix/improve it?
>>>
>>> Sorry I don't understand what you mean. But if the ratio is 0.0 all is
>>> good.
>>>
>>> Could you include the full output from cfstats for the CF you are
>>>looking
>>> at ?
>>>
>>> Cheers
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
>>>
>>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess
>>>8000 is
>>> bit high. Is there a way to fix/improve it?
>>>
>>> Thanks,
>>> Eran Chinthaka Withana
>>>
>>>
>>> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aa...@thelastpickle.com>
>>> wrote:
>>>>
>>>> Out of interest what does cfstats say about the bloom filter stats ? A
>>>> high false positive could lead to a low key cache hit rate.
>>>>
>>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>>> amount of keys I set as keys_cached?
>>>>
>>>> See key_cache_save_period when creating the CF.
>>>>
>>>> Cheers
>>>>
>>>>
>>>> -----------------
>>>> Aaron Morton
>>>> Freelance Developer
>>>> @aaronmorton
>>>> http://www.thelastpickle.com
>>>>
>>>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>>>>
>>>> Hi,
>>>>
>>>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80%
>>>> (using the numerical values). This is visible in cfstats too. But I'm
>>>> getting less than 20% (or sometimes even 0%) key cache hit rate.
>>>>Well, the
>>>> data access pattern is not the issue here as I know they are
>>>>retrieving the
>>>> same row multiple times. I'm using hector client with dynamic load
>>>>balancing
>>>> policy with consistency ONE for both reads and writes. Any ideas on
>>>>how to
>>>> find the issue and fix this?
>>>>
>>>> Here is what I see on cfstats.
>>>>
>>>> Key cache capacity: 16637958
>>>> Key cache size: 16637958
>>>> Key cache hit rate: 0.045454545454545456
>>>>
>>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>>> amount of keys I set as keys_cached?
>>>>
>>>> Thanks,
>>>> Eran
>>>>
>>>>
>>>
>>>
>>
>
>
>
>--
>Jonathan Ellis
>Project Chair, Apache Cassandra
>co-founder of DataStax, the source for professional Cassandra support
>http://www.datastax.com
Re: Key cache hit rate issue
Posted by Jonathan Ellis <jb...@gmail.com>.
So, you have roughly 1/6 of your (physical) row keys cached and about
1/4 cache hit rate, which doesn't sound unreasonable to me. Remember,
each logical key may be spread across multiple physical sstables --
each (key, sstable) pair is one entry in the key cache.
On Thu, Feb 16, 2012 at 1:48 PM, Eran Chinthaka Withana
<er...@gmail.com> wrote:
> Hi Aaron,
>
> Here it is.
>
> Keyspace: XXXX
> Read Count: 1123637972
> Read Latency: 5.757938114343114 ms.
> Write Count: 128201833
> Write Latency: 0.0682576607387509 ms.
> Pending Tasks: 0
> Column Family: YY
> SSTable count: 18
> Space used (live): 103318720685
> Space used (total): 103318720685
> Number of Keys (estimate): 92404992
> Memtable Columns Count: 1425580
> Memtable Data Size: 359655747
> Memtable Switch Count: 2522
> Read Count: 1123637972
> Read Latency: 14.731 ms.
> Write Count: 128201833
> Write Latency: NaN ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 1488
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 331522920
> Key cache capacity: 16637958
> Key cache size: 16637958
> Key cache hit rate: 0.2708333333333333
> Row cache: disabled
> Compacted row minimum size: 51
> Compacted row maximum size: 6866
> Compacted row mean size: 2560
>
> Thanks,
> Eran Chinthaka Withana
>
>
>
> On Thu, Feb 16, 2012 at 12:30 AM, aaron morton <aa...@thelastpickle.com>
> wrote:
>>
>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 is
>> bit high. Is there a way to fix/improve it?
>>
>> Sorry I don't understand what you mean. But if the ratio is 0.0 all is
>> good.
>>
>> Could you include the full output from cfstats for the CF you are looking
>> at ?
>>
>> Cheers
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
>>
>> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 is
>> bit high. Is there a way to fix/improve it?
>>
>> Thanks,
>> Eran Chinthaka Withana
>>
>>
>> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aa...@thelastpickle.com>
>> wrote:
>>>
>>> Out of interest what does cfstats say about the bloom filter stats ? A
>>> high false positive could lead to a low key cache hit rate.
>>>
>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>> amount of keys I set as keys_cached?
>>>
>>> See key_cache_save_period when creating the CF.
>>>
>>> Cheers
>>>
>>>
>>> -----------------
>>> Aaron Morton
>>> Freelance Developer
>>> @aaronmorton
>>> http://www.thelastpickle.com
>>>
>>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>>>
>>> Hi,
>>>
>>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80%
>>> (using the numerical values). This is visible in cfstats too. But I'm
>>> getting less than 20% (or sometimes even 0%) key cache hit rate. Well, the
>>> data access pattern is not the issue here as I know they are retrieving the
>>> same row multiple times. I'm using hector client with dynamic load balancing
>>> policy with consistency ONE for both reads and writes. Any ideas on how to
>>> find the issue and fix this?
>>>
>>> Here is what I see on cfstats.
>>>
>>> Key cache capacity: 16637958
>>> Key cache size: 16637958
>>> Key cache hit rate: 0.045454545454545456
>>>
>>> Also, is there a way to warm start the key cache, meaning pre-load the
>>> amount of keys I set as keys_cached?
>>>
>>> Thanks,
>>> Eran
>>>
>>>
>>
>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Key cache hit rate issue
Posted by Eran Chinthaka Withana <er...@gmail.com>.
Hi Aaron,
Here it is.
Keyspace: XXXX
Read Count: 1123637972
Read Latency: 5.757938114343114 ms.
Write Count: 128201833
Write Latency: 0.0682576607387509 ms.
Pending Tasks: 0
Column Family: YY
SSTable count: 18
Space used (live): 103318720685
Space used (total): 103318720685
Number of Keys (estimate): 92404992
Memtable Columns Count: 1425580
Memtable Data Size: 359655747
Memtable Switch Count: 2522
Read Count: 1123637972
Read Latency: 14.731 ms.
Write Count: 128201833
Write Latency: NaN ms.
Pending Tasks: 0
Bloom Filter False Postives: 1488
Bloom Filter False Ratio: 0.00000
Bloom Filter Space Used: 331522920
Key cache capacity: 16637958
Key cache size: 16637958
Key cache hit rate: 0.2708333333333333
Row cache: disabled
Compacted row minimum size: 51
Compacted row maximum size: 6866
Compacted row mean size: 2560
Thanks,
Eran Chinthaka Withana
On Thu, Feb 16, 2012 at 12:30 AM, aaron morton <aa...@thelastpickle.com>wrote:
> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 is
> bit high. Is there a way to fix/improve it?
>
> Sorry I don't understand what you mean. But if the ratio is 0.0 all is
> good.
>
> Could you include the full output from cfstats for the CF you are looking
> at ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
>
> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 is
> bit high. Is there a way to fix/improve it?
>
> Thanks,
> Eran Chinthaka Withana
>
>
> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aa...@thelastpickle.com>wrote:
>
>> Out of interest what does cfstats say about the bloom filter stats ? A
>> high false positive could lead to a low key cache hit rate.
>>
>> Also, is there a way to warm start the key cache, meaning pre-load the
>> amount of keys I set as keys_cached?
>>
>> See key_cache_save_period when creating the CF.
>>
>> Cheers
>>
>>
>> -----------------
>> Aaron Morton
>> Freelance Developer
>> @aaronmorton
>> http://www.thelastpickle.com
>>
>> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>>
>> Hi,
>>
>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80%
>> (using the numerical values). This is visible in cfstats too. But I'm
>> getting less than 20% (or sometimes even 0%) key cache hit rate. Well, the
>> data access pattern is not the issue here as I know they are retrieving the
>> same row multiple times. I'm using hector client with dynamic load
>> balancing policy with consistency ONE for both reads and writes. Any ideas
>> on how to find the issue and fix this?
>>
>> Here is what I see on cfstats.
>>
>> Key cache capacity: 16637958
>> Key cache size: 16637958
>> Key cache hit rate: 0.045454545454545456
>>
>> Also, is there a way to warm start the key cache, meaning pre-load the
>> amount of keys I set as keys_cached?
>>
>> Thanks,
>> Eran
>>
>>
>>
>
>
Re: Key cache hit rate issue
Posted by aaron morton <aa...@thelastpickle.com>.
> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 is bit high. Is there a way to fix/improve it?
Sorry I don't understand what you mean. But if the ratio is 0.0 all is good.
Could you include the full output from cfstats for the CF you are looking at ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 15/02/2012, at 1:00 PM, Eran Chinthaka Withana wrote:
> Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 is bit high. Is there a way to fix/improve it?
>
> Thanks,
> Eran Chinthaka Withana
>
>
> On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aa...@thelastpickle.com> wrote:
> Out of interest what does cfstats say about the bloom filter stats ? A high false positive could lead to a low key cache hit rate.
>
>> Also, is there a way to warm start the key cache, meaning pre-load the amount of keys I set as keys_cached?
>
> See key_cache_save_period when creating the CF.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>
>> Hi,
>>
>> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80% (using the numerical values). This is visible in cfstats too. But I'm getting less than 20% (or sometimes even 0%) key cache hit rate. Well, the data access pattern is not the issue here as I know they are retrieving the same row multiple times. I'm using hector client with dynamic load balancing policy with consistency ONE for both reads and writes. Any ideas on how to find the issue and fix this?
>>
>> Here is what I see on cfstats.
>>
>> Key cache capacity: 16637958
>> Key cache size: 16637958
>> Key cache hit rate: 0.045454545454545456
>>
>> Also, is there a way to warm start the key cache, meaning pre-load the amount of keys I set as keys_cached?
>>
>> Thanks,
>> Eran
>
>
Re: Key cache hit rate issue
Posted by Eran Chinthaka Withana <er...@gmail.com>.
Its in the order of 261 to 8000 and the ratio is 0.00. But i guess 8000 is
bit high. Is there a way to fix/improve it?
Thanks,
Eran Chinthaka Withana
On Tue, Feb 14, 2012 at 3:42 PM, aaron morton <aa...@thelastpickle.com>wrote:
> Out of interest what does cfstats say about the bloom filter stats ? A
> high false positive could lead to a low key cache hit rate.
>
> Also, is there a way to warm start the key cache, meaning pre-load the
> amount of keys I set as keys_cached?
>
> See key_cache_save_period when creating the CF.
>
> Cheers
>
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
>
> Hi,
>
> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80% (using
> the numerical values). This is visible in cfstats too. But I'm getting less
> than 20% (or sometimes even 0%) key cache hit rate. Well, the data access
> pattern is not the issue here as I know they are retrieving the same row
> multiple times. I'm using hector client with dynamic load balancing policy
> with consistency ONE for both reads and writes. Any ideas on how to find
> the issue and fix this?
>
> Here is what I see on cfstats.
>
> Key cache capacity: 16637958
> Key cache size: 16637958
> Key cache hit rate: 0.045454545454545456
>
> Also, is there a way to warm start the key cache, meaning pre-load the
> amount of keys I set as keys_cached?
>
> Thanks,
> Eran
>
>
>
Re: Key cache hit rate issue
Posted by aaron morton <aa...@thelastpickle.com>.
Out of interest what does cfstats say about the bloom filter stats ? A high false positive could lead to a low key cache hit rate.
> Also, is there a way to warm start the key cache, meaning pre-load the amount of keys I set as keys_cached?
See key_cache_save_period when creating the CF.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 15/02/2012, at 5:54 AM, Eran Chinthaka Withana wrote:
> Hi,
>
> I'm using Cassandra 1.0.7 and I've set the keys_cached to about 80% (using the numerical values). This is visible in cfstats too. But I'm getting less than 20% (or sometimes even 0%) key cache hit rate. Well, the data access pattern is not the issue here as I know they are retrieving the same row multiple times. I'm using hector client with dynamic load balancing policy with consistency ONE for both reads and writes. Any ideas on how to find the issue and fix this?
>
> Here is what I see on cfstats.
>
> Key cache capacity: 16637958
> Key cache size: 16637958
> Key cache hit rate: 0.045454545454545456
>
> Also, is there a way to warm start the key cache, meaning pre-load the amount of keys I set as keys_cached?
>
> Thanks,
> Eran