You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Varun Sharma <va...@pinterest.com> on 2013/01/27 04:05:57 UTC
LRU stats
Hi,
I am running hbase 0.94.3 and I am trying to understand the LRU cache stats
here:
LRU Stats: total=428.75 MB, free=4.05 GB, max=4.46 GB, blocks=35961,
accesses=23808283, hits=23098837, hitRatio=97.02%,
cachingAccesses=23135166, cachingHits=23066278,
cachingHitsRatio=99.70%, *evictions=0,
evicted=33211*
*
*
I don't understanding why the number of evictions is 0 - which means that
the number of eviction thread runs is 0, AFAIK. Also, I see that 30K blocks
were evicted - how is that possible ? I have set the hfile evict on close
to the default value - which is false - so blocks should not be evicted
when hfiles are closed. Where are the evicted blocks coming from when the
block is mostly empty and no eviction runs have occured.
Thanks
Varun
Re: LRU stats
Posted by Jean-Daniel Cryans <jd...@apache.org>.
IIRC when a file closes it will evict its own blocks since they won't
be used after that.
J-D
On Sun, Jan 27, 2013 at 1:04 AM, Varun Sharma <va...@pinterest.com> wrote:
> Since i am using only 10 % of allocated cache, I think EvictionThread never
> ran - hence, I see the value 0. What's mysterious is who is calling
> evictBlock then, because there are non zero evicted blocks ?
>
> On Sat, Jan 26, 2013 at 9:24 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Evicted block count stat is gathered in:
>>
>> protected long evictBlock(CachedBlock block) {
>>
>> map.remove(block.getCacheKey());
>>
>> updateSizeMetrics(block, true);
>>
>> elements.decrementAndGet();
>>
>> stats.evicted();
>>
>> In EvictionThread#run():
>>
>> LruBlockCache cache = this.cache.get();
>>
>> if(cache == null) break;
>>
>> cache.evict();
>>
>> where cache is declared:
>>
>> private WeakReference<LruBlockCache> cache;
>>
>> Looks like cache became null (due to heap pressure), leading to 0 evictions
>> that you saw.
>>
>>
>> Cheers
>>
>>
>> On Sat, Jan 26, 2013 at 7:05 PM, Varun Sharma <va...@pinterest.com> wrote:
>>
>> > Hi,
>> >
>> > I am running hbase 0.94.3 and I am trying to understand the LRU cache
>> stats
>> > here:
>> >
>> > LRU Stats: total=428.75 MB, free=4.05 GB, max=4.46 GB, blocks=35961,
>> > accesses=23808283, hits=23098837, hitRatio=97.02%,
>> > cachingAccesses=23135166, cachingHits=23066278,
>> > cachingHitsRatio=99.70%, *evictions=0,
>> > evicted=33211*
>> > *
>> > *
>> > I don't understanding why the number of evictions is 0 - which means that
>> > the number of eviction thread runs is 0, AFAIK. Also, I see that 30K
>> blocks
>> > were evicted - how is that possible ? I have set the hfile evict on close
>> > to the default value - which is false - so blocks should not be evicted
>> > when hfiles are closed. Where are the evicted blocks coming from when the
>> > block is mostly empty and no eviction runs have occured.
>> >
>> > Thanks
>> > Varun
>> >
>>
Re: LRU stats
Posted by Varun Sharma <va...@pinterest.com>.
Since i am using only 10 % of allocated cache, I think EvictionThread never
ran - hence, I see the value 0. What's mysterious is who is calling
evictBlock then, because there are non zero evicted blocks ?
On Sat, Jan 26, 2013 at 9:24 PM, Ted Yu <yu...@gmail.com> wrote:
> Evicted block count stat is gathered in:
>
> protected long evictBlock(CachedBlock block) {
>
> map.remove(block.getCacheKey());
>
> updateSizeMetrics(block, true);
>
> elements.decrementAndGet();
>
> stats.evicted();
>
> In EvictionThread#run():
>
> LruBlockCache cache = this.cache.get();
>
> if(cache == null) break;
>
> cache.evict();
>
> where cache is declared:
>
> private WeakReference<LruBlockCache> cache;
>
> Looks like cache became null (due to heap pressure), leading to 0 evictions
> that you saw.
>
>
> Cheers
>
>
> On Sat, Jan 26, 2013 at 7:05 PM, Varun Sharma <va...@pinterest.com> wrote:
>
> > Hi,
> >
> > I am running hbase 0.94.3 and I am trying to understand the LRU cache
> stats
> > here:
> >
> > LRU Stats: total=428.75 MB, free=4.05 GB, max=4.46 GB, blocks=35961,
> > accesses=23808283, hits=23098837, hitRatio=97.02%,
> > cachingAccesses=23135166, cachingHits=23066278,
> > cachingHitsRatio=99.70%, *evictions=0,
> > evicted=33211*
> > *
> > *
> > I don't understanding why the number of evictions is 0 - which means that
> > the number of eviction thread runs is 0, AFAIK. Also, I see that 30K
> blocks
> > were evicted - how is that possible ? I have set the hfile evict on close
> > to the default value - which is false - so blocks should not be evicted
> > when hfiles are closed. Where are the evicted blocks coming from when the
> > block is mostly empty and no eviction runs have occured.
> >
> > Thanks
> > Varun
> >
>
Re: LRU stats
Posted by Ted Yu <yu...@gmail.com>.
Evicted block count stat is gathered in:
protected long evictBlock(CachedBlock block) {
map.remove(block.getCacheKey());
updateSizeMetrics(block, true);
elements.decrementAndGet();
stats.evicted();
In EvictionThread#run():
LruBlockCache cache = this.cache.get();
if(cache == null) break;
cache.evict();
where cache is declared:
private WeakReference<LruBlockCache> cache;
Looks like cache became null (due to heap pressure), leading to 0 evictions
that you saw.
Cheers
On Sat, Jan 26, 2013 at 7:05 PM, Varun Sharma <va...@pinterest.com> wrote:
> Hi,
>
> I am running hbase 0.94.3 and I am trying to understand the LRU cache stats
> here:
>
> LRU Stats: total=428.75 MB, free=4.05 GB, max=4.46 GB, blocks=35961,
> accesses=23808283, hits=23098837, hitRatio=97.02%,
> cachingAccesses=23135166, cachingHits=23066278,
> cachingHitsRatio=99.70%, *evictions=0,
> evicted=33211*
> *
> *
> I don't understanding why the number of evictions is 0 - which means that
> the number of eviction thread runs is 0, AFAIK. Also, I see that 30K blocks
> were evicted - how is that possible ? I have set the hfile evict on close
> to the default value - which is false - so blocks should not be evicted
> when hfiles are closed. Where are the evicted blocks coming from when the
> block is mostly empty and no eviction runs have occured.
>
> Thanks
> Varun
>