You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by yun peng <pe...@gmail.com> on 2013/05/29 13:10:53 UTC

How does BlockCache find relevant Blocks?

Hi, All,
Given a large-sized BlockCache, I am wondering how HBase performs to search
a block of a requested rowkey. Is there any index structure (in-memory)
inside BlockCache? or it search in a brute force way, (which seems unlikely
though :-...)

I am aware of reference guide online, but have not yet find the answer...
http://hbase.apache.org/book.html#block.cache

Thx,
Yun

Re: How does BlockCache find relevant Blocks?

Posted by Anoop John <an...@gmail.com>.
Block cache maintain the blocks in memory with a key, value DS.   The value
is the block and the key (BlockCacheKey) includes the HFile name and offset
of the block in this file...    As I said 1st we will determine in which
block this RK can be found. So we know the HFile name and offset now.  We
can create the BlockCacheKey and do a get(key)

-Anoop-

On Wed, May 29, 2013 at 5:37 PM, yun peng <pe...@gmail.com> wrote:

> Thanks for the clarification, and I am interested in knowing a bit deep...
>
> So how does hbase maintains which block is in memory and which is not.
> is there an extra level of index (like meta data index) here in memory? if
> there
> is, is it a hash index or other?...
>
> Regards
> Yun
>
>
> On Wed, May 29, 2013 at 7:45 AM, Anoop John <an...@gmail.com> wrote:
>
> > There is an index for the blocks in a HFile. This index contains details
> > like start row in the block, its offset and length in the HFile...   So
> as
> > a 1st setp to get a rowkey, we will find this rk can be present in which
> > HFile block.. (I am assuming only one HFile as of now)..   Now we will
> see
> > whether this block is already present in the block cache or not. If so
> try
> > finding the row within that block..   If this HFile block is not present
> in
> > the cache, we will load it from HDFS..
> >
> > Hope this helps u..
> >
> > -Anoop-
> >
> > On Wed, May 29, 2013 at 4:40 PM, yun peng <pe...@gmail.com> wrote:
> >
> > > Hi, All,
> > > Given a large-sized BlockCache, I am wondering how HBase performs to
> > search
> > > a block of a requested rowkey. Is there any index structure (in-memory)
> > > inside BlockCache? or it search in a brute force way, (which seems
> > unlikely
> > > though :-...)
> > >
> > > I am aware of reference guide online, but have not yet find the
> answer...
> > > http://hbase.apache.org/book.html#block.cache
> > >
> > > Thx,
> > > Yun
> > >
> >
>

Re: How does BlockCache find relevant Blocks?

Posted by yun peng <pe...@gmail.com>.
Thanks for the clarification, and I am interested in knowing a bit deep...

So how does hbase maintains which block is in memory and which is not.
is there an extra level of index (like meta data index) here in memory? if
there
is, is it a hash index or other?...

Regards
Yun


On Wed, May 29, 2013 at 7:45 AM, Anoop John <an...@gmail.com> wrote:

> There is an index for the blocks in a HFile. This index contains details
> like start row in the block, its offset and length in the HFile...   So as
> a 1st setp to get a rowkey, we will find this rk can be present in which
> HFile block.. (I am assuming only one HFile as of now)..   Now we will see
> whether this block is already present in the block cache or not. If so try
> finding the row within that block..   If this HFile block is not present in
> the cache, we will load it from HDFS..
>
> Hope this helps u..
>
> -Anoop-
>
> On Wed, May 29, 2013 at 4:40 PM, yun peng <pe...@gmail.com> wrote:
>
> > Hi, All,
> > Given a large-sized BlockCache, I am wondering how HBase performs to
> search
> > a block of a requested rowkey. Is there any index structure (in-memory)
> > inside BlockCache? or it search in a brute force way, (which seems
> unlikely
> > though :-...)
> >
> > I am aware of reference guide online, but have not yet find the answer...
> > http://hbase.apache.org/book.html#block.cache
> >
> > Thx,
> > Yun
> >
>

Re: How does BlockCache find relevant Blocks?

Posted by Anoop John <an...@gmail.com>.
There is an index for the blocks in a HFile. This index contains details
like start row in the block, its offset and length in the HFile...   So as
a 1st setp to get a rowkey, we will find this rk can be present in which
HFile block.. (I am assuming only one HFile as of now)..   Now we will see
whether this block is already present in the block cache or not. If so try
finding the row within that block..   If this HFile block is not present in
the cache, we will load it from HDFS..

Hope this helps u..

-Anoop-

On Wed, May 29, 2013 at 4:40 PM, yun peng <pe...@gmail.com> wrote:

> Hi, All,
> Given a large-sized BlockCache, I am wondering how HBase performs to search
> a block of a requested rowkey. Is there any index structure (in-memory)
> inside BlockCache? or it search in a brute force way, (which seems unlikely
> though :-...)
>
> I am aware of reference guide online, but have not yet find the answer...
> http://hbase.apache.org/book.html#block.cache
>
> Thx,
> Yun
>