Prefetching Indexes of HFiles

[Anthony Nguyen <an...@gmail.com>]

Hi all,

Is there a better way to warm HFile indexes other than scanning through my
datasets? I did see PREFETCH_BLOCKS_ON_OPEN, but the warning that it "is
not a good idea if the data to be preloaded will not fit into the
blockcache" makes me wary. Why would this be a bad idea?

Thanks!

Re: Prefetching Indexes of HFiles

[Anthony Nguyen <an...@gmail.com>]

Thanks! If I understand it correctly, we need to at least open every HFile
once to load root indexes into the cache, right?

Another question - I went through Nick's BlockCache 101 and was intrigued
by BucketCache / CombinedBlockCache. If I used BucketCache in file mode
with SSDs, I see that I can persist data blocks across cluster lifecycles
and therefore have a warmed cluster. Would it be possible for me to modify
it such that I could also store the bloom filters / index blocks which are
currently only stored in the LruBlockCache into the BucketCache?

Thanks again.

On Thu, Sep 24, 2015 at 5:13 PM, Ted Yu <yu...@gmail.com> wrote:

> I think the option you mentioned is for data blocks.
>
> As for index blocks, please refer to HFile V2 design doc.
> Also see this:
> https://issues.apache.org/jira/browse/HBASE
>
> -3857?focusedCommentId=13031489&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13031489
>
> Cheers
>
> On Thu, Sep 24, 2015 at 1:16 PM, Anthony Nguyen <
> anthony.an.nguyen@gmail.com
> > wrote:
>
> > Hi all,
> >
> > Is there a better way to warm HFile indexes other than scanning through
> my
> > datasets? I did see PREFETCH_BLOCKS_ON_OPEN, but the warning that it "is
> > not a good idea if the data to be preloaded will not fit into the
> > blockcache" makes me wary. Why would this be a bad idea?
> >
> > Thanks!
> >
>

Re: Prefetching Indexes of HFiles

[Ted Yu <yu...@gmail.com>]

I think the option you mentioned is for data blocks.

As for index blocks, please refer to HFile V2 design doc.
Also see this:
https://issues.apache.org/jira/browse/HBASE
-3857?focusedCommentId=13031489&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13031489

Cheers

On Thu, Sep 24, 2015 at 1:16 PM, Anthony Nguyen <anthony.an.nguyen@gmail.com
> wrote:

> Hi all,
>
> Is there a better way to warm HFile indexes other than scanning through my
> datasets? I did see PREFETCH_BLOCKS_ON_OPEN, but the warning that it "is
> not a good idea if the data to be preloaded will not fit into the
> blockcache" makes me wary. Why would this be a bad idea?
>
> Thanks!
>