You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Eric Czech <ec...@gmail.com> on 2013/02/22 16:39:14 UTC
Prewarming in-memory column families
Hi everyone,
Are blocks for in-memory column families automatically loaded in to the
block cache on restart? If not, would anyone recommend running a scan with
.setCacheBlocks(true) after a restart for in-memory column families?
Thanks!
Re: Prewarming in-memory column families
Posted by Eric Czech <ec...@gmail.com>.
Thanks guys.
I think I'll use the initial scan then but limit the rows used to a smaller
subset that I know should fit in the in-memory part of the block cache.
On Tue, Feb 26, 2013 at 8:16 PM, Jean-Daniel Cryans <jd...@apache.org>wrote:
> It's touchy... what if the data set doesn't fit in the in-memory's
> part of the block cache (which is 25%)? Maybe the user only wants to
> keep "in-memory" those edits that are being used? What about the IO
> hit of assigning those regions at startup that would now need to read
> X GBs all at once?
>
> FWIW I've never been a fan of that setting.
>
> J-D
>
> On Tue, Feb 26, 2013 at 5:04 PM, Sergey Shelukhin
> <se...@hortonworks.com> wrote:
> > should we make this built-in? Sounds like default user intent for
> in-memory.
> >
> > On Tue, Feb 26, 2013 at 2:13 PM, Stack <st...@duboce.net> wrote:
> >
> >> On Fri, Feb 22, 2013 at 7:39 AM, Eric Czech <ec...@gmail.com> wrote:
> >>
> >> > Hi everyone,
> >> >
> >> > Are blocks for in-memory column families automatically loaded in to
> the
> >> > block cache on restart?
> >>
> >>
> >> No
> >>
> >>
> >> > If not, would anyone recommend running a scan with
> >> > .setCacheBlocks(true) after a restart for in-memory column families?
> >> >
> >>
> >> Yes.
> >>
> >> It should be easy verifying whether the above warmup had an effect.
> >>
> >> Good luck,
> >> St.Ack
> >>
>
Re: Prewarming in-memory column families
Posted by Jean-Daniel Cryans <jd...@apache.org>.
It's touchy... what if the data set doesn't fit in the in-memory's
part of the block cache (which is 25%)? Maybe the user only wants to
keep "in-memory" those edits that are being used? What about the IO
hit of assigning those regions at startup that would now need to read
X GBs all at once?
FWIW I've never been a fan of that setting.
J-D
On Tue, Feb 26, 2013 at 5:04 PM, Sergey Shelukhin
<se...@hortonworks.com> wrote:
> should we make this built-in? Sounds like default user intent for in-memory.
>
> On Tue, Feb 26, 2013 at 2:13 PM, Stack <st...@duboce.net> wrote:
>
>> On Fri, Feb 22, 2013 at 7:39 AM, Eric Czech <ec...@gmail.com> wrote:
>>
>> > Hi everyone,
>> >
>> > Are blocks for in-memory column families automatically loaded in to the
>> > block cache on restart?
>>
>>
>> No
>>
>>
>> > If not, would anyone recommend running a scan with
>> > .setCacheBlocks(true) after a restart for in-memory column families?
>> >
>>
>> Yes.
>>
>> It should be easy verifying whether the above warmup had an effect.
>>
>> Good luck,
>> St.Ack
>>
Re: Prewarming in-memory column families
Posted by Sergey Shelukhin <se...@hortonworks.com>.
should we make this built-in? Sounds like default user intent for in-memory.
On Tue, Feb 26, 2013 at 2:13 PM, Stack <st...@duboce.net> wrote:
> On Fri, Feb 22, 2013 at 7:39 AM, Eric Czech <ec...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > Are blocks for in-memory column families automatically loaded in to the
> > block cache on restart?
>
>
> No
>
>
> > If not, would anyone recommend running a scan with
> > .setCacheBlocks(true) after a restart for in-memory column families?
> >
>
> Yes.
>
> It should be easy verifying whether the above warmup had an effect.
>
> Good luck,
> St.Ack
>
Re: Prewarming in-memory column families
Posted by Stack <st...@duboce.net>.
On Fri, Feb 22, 2013 at 7:39 AM, Eric Czech <ec...@gmail.com> wrote:
> Hi everyone,
>
> Are blocks for in-memory column families automatically loaded in to the
> block cache on restart?
No
> If not, would anyone recommend running a scan with
> .setCacheBlocks(true) after a restart for in-memory column families?
>
Yes.
It should be easy verifying whether the above warmup had an effect.
Good luck,
St.Ack