You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jason Huang <ja...@icare.com> on 2012/09/12 23:15:23 UTC
HBase 'Real-Time' reads?
This might be a naive question but I am not able to find a good answer
from searching online.
The online guide mentioned that "Puts and Deletes are collected into
an in-memory structure called the MemStore. Before the MemStore is
update the changes are written to a Write Ahead Log (WAL) to enable
recovery in case a server crashes. When it reaches a certain size the
MemStore is flushed to disk into StoreFile."
So, if an application tried to query a certain piece of data that
hasn't been flushed to disk into StoreFile yet, where is HBase
designed to get that piece of data? Is it going to the Region servers
and tried to get the previous version of this data, or is it smart
enough to go to the MemStore or WAL to get the most recent version of
data?
thanks!
Jason
Re: HBase 'Real-Time' reads?
Posted by Jason Huang <ja...@icare.com>.
I see now.
Thanks for the quick response and clear explanation.
Jason
On Wed, Sep 12, 2012 at 5:28 PM, Adrien Mogenet
<ad...@gmail.com> wrote:
> I think you misunderstand concept of "memstore". That's just the name of
> the temporary in-memory storage. Each region has its own memstore, and thus
> it's located on the regionserver itself.
>
> On Wed, Sep 12, 2012 at 11:24 PM, Jason Huang <ja...@icare.com> wrote:
>
>> So - I guess at the time of the query we don't know if the data is in
>> Memstore or in the RegionServer. In order to ensure we get the most
>> recent version of data, every Hbase Read query will first go to
>> Memstore and see if the data is there, and then go to RegionServers if
>> it couldn't find that data in Memstore?
>>
>> thanks,
>>
>> Jason
>>
>> On Wed, Sep 12, 2012 at 5:19 PM, Adrien Mogenet
>> <ad...@gmail.com> wrote:
>> > WAL is just there for recover. Reads will meet the Memstore on their read
>> > path, that's how LSM Trees are working.
>> >
>> > On Wed, Sep 12, 2012 at 11:15 PM, Jason Huang <ja...@icare.com>
>> wrote:
>> >
>> >> This might be a naive question but I am not able to find a good answer
>> >> from searching online.
>> >>
>> >> The online guide mentioned that "Puts and Deletes are collected into
>> >> an in-memory structure called the MemStore. Before the MemStore is
>> >> update the changes are written to a Write Ahead Log (WAL) to enable
>> >> recovery in case a server crashes. When it reaches a certain size the
>> >> MemStore is flushed to disk into StoreFile."
>> >>
>> >> So, if an application tried to query a certain piece of data that
>> >> hasn't been flushed to disk into StoreFile yet, where is HBase
>> >> designed to get that piece of data? Is it going to the Region servers
>> >> and tried to get the previous version of this data, or is it smart
>> >> enough to go to the MemStore or WAL to get the most recent version of
>> >> data?
>> >>
>> >> thanks!
>> >>
>> >> Jason
>> >>
>> >
>> >
>> > --
>> > Adrien Mogenet
>> > 06.59.16.64.22
>> > http://www.mogenet.me
>>
>
>
>
> --
> Adrien Mogenet
> 06.59.16.64.22
> http://www.mogenet.me
Re: HBase 'Real-Time' reads?
Posted by Adrien Mogenet <ad...@gmail.com>.
I think you misunderstand concept of "memstore". That's just the name of
the temporary in-memory storage. Each region has its own memstore, and thus
it's located on the regionserver itself.
On Wed, Sep 12, 2012 at 11:24 PM, Jason Huang <ja...@icare.com> wrote:
> So - I guess at the time of the query we don't know if the data is in
> Memstore or in the RegionServer. In order to ensure we get the most
> recent version of data, every Hbase Read query will first go to
> Memstore and see if the data is there, and then go to RegionServers if
> it couldn't find that data in Memstore?
>
> thanks,
>
> Jason
>
> On Wed, Sep 12, 2012 at 5:19 PM, Adrien Mogenet
> <ad...@gmail.com> wrote:
> > WAL is just there for recover. Reads will meet the Memstore on their read
> > path, that's how LSM Trees are working.
> >
> > On Wed, Sep 12, 2012 at 11:15 PM, Jason Huang <ja...@icare.com>
> wrote:
> >
> >> This might be a naive question but I am not able to find a good answer
> >> from searching online.
> >>
> >> The online guide mentioned that "Puts and Deletes are collected into
> >> an in-memory structure called the MemStore. Before the MemStore is
> >> update the changes are written to a Write Ahead Log (WAL) to enable
> >> recovery in case a server crashes. When it reaches a certain size the
> >> MemStore is flushed to disk into StoreFile."
> >>
> >> So, if an application tried to query a certain piece of data that
> >> hasn't been flushed to disk into StoreFile yet, where is HBase
> >> designed to get that piece of data? Is it going to the Region servers
> >> and tried to get the previous version of this data, or is it smart
> >> enough to go to the MemStore or WAL to get the most recent version of
> >> data?
> >>
> >> thanks!
> >>
> >> Jason
> >>
> >
> >
> > --
> > Adrien Mogenet
> > 06.59.16.64.22
> > http://www.mogenet.me
>
--
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me
Re: HBase 'Real-Time' reads?
Posted by Jason Huang <ja...@icare.com>.
So - I guess at the time of the query we don't know if the data is in
Memstore or in the RegionServer. In order to ensure we get the most
recent version of data, every Hbase Read query will first go to
Memstore and see if the data is there, and then go to RegionServers if
it couldn't find that data in Memstore?
thanks,
Jason
On Wed, Sep 12, 2012 at 5:19 PM, Adrien Mogenet
<ad...@gmail.com> wrote:
> WAL is just there for recover. Reads will meet the Memstore on their read
> path, that's how LSM Trees are working.
>
> On Wed, Sep 12, 2012 at 11:15 PM, Jason Huang <ja...@icare.com> wrote:
>
>> This might be a naive question but I am not able to find a good answer
>> from searching online.
>>
>> The online guide mentioned that "Puts and Deletes are collected into
>> an in-memory structure called the MemStore. Before the MemStore is
>> update the changes are written to a Write Ahead Log (WAL) to enable
>> recovery in case a server crashes. When it reaches a certain size the
>> MemStore is flushed to disk into StoreFile."
>>
>> So, if an application tried to query a certain piece of data that
>> hasn't been flushed to disk into StoreFile yet, where is HBase
>> designed to get that piece of data? Is it going to the Region servers
>> and tried to get the previous version of this data, or is it smart
>> enough to go to the MemStore or WAL to get the most recent version of
>> data?
>>
>> thanks!
>>
>> Jason
>>
>
>
> --
> Adrien Mogenet
> 06.59.16.64.22
> http://www.mogenet.me
Re: HBase 'Real-Time' reads?
Posted by Adrien Mogenet <ad...@gmail.com>.
WAL is just there for recover. Reads will meet the Memstore on their read
path, that's how LSM Trees are working.
On Wed, Sep 12, 2012 at 11:15 PM, Jason Huang <ja...@icare.com> wrote:
> This might be a naive question but I am not able to find a good answer
> from searching online.
>
> The online guide mentioned that "Puts and Deletes are collected into
> an in-memory structure called the MemStore. Before the MemStore is
> update the changes are written to a Write Ahead Log (WAL) to enable
> recovery in case a server crashes. When it reaches a certain size the
> MemStore is flushed to disk into StoreFile."
>
> So, if an application tried to query a certain piece of data that
> hasn't been flushed to disk into StoreFile yet, where is HBase
> designed to get that piece of data? Is it going to the Region servers
> and tried to get the previous version of this data, or is it smart
> enough to go to the MemStore or WAL to get the most recent version of
> data?
>
> thanks!
>
> Jason
>
--
Adrien Mogenet
06.59.16.64.22
http://www.mogenet.me