You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Hari Sreekumar <hs...@clickable.com> on 2011/02/20 15:38:58 UTC
Clarification regarding HBase reads
Hi,
I was going through the HBase architecture blog by Lars George (
http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html) and
I just wanted a clarification regarding how HBase reads data. The blog
mentions that :
Next the HRegionServer opens the region it creates a corresponding
HRegion object.
When the HRegion is "opened" it sets up a Store instance for each
HColumnFamily for every table as defined by the user beforehand. Each of
the Store instances can in turn have one or more StoreFile instances, which
are lightweight wrappers around the actual storage file called HFile. A
HRegion also has a MemStore and a HLog instance. We will now have a look at
how they work together but also where there are exceptions to the rule.
Does this mean that a store instance is opened for all tables present in
HBase irrespective of which table we are querying and for all
columnfamilies? Is this why I generally see people avoiding large number of
tables/large number of column families. If not, what is the reason for that?
Is it true at all that we should avoid too many tables/CFs ?
Thanks,
Hari
Re: Clarification regarding HBase reads
Posted by Lars George <la...@gmail.com>.
Hi Hari,
Thanks for bringing up that this is not entirely clear. Just to add to
what was said here already, this is all happening when a table is
enabled. This also happens when you restart HBase for example. So as
soon as you have a table open and you can see it in the UI then the
system has created all these objects for you. By the time you query
there are no extra resources opened or added, HBase has the files
open, the block index loaded and in memory. The actual objects are not
that much of a problem, it is mainly the issue Ted points out and also
file handles plus connection handlers (the xcievers). There is also
work going on (but is stalling and restarting few times already) to
make the DFSClient using Channels and NIO, doing away with the
resources requirements (while taking a slight performances hit).
This is an interesting topic though and there is a lot to be done it seems.
Lars
On Sun, Feb 20, 2011 at 3:38 PM, Hari Sreekumar
<hs...@clickable.com> wrote:
> Hi,
>
> I was going through the HBase architecture blog by Lars George (
> http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html) and
> I just wanted a clarification regarding how HBase reads data. The blog
> mentions that :
>
> Next the HRegionServer opens the region it creates a corresponding
> HRegion object.
> When the HRegion is "opened" it sets up a Store instance for each
> HColumnFamily for every table as defined by the user beforehand. Each of
> the Store instances can in turn have one or more StoreFile instances, which
> are lightweight wrappers around the actual storage file called HFile. A
> HRegion also has a MemStore and a HLog instance. We will now have a look at
> how they work together but also where there are exceptions to the rule.
>
> Does this mean that a store instance is opened for all tables present in
> HBase irrespective of which table we are querying and for all
> columnfamilies? Is this why I generally see people avoiding large number of
> tables/large number of column families. If not, what is the reason for that?
> Is it true at all that we should avoid too many tables/CFs ?
>
> Thanks,
> Hari
>
Re: Clarification regarding HBase reads
Posted by Chris Tarnas <cf...@email.com>.
We do our indexes by using an index family in the same table we are indexing and make sure that no index rowkey could possibly be a valid data rowkey. This does not guarantee they will be in the same transaction but it does allow you to batch your puts for both data and index together.
-chris
On Feb 20, 2011, at 9:49 PM, Hari Sreekumar wrote:
> All right, I understand the integrity cost is there because we don't get
> transactions in HBase over multiple tables. Thanks a lot for your time and
> help :)
>
> Hari
>
> On Mon, Feb 21, 2011 at 11:09 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> One particular region is for only one table. So the answer to your first
>> question is no.
>>
>> For your use case, you need to consider the cost of maintaining 4 index
>> tables (in terms of data integrity).
>> You should try to minimize the number of index tables.
>>
>> On Sun, Feb 20, 2011 at 7:50 PM, Hari Sreekumar <hsreekumar@clickable.com
>>> wrote:
>>
>>> So if I have 10 tables each with 2 families, I'd open up 20 stores
>> whenever
>>> I open a region for reading? Is it a problem to have too many tables.
>> e.g,
>>> if I have 1 big table and 4 indexing tables for the big table? Are there
>>> any
>>> potential issues with this?
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Sun, Feb 20, 2011 at 8:47 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>>>> Does this mean that a store instance is opened for all tables
>> present
>>> in
>>>>>> HBase irrespective of which table we are querying and for all
>>>>>> columnfamilies?
>>>> No. The blog says Store instance is for each family.
>>>>
>>>> You should generally avoid multiple column families. But we can help
>> you
>>>> analyze your use case.
>>>> If you read through https://issues.apache.org/jira/browse/HBASE-3149,
>>> you
>>>> would better understand current implementation.
>>>>
>>>> On Sun, Feb 20, 2011 at 6:38 AM, Hari Sreekumar <
>>> hsreekumar@clickable.com
>>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I was going through the HBase architecture blog by Lars George (
>>>>>
>> http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html)
>>>> and
>>>>> I just wanted a clarification regarding how HBase reads data. The
>> blog
>>>>> mentions that :
>>>>>
>>>>> Next the HRegionServer opens the region it creates a corresponding
>>>>> HRegion object.
>>>>> When the HRegion is "opened" it sets up a Store instance for each
>>>>> HColumnFamily for every table as defined by the user beforehand. Each
>>> of
>>>>> the Store instances can in turn have one or more StoreFile instances,
>>>> which
>>>>> are lightweight wrappers around the actual storage file called HFile.
>> A
>>>>> HRegion also has a MemStore and a HLog instance. We will now have a
>>> look
>>>> at
>>>>> how they work together but also where there are exceptions to the
>> rule.
>>>>>
>>>>> Does this mean that a store instance is opened for all tables present
>>> in
>>>>> HBase irrespective of which table we are querying and for all
>>>>> columnfamilies? Is this why I generally see people avoiding large
>>> number
>>>> of
>>>>> tables/large number of column families. If not, what is the reason
>> for
>>>>> that?
>>>>> Is it true at all that we should avoid too many tables/CFs ?
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>
>>>
>>
Re: Clarification regarding HBase reads
Posted by Hari Sreekumar <hs...@clickable.com>.
All right, I understand the integrity cost is there because we don't get
transactions in HBase over multiple tables. Thanks a lot for your time and
help :)
Hari
On Mon, Feb 21, 2011 at 11:09 AM, Ted Yu <yu...@gmail.com> wrote:
> One particular region is for only one table. So the answer to your first
> question is no.
>
> For your use case, you need to consider the cost of maintaining 4 index
> tables (in terms of data integrity).
> You should try to minimize the number of index tables.
>
> On Sun, Feb 20, 2011 at 7:50 PM, Hari Sreekumar <hsreekumar@clickable.com
> >wrote:
>
> > So if I have 10 tables each with 2 families, I'd open up 20 stores
> whenever
> > I open a region for reading? Is it a problem to have too many tables.
> e.g,
> > if I have 1 big table and 4 indexing tables for the big table? Are there
> > any
> > potential issues with this?
> >
> > Thanks,
> > Hari
> >
> > On Sun, Feb 20, 2011 at 8:47 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > >> Does this mean that a store instance is opened for all tables
> present
> > in
> > > >> HBase irrespective of which table we are querying and for all
> > > >> columnfamilies?
> > > No. The blog says Store instance is for each family.
> > >
> > > You should generally avoid multiple column families. But we can help
> you
> > > analyze your use case.
> > > If you read through https://issues.apache.org/jira/browse/HBASE-3149,
> > you
> > > would better understand current implementation.
> > >
> > > On Sun, Feb 20, 2011 at 6:38 AM, Hari Sreekumar <
> > hsreekumar@clickable.com
> > > >wrote:
> > >
> > > > Hi,
> > > >
> > > > I was going through the HBase architecture blog by Lars George (
> > > >
> http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html)
> > > and
> > > > I just wanted a clarification regarding how HBase reads data. The
> blog
> > > > mentions that :
> > > >
> > > > Next the HRegionServer opens the region it creates a corresponding
> > > > HRegion object.
> > > > When the HRegion is "opened" it sets up a Store instance for each
> > > > HColumnFamily for every table as defined by the user beforehand. Each
> > of
> > > > the Store instances can in turn have one or more StoreFile instances,
> > > which
> > > > are lightweight wrappers around the actual storage file called HFile.
> A
> > > > HRegion also has a MemStore and a HLog instance. We will now have a
> > look
> > > at
> > > > how they work together but also where there are exceptions to the
> rule.
> > > >
> > > > Does this mean that a store instance is opened for all tables present
> > in
> > > > HBase irrespective of which table we are querying and for all
> > > > columnfamilies? Is this why I generally see people avoiding large
> > number
> > > of
> > > > tables/large number of column families. If not, what is the reason
> for
> > > > that?
> > > > Is it true at all that we should avoid too many tables/CFs ?
> > > >
> > > > Thanks,
> > > > Hari
> > > >
> > >
> >
>
Re: Clarification regarding HBase reads
Posted by Ted Yu <yu...@gmail.com>.
One particular region is for only one table. So the answer to your first
question is no.
For your use case, you need to consider the cost of maintaining 4 index
tables (in terms of data integrity).
You should try to minimize the number of index tables.
On Sun, Feb 20, 2011 at 7:50 PM, Hari Sreekumar <hs...@clickable.com>wrote:
> So if I have 10 tables each with 2 families, I'd open up 20 stores whenever
> I open a region for reading? Is it a problem to have too many tables. e.g,
> if I have 1 big table and 4 indexing tables for the big table? Are there
> any
> potential issues with this?
>
> Thanks,
> Hari
>
> On Sun, Feb 20, 2011 at 8:47 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > >> Does this mean that a store instance is opened for all tables present
> in
> > >> HBase irrespective of which table we are querying and for all
> > >> columnfamilies?
> > No. The blog says Store instance is for each family.
> >
> > You should generally avoid multiple column families. But we can help you
> > analyze your use case.
> > If you read through https://issues.apache.org/jira/browse/HBASE-3149,
> you
> > would better understand current implementation.
> >
> > On Sun, Feb 20, 2011 at 6:38 AM, Hari Sreekumar <
> hsreekumar@clickable.com
> > >wrote:
> >
> > > Hi,
> > >
> > > I was going through the HBase architecture blog by Lars George (
> > > http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html)
> > and
> > > I just wanted a clarification regarding how HBase reads data. The blog
> > > mentions that :
> > >
> > > Next the HRegionServer opens the region it creates a corresponding
> > > HRegion object.
> > > When the HRegion is "opened" it sets up a Store instance for each
> > > HColumnFamily for every table as defined by the user beforehand. Each
> of
> > > the Store instances can in turn have one or more StoreFile instances,
> > which
> > > are lightweight wrappers around the actual storage file called HFile. A
> > > HRegion also has a MemStore and a HLog instance. We will now have a
> look
> > at
> > > how they work together but also where there are exceptions to the rule.
> > >
> > > Does this mean that a store instance is opened for all tables present
> in
> > > HBase irrespective of which table we are querying and for all
> > > columnfamilies? Is this why I generally see people avoiding large
> number
> > of
> > > tables/large number of column families. If not, what is the reason for
> > > that?
> > > Is it true at all that we should avoid too many tables/CFs ?
> > >
> > > Thanks,
> > > Hari
> > >
> >
>
Re: Clarification regarding HBase reads
Posted by Hari Sreekumar <hs...@clickable.com>.
So if I have 10 tables each with 2 families, I'd open up 20 stores whenever
I open a region for reading? Is it a problem to have too many tables. e.g,
if I have 1 big table and 4 indexing tables for the big table? Are there any
potential issues with this?
Thanks,
Hari
On Sun, Feb 20, 2011 at 8:47 PM, Ted Yu <yu...@gmail.com> wrote:
> >> Does this mean that a store instance is opened for all tables present in
> >> HBase irrespective of which table we are querying and for all
> >> columnfamilies?
> No. The blog says Store instance is for each family.
>
> You should generally avoid multiple column families. But we can help you
> analyze your use case.
> If you read through https://issues.apache.org/jira/browse/HBASE-3149, you
> would better understand current implementation.
>
> On Sun, Feb 20, 2011 at 6:38 AM, Hari Sreekumar <hsreekumar@clickable.com
> >wrote:
>
> > Hi,
> >
> > I was going through the HBase architecture blog by Lars George (
> > http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html)
> and
> > I just wanted a clarification regarding how HBase reads data. The blog
> > mentions that :
> >
> > Next the HRegionServer opens the region it creates a corresponding
> > HRegion object.
> > When the HRegion is "opened" it sets up a Store instance for each
> > HColumnFamily for every table as defined by the user beforehand. Each of
> > the Store instances can in turn have one or more StoreFile instances,
> which
> > are lightweight wrappers around the actual storage file called HFile. A
> > HRegion also has a MemStore and a HLog instance. We will now have a look
> at
> > how they work together but also where there are exceptions to the rule.
> >
> > Does this mean that a store instance is opened for all tables present in
> > HBase irrespective of which table we are querying and for all
> > columnfamilies? Is this why I generally see people avoiding large number
> of
> > tables/large number of column families. If not, what is the reason for
> > that?
> > Is it true at all that we should avoid too many tables/CFs ?
> >
> > Thanks,
> > Hari
> >
>
Re: Clarification regarding HBase reads
Posted by Ted Yu <yu...@gmail.com>.
>> Does this mean that a store instance is opened for all tables present in
>> HBase irrespective of which table we are querying and for all
>> columnfamilies?
No. The blog says Store instance is for each family.
You should generally avoid multiple column families. But we can help you
analyze your use case.
If you read through https://issues.apache.org/jira/browse/HBASE-3149, you
would better understand current implementation.
On Sun, Feb 20, 2011 at 6:38 AM, Hari Sreekumar <hs...@clickable.com>wrote:
> Hi,
>
> I was going through the HBase architecture blog by Lars George (
> http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html) and
> I just wanted a clarification regarding how HBase reads data. The blog
> mentions that :
>
> Next the HRegionServer opens the region it creates a corresponding
> HRegion object.
> When the HRegion is "opened" it sets up a Store instance for each
> HColumnFamily for every table as defined by the user beforehand. Each of
> the Store instances can in turn have one or more StoreFile instances, which
> are lightweight wrappers around the actual storage file called HFile. A
> HRegion also has a MemStore and a HLog instance. We will now have a look at
> how they work together but also where there are exceptions to the rule.
>
> Does this mean that a store instance is opened for all tables present in
> HBase irrespective of which table we are querying and for all
> columnfamilies? Is this why I generally see people avoiding large number of
> tables/large number of column families. If not, what is the reason for
> that?
> Is it true at all that we should avoid too many tables/CFs ?
>
> Thanks,
> Hari
>