You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by matan <ma...@cloudaloe.org> on 2012/11/28 15:24:09 UTC
Column family names and data size on disk
Hi,
I am sort of wondering why does the Column Family name repeat inside the
HFile for every Key/Value pair. This repetition presumably implies that
column family names should be kept short and cryptic. Is it because an HFile
may contain Key/Value pairs for more than a single column family?
Just trying to improve my understanding of why and how HBase works... your
knowledgeable insight is most welcome.
Thanks,
Matan
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Column-family-names-and-data-size-on-disk-tp4034507.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Column family names and data size on disk
Posted by ramkrishna vasudevan <ra...@gmail.com>.
I see .. Thanks Stack..
Regards
Ram
On Thu, Nov 29, 2012 at 1:19 AM, yonghu <yo...@gmail.com> wrote:
> I like the illustration of Stack.
>
> regards!
>
> Yong
>
> On Wed, Nov 28, 2012 at 6:56 PM, Stack <st...@duboce.net> wrote:
> > On Wed, Nov 28, 2012 at 6:40 AM, matan <ma...@cloudaloe.org> wrote:
> >
> >> Why does the CF have to be in the HFile, isn't the entire HFile
> dedicated
> >> to
> >> just one CF to start with (I'm speaking at the HBase architecture level,
> >> trying to figure why it is working as like it is).
> >>
> >
> >
> > The CF is repeated in each key because it was thought that one day an
> hfile
> > could hold keyvalues from multiple CFs; e.g. if hbase implemented
> locality
> > groups.
> >
> > St.Ack
>
Re: Column family names and data size on disk
Posted by yonghu <yo...@gmail.com>.
I like the illustration of Stack.
regards!
Yong
On Wed, Nov 28, 2012 at 6:56 PM, Stack <st...@duboce.net> wrote:
> On Wed, Nov 28, 2012 at 6:40 AM, matan <ma...@cloudaloe.org> wrote:
>
>> Why does the CF have to be in the HFile, isn't the entire HFile dedicated
>> to
>> just one CF to start with (I'm speaking at the HBase architecture level,
>> trying to figure why it is working as like it is).
>>
>
>
> The CF is repeated in each key because it was thought that one day an hfile
> could hold keyvalues from multiple CFs; e.g. if hbase implemented locality
> groups.
>
> St.Ack
Re: Column family names and data size on disk
Posted by Stack <st...@duboce.net>.
On Wed, Nov 28, 2012 at 6:40 AM, matan <ma...@cloudaloe.org> wrote:
> Why does the CF have to be in the HFile, isn't the entire HFile dedicated
> to
> just one CF to start with (I'm speaking at the HBase architecture level,
> trying to figure why it is working as like it is).
>
The CF is repeated in each key because it was thought that one day an hfile
could hold keyvalues from multiple CFs; e.g. if hbase implemented locality
groups.
St.Ack
Re: Column family names and data size on disk
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Nice link .... Did not know about this.
Regards
Ram
On Wed, Nov 28, 2012 at 9:03 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Along with whatever Ram sir has said, you may also find this link useful :
> http://blog.cloudera.com/blog/2012/06/hbase-io-hfile-input-output/
>
> Regards,
> Mohammad Tariq
>
>
>
> On Wed, Nov 28, 2012 at 8:44 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > Also think of the keyvalue that is in the memstore.
> > The entire keyvalue is used for byte ordering.
> >
> > So in the memory if i have CF1 and CF2, by ordering Cf1 should be sorted
> > first. This also is a reason why Col family is part of the keyvalue
> which
> > when flushed into HFile goes with the same format.
> >
> > Regards
> > Ram
> >
> > On Wed, Nov 28, 2012 at 8:28 PM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> > > I can find only the JIRA id pertaining to it
> > >
> > > https://issues.apache.org/jira/browse/HBASE-4218.
> > >
> > > Basically what i can understand from the design is HBase is a mulit
> level
> > > keyvalue map
> > >
> > > {rowkey->CF->Cols->TimeStamp}>value
> > >
> > > Map<Rowkey,<Map<CF,Map<colQual,Map<TimeStamp, Value>>>
> > >
> > > So every cell has all the info(rowkey, cf, qual, ts and value).
> > > And ideally the encoding algo is basically to avoid such repetitions.
> > > Anyone needs to add on this? Pls feel to do so...
> > >
> > > Regards
> > > Ram
> > >
> > >
> > > On Wed, Nov 28, 2012 at 8:10 PM, matan <ma...@cloudaloe.org> wrote:
> > >
> > >> Thanks Ram,
> > >>
> > >> Why does the CF have to be in the HFile, isn't the entire HFile
> > dedicated
> > >> to
> > >> just one CF to start with (I'm speaking at the HBase architecture
> level,
> > >> trying to figure why it is working as like it is).
> > >>
> > >> That was my main interest in my question, but could you indicate a tad
> > >> more
> > >> what those encoding algorithms should be useful for or suggest a link
> > for
> > >> reading about them?
> > >>
> > >> Thanks,
> > >> Matan
> > >>
> > >>
> > >>
> > >> --
> > >> View this message in context:
> > >>
> >
> http://apache-hbase.679495.n3.nabble.com/Column-family-names-and-data-size-on-disk-tp4034507p4034509.html
> > >> Sent from the HBase User mailing list archive at Nabble.com.
> > >>
> > >
> > >
> >
>
Re: Column family names and data size on disk
Posted by Mohammad Tariq <do...@gmail.com>.
Along with whatever Ram sir has said, you may also find this link useful :
http://blog.cloudera.com/blog/2012/06/hbase-io-hfile-input-output/
Regards,
Mohammad Tariq
On Wed, Nov 28, 2012 at 8:44 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> Also think of the keyvalue that is in the memstore.
> The entire keyvalue is used for byte ordering.
>
> So in the memory if i have CF1 and CF2, by ordering Cf1 should be sorted
> first. This also is a reason why Col family is part of the keyvalue which
> when flushed into HFile goes with the same format.
>
> Regards
> Ram
>
> On Wed, Nov 28, 2012 at 8:28 PM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > I can find only the JIRA id pertaining to it
> >
> > https://issues.apache.org/jira/browse/HBASE-4218.
> >
> > Basically what i can understand from the design is HBase is a mulit level
> > keyvalue map
> >
> > {rowkey->CF->Cols->TimeStamp}>value
> >
> > Map<Rowkey,<Map<CF,Map<colQual,Map<TimeStamp, Value>>>
> >
> > So every cell has all the info(rowkey, cf, qual, ts and value).
> > And ideally the encoding algo is basically to avoid such repetitions.
> > Anyone needs to add on this? Pls feel to do so...
> >
> > Regards
> > Ram
> >
> >
> > On Wed, Nov 28, 2012 at 8:10 PM, matan <ma...@cloudaloe.org> wrote:
> >
> >> Thanks Ram,
> >>
> >> Why does the CF have to be in the HFile, isn't the entire HFile
> dedicated
> >> to
> >> just one CF to start with (I'm speaking at the HBase architecture level,
> >> trying to figure why it is working as like it is).
> >>
> >> That was my main interest in my question, but could you indicate a tad
> >> more
> >> what those encoding algorithms should be useful for or suggest a link
> for
> >> reading about them?
> >>
> >> Thanks,
> >> Matan
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://apache-hbase.679495.n3.nabble.com/Column-family-names-and-data-size-on-disk-tp4034507p4034509.html
> >> Sent from the HBase User mailing list archive at Nabble.com.
> >>
> >
> >
>
Re: Column family names and data size on disk
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Also think of the keyvalue that is in the memstore.
The entire keyvalue is used for byte ordering.
So in the memory if i have CF1 and CF2, by ordering Cf1 should be sorted
first. This also is a reason why Col family is part of the keyvalue which
when flushed into HFile goes with the same format.
Regards
Ram
On Wed, Nov 28, 2012 at 8:28 PM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> I can find only the JIRA id pertaining to it
>
> https://issues.apache.org/jira/browse/HBASE-4218.
>
> Basically what i can understand from the design is HBase is a mulit level
> keyvalue map
>
> {rowkey->CF->Cols->TimeStamp}>value
>
> Map<Rowkey,<Map<CF,Map<colQual,Map<TimeStamp, Value>>>
>
> So every cell has all the info(rowkey, cf, qual, ts and value).
> And ideally the encoding algo is basically to avoid such repetitions.
> Anyone needs to add on this? Pls feel to do so...
>
> Regards
> Ram
>
>
> On Wed, Nov 28, 2012 at 8:10 PM, matan <ma...@cloudaloe.org> wrote:
>
>> Thanks Ram,
>>
>> Why does the CF have to be in the HFile, isn't the entire HFile dedicated
>> to
>> just one CF to start with (I'm speaking at the HBase architecture level,
>> trying to figure why it is working as like it is).
>>
>> That was my main interest in my question, but could you indicate a tad
>> more
>> what those encoding algorithms should be useful for or suggest a link for
>> reading about them?
>>
>> Thanks,
>> Matan
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-hbase.679495.n3.nabble.com/Column-family-names-and-data-size-on-disk-tp4034507p4034509.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>
>
Re: Column family names and data size on disk
Posted by ramkrishna vasudevan <ra...@gmail.com>.
I can find only the JIRA id pertaining to it
https://issues.apache.org/jira/browse/HBASE-4218.
Basically what i can understand from the design is HBase is a mulit level
keyvalue map
{rowkey->CF->Cols->TimeStamp}>value
Map<Rowkey,<Map<CF,Map<colQual,Map<TimeStamp, Value>>>
So every cell has all the info(rowkey, cf, qual, ts and value).
And ideally the encoding algo is basically to avoid such repetitions.
Anyone needs to add on this? Pls feel to do so...
Regards
Ram
On Wed, Nov 28, 2012 at 8:10 PM, matan <ma...@cloudaloe.org> wrote:
> Thanks Ram,
>
> Why does the CF have to be in the HFile, isn't the entire HFile dedicated
> to
> just one CF to start with (I'm speaking at the HBase architecture level,
> trying to figure why it is working as like it is).
>
> That was my main interest in my question, but could you indicate a tad more
> what those encoding algorithms should be useful for or suggest a link for
> reading about them?
>
> Thanks,
> Matan
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Column-family-names-and-data-size-on-disk-tp4034507p4034509.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
Re: Column family names and data size on disk
Posted by matan <ma...@cloudaloe.org>.
Thanks Ram,
Why does the CF have to be in the HFile, isn't the entire HFile dedicated to
just one CF to start with (I'm speaking at the HBase architecture level,
trying to figure why it is working as like it is).
That was my main interest in my question, but could you indicate a tad more
what those encoding algorithms should be useful for or suggest a link for
reading about them?
Thanks,
Matan
--
View this message in context: http://apache-hbase.679495.n3.nabble.com/Column-family-names-and-data-size-on-disk-tp4034507p4034509.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: Column family names and data size on disk
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Hi Matan
Yes for every cell in the Hfile the rowkey, col family will be repeating.
That is because every cell is unique wrt to the rowkey, colfamily and the
qualifier.
To encode this we have some encoding algos available in 0.94 and above.
Try them !!!
Hope this helps.
Regards
Ram
On Wed, Nov 28, 2012 at 7:54 PM, matan <ma...@cloudaloe.org> wrote:
> Hi,
>
> I am sort of wondering why does the Column Family name repeat inside the
> HFile for every Key/Value pair. This repetition presumably implies that
> column family names should be kept short and cryptic. Is it because an
> HFile
> may contain Key/Value pairs for more than a single column family?
>
> Just trying to improve my understanding of why and how HBase works... your
> knowledgeable insight is most welcome.
>
> Thanks,
> Matan
>
>
>
> --
> View this message in context:
> http://apache-hbase.679495.n3.nabble.com/Column-family-names-and-data-size-on-disk-tp4034507.html
> Sent from the HBase User mailing list archive at Nabble.com.
>