You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Fuad Efendi <fu...@efendi.ca> on 2012/10/01 15:50:00 UTC
Column Qualifier space requirements
Hi,
Is column qualifier physically stored in a Cell? Or pointer to it? Do we
need to care about long size such as
my_very_long_qualifier:1
(size of a value is small in comparison to size of qualifier)
thanks
Re: Column Qualifier space requirements
Posted by Doug Meil <do...@explorysmedical.com>.
Hi there, there is a separate Store per ColumnFamily, which results in a
separate StoreFile.
http://hbase.apache.org/book.html#regions.arch
This section has a description of what the files look like on HDFSŠ
http://hbase.apache.org/book.html#trouble.namenode
On 10/3/12 10:35 AM, "Fuad Efendi" <fu...@efendi.ca> wrote:
>Hi Anoop,
>
>
>Thanks for the response!
>
>- I thought that each Column Family is associated with separate MapFile in
>Hadoop... but it was before 0.20... I found details at
>http://www.cloudera.com/blog/2012/06/hbase-io-hfile-input-output/
>
>-Fuad
>
>
>
RE: Column Qualifier space requirements
Posted by Fuad Efendi <fu...@efendi.ca>.
Hi Anoop,
Thanks for the response!
- I thought that each Column Family is associated with separate MapFile in
Hadoop... but it was before 0.20... I found details at
http://www.cloudera.com/blog/2012/06/hbase-io-hfile-input-output/
-Fuad
RE: Column Qualifier space requirements
Posted by Anoop Sam John <an...@huawei.com>.
>It means that in order to save space I need to use smallest Column
Qualifier (and sometimes it makes sense)...
Yes
>However, why Column Family (byte array) is repeated for each KeyValue? Is
it physically repeated for each cell?
Yes CF byte[] also physically stored in every cell (every KV).. At the storage level HBase deals with KVs and every KV is having a full identity of its.[rowkey+cf+qualifier+timestamp]
If you are working with 0.94 versions there is one optimization done wrt space saving. This is known as block level encoding. See the interface DataBlockEncoder and diff implementations available for it...
-Anoop-
________________________________________
From: Fuad Efendi [fuad@efendi.ca]
Sent: Tuesday, October 02, 2012 2:40 AM
To: user@hbase.apache.org
Subject: Re: Column Qualifier space requirements
Thanks;
It means that in order to save space I need to use smallest Column
Qualifier (and sometimes it makes sense)...
However, why Column Family (byte array) is repeated for each KeyValue? Is
it physically repeated for each cell?
>
>Hi there, take a look at the Hbase Refguide here...
>
>http://hbase.apache.org/book.html#regions.arch
>
>For this section...
>
>9.7.5.4. KeyValue
>
>
>
>
>
>>Hi,
>>
>>Is column qualifier physically stored in a Cell? Or pointer to it? Do we
>>need to care about long size such as
>>my_very_long_qualifier:1
>>
>>(size of a value is small in comparison to size of qualifierŠ)
>>
>>thanks
>>
>>
>
>
Re: Column Qualifier space requirements
Posted by Fuad Efendi <fu...@efendi.ca>.
Thanks;
It means that in order to save space I need to use smallest Column
Qualifier (and sometimes it makes sense)...
However, why Column Family (byte array) is repeated for each KeyValue? Is
it physically repeated for each cell?
>
>Hi there, take a look at the Hbase Refguide here...
>
>http://hbase.apache.org/book.html#regions.arch
>
>For this section...
>
>9.7.5.4. KeyValue
>
>
>
>
>
>>Hi,
>>
>>Is column qualifier physically stored in a Cell? Or pointer to it? Do we
>>need to care about long size such as
>>my_very_long_qualifier:1
>>
>>(size of a value is small in comparison to size of qualifierŠ)
>>
>>thanks
>>
>>
>
>
Re: Column Qualifier space requirements
Posted by Doug Meil <do...@explorysmedical.com>.
Hi there, take a look at the Hbase Refguide here...
http://hbase.apache.org/book.html#regions.arch
For this section...
9.7.5.4. KeyValue
On 10/1/12 9:50 AM, "Fuad Efendi" <fu...@efendi.ca> wrote:
>Hi,
>
>Is column qualifier physically stored in a Cell? Or pointer to it? Do we
>need to care about long size such as
>my_very_long_qualifier:1
>
>(size of a value is small in comparison to size of qualifierŠ)
>
>thanks
>
>
Re: Column Qualifier space requirements
Posted by Fuad Efendi <fu...@efendi.ca>.
I found excellent answer,
http://prafull-blog.blogspot.ca/2012/06/how-to-calculate-record-size-of-hba
se.html
Quote:
"So to calculate the record size:
Fixed part needed by KeyValue format = Key Length + Value Length + Row
Length + CF Length + Timestamp + Key Value = ( 4 + 4 + 2 + 1 + 8 + 1) = 20
Bytes
Variable part needed by KeyValue format = Row + Column Family + Column
Qualifier + Value"
Thanks,
Fuad
On 12-10-01 9:50 AM, "Fuad Efendi" <fu...@efendi.ca> wrote:
>Hi,
>
>Is column qualifier physically stored in a Cell? Or pointer to it? Do we
>need to care about long size such as
>my_very_long_qualifier:1
>
>(size of a value is small in comparison to size of qualifier?)
>
>thanks
>
>