import data from oracle to hbase: how much more space need to reserver?

["Daniel,Wu" <ha...@163.com>]

if a table in oracle has a size of 100T, and then put it into hbase, how much space normally will hbase take?

Re: import data from oracle to hbase: how much more space need to reserver?

[Jean-Daniel Cryans <jd...@apache.org>]

Yes, all the keys are stored along all the values.

J-D

On Mon, Aug 8, 2011 at 11:50 PM, Steinmaurer Thomas
<Th...@scch.at> wrote:
> Hello,
>
> this is an interesting link. If I get one thing right, is it really true
> that the row-key is stored per cell? If so, I guess we have to re-think
> our current row-key which is combined by three fields using 48 bytes
> total.
>
> Thanks,
> Thomas
>
> -----Original Message-----
> From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
> Jean-Daniel Cryans
> Sent: Montag, 08. August 2011 20:26
> To: user@hbase.apache.org
> Subject: Re: import data from oracle to hbase: how much more space need
> to reserver?
>
> Using compression is important but so is designing the keys to be as
> small as possible. That means using a family name of 1 character if
> possible (we use "d" for "default" usually here), but also taking time
> to cleverly design your row keys. OpenTSDB is a good example:
> http://opentsdb.net/schema.html
>
> Shaving off 5 bytes of a key on 10B cells is about 46GB in savings
> pre-compression, but that also counts when storing that in caches,
> compacting, etc.
>
> J-D
>
> On Sun, Aug 7, 2011 at 8:35 PM, alex zhang <al...@gmail.com>
> wrote:
>> it's depend on the orginal column number in oracle, and the data
>> length in the oracle.
>> the key in hbase include each row key+column name+timestape+TTL. so if
>
>> you have many column and the actually column value is very small, then
>
>> you the hbase table is much bigger then table in oracle.
>> So be sure use compress in hbase.
>>
>> I had a test table in oracle is 3G, after import to hbase, it's 28G.
>> with gz compress it about 4G, with lzo compress it's about 7G.
>>
>> Thanks
>> Zhang, Gui
>>
>> 2011/8/6 Daniel,Wu <ha...@163.com>
>>
>>> if a table in oracle has a size of 100T, and then put it into hbase,
>>> how much space normally will hbase take?
>>>
>>
>

RE: import data from oracle to hbase: how much more space need to reserver?

[Steinmaurer Thomas <Th...@scch.at>]

Hello,

this is an interesting link. If I get one thing right, is it really true
that the row-key is stored per cell? If so, I guess we have to re-think
our current row-key which is combined by three fields using 48 bytes
total.

Thanks,
Thomas

-----Original Message-----
From: jdcryans@gmail.com [mailto:jdcryans@gmail.com] On Behalf Of
Jean-Daniel Cryans
Sent: Montag, 08. August 2011 20:26
To: user@hbase.apache.org
Subject: Re: import data from oracle to hbase: how much more space need
to reserver?

Using compression is important but so is designing the keys to be as
small as possible. That means using a family name of 1 character if
possible (we use "d" for "default" usually here), but also taking time
to cleverly design your row keys. OpenTSDB is a good example:
http://opentsdb.net/schema.html

Shaving off 5 bytes of a key on 10B cells is about 46GB in savings
pre-compression, but that also counts when storing that in caches,
compacting, etc.

J-D

On Sun, Aug 7, 2011 at 8:35 PM, alex zhang <al...@gmail.com>
wrote:
> it's depend on the orginal column number in oracle, and the data 
> length in the oracle.
> the key in hbase include each row key+column name+timestape+TTL. so if

> you have many column and the actually column value is very small, then

> you the hbase table is much bigger then table in oracle.
> So be sure use compress in hbase.
>
> I had a test table in oracle is 3G, after import to hbase, it's 28G.
> with gz compress it about 4G, with lzo compress it's about 7G.
>
> Thanks
> Zhang, Gui
>
> 2011/8/6 Daniel,Wu <ha...@163.com>
>
>> if a table in oracle has a size of 100T, and then put it into hbase, 
>> how much space normally will hbase take?
>>
>

Re: import data from oracle to hbase: how much more space need to reserver?

[Jean-Daniel Cryans <jd...@apache.org>]

Using compression is important but so is designing the keys to be as
small as possible. That means using a family name of 1 character if
possible (we use "d" for "default" usually here), but also taking time
to cleverly design your row keys. OpenTSDB is a good example:
http://opentsdb.net/schema.html

Shaving off 5 bytes of a key on 10B cells is about 46GB in savings
pre-compression, but that also counts when storing that in caches,
compacting, etc.

J-D

On Sun, Aug 7, 2011 at 8:35 PM, alex zhang <al...@gmail.com> wrote:
> it's depend on the orginal column number in oracle, and the data length in
> the oracle.
> the key in hbase include each row key+column name+timestape+TTL. so if you
> have many column and the actually column value is very small, then you the
> hbase table is much bigger then table in oracle.
> So be sure use compress in hbase.
>
> I had a test table in oracle is 3G, after import to hbase, it's 28G.
> with gz compress it about 4G, with lzo compress it's about 7G.
>
> Thanks
> Zhang, Gui
>
> 2011/8/6 Daniel,Wu <ha...@163.com>
>
>> if a table in oracle has a size of 100T, and then put it into hbase, how
>> much space normally will hbase take?
>>
>

Re: import data from oracle to hbase: how much more space need to reserver?

[alex zhang <al...@gmail.com>]

it's depend on the orginal column number in oracle, and the data length in
the oracle.
the key in hbase include each row key+column name+timestape+TTL. so if you
have many column and the actually column value is very small, then you the
hbase table is much bigger then table in oracle.
So be sure use compress in hbase.

I had a test table in oracle is 3G, after import to hbase, it's 28G.
with gz compress it about 4G, with lzo compress it's about 7G.

Thanks
Zhang, Gui

2011/8/6 Daniel,Wu <ha...@163.com>

> if a table in oracle has a size of 100T, and then put it into hbase, how
> much space normally will hbase take?
>