You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by llpind <so...@hotmail.com> on 2009/05/13 00:03:23 UTC
HBase Data Model
Hello all,
Our company has been looking into Hadoop & HBase, and has decided to put up
a test cluster. I've got Hadoop (0.19.1) with HBase cluster up and running
on 4 boxes. Currently we store our data in an Oracle database; I'd like
ideas on how I can model a specific set of tables into HBase.
The current project has two tables with a many to many relationship. Need a
way to go back and forth between these tables (e.g. given a record of tableA
grab many records from tableB. and vice-versa). I would like your opinion
on how I would go about designing a model for this in HBase. I'm assuming
it would be most efficient in a single HBase table. Thanks.
--
View this message in context: http://www.nabble.com/HBase-Data-Model-tp23511426p23511426.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase Data Model
Posted by stack <st...@duboce.net>.
Hopefully we'll have in-memory tables well before then.
St.Ack
On Wed, May 13, 2009 at 4:37 PM, llpind <so...@hotmail.com> wrote:
>
> Thanks Amandeep for your thoughts.
>
> I mentioned the million range simply for this particular test case. Once
> we
> are up and running and satisfied with the performance, we will start
> scaling
> towards the target system. The target system will be somewhere between
> 2-20
> terabytes (As far as I know). Although, the system will still be in the
> simple data model form I described above.
>
> --
> View this message in context:
> http://www.nabble.com/HBase-Data-Model-tp23511426p23531880.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
Re: HBase Data Model
Posted by llpind <so...@hotmail.com>.
Thanks Amandeep for your thoughts.
I mentioned the million range simply for this particular test case. Once we
are up and running and satisfied with the performance, we will start scaling
towards the target system. The target system will be somewhere between 2-20
terabytes (As far as I know). Although, the system will still be in the
simple data model form I described above.
--
View this message in context: http://www.nabble.com/HBase-Data-Model-tp23511426p23531880.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase Data Model
Posted by Amandeep Khurana <am...@gmail.com>.
For a few million records like this, hbase isnt the optimal solution in all
probability. It might work better to store them in memory instead. You'd be
good with 4GB RAM for this kind of a graph and for persistent storage, you
can simply dump it into a log file or an xml, which can be read if the
system crashes. Or you can serialize the data and store in Berkeley DB or
some column oriented data store. Hbase would be useful only if you are
talking of much bigger data sets. My data set, which contains about 20
million records is still small and probably just on the threshold of what
hbase might be optimal for (debatable - i dont have exact numbers).
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Wed, May 13, 2009 at 1:43 PM, Amandeep Khurana <am...@gmail.com> wrote:
> I store an entire graph in a single hbase table. This has essentially
> many-many relationships. The model is similar to the one you described in
> your mail. There are multiple column families and for every "type" of data
> point (row id), different set of families contain data. In your design it
> seems that a single column family will be filled only for a particular type
> of data point, which makes it unnecessary to store in a single table.
> Infact, you might be better off not using hbase for it. In my setup, a
> column family can contain data for different data points... So that makes it
> like a giant sparse matrix of some sort (not exactly though).
>
>
> Amandeep Khurana
> Computer Science Graduate Student
> University of California, Santa Cruz
>
>
>
> On Wed, May 13, 2009 at 12:24 PM, llpind <so...@hotmail.com> wrote:
>
>>
>> Thanks. Thats cool, I'm interested in indexes. Here is a classic
>> student/course example:
>>
>> RDBMS
>>
>> TBL_STUDENT: student_id, student_name, student_address
>> TBL_COURSES: course_id, student_id, course_type
>> TBL_COURSE_TYPES: course_type, course_desc
>>
>> 1st shot at HBase (1 HBase table):
>>
>> Key: ST:<student_id> example: ST:4423
>> Column Family: course
>> CoulmnEntries: course:<Type>:<Value> example: course:Math:Jon
>> Notice that each entity will be a new column and there is not really any
>> ‘value’ in the column. The column itself is holding the valuable
>> information.
>>
>> Key: CS:<Type>:<Value> example: CS:Math:Phil
>> Column Family: student
>> Column Entries: student:<student_id> example: student:4423 or student:5656
>>
>> Key: VL:<Value> example: VL:John
>> Column Family: type
>> Column Entries: type:<course> example: type:Math or type:Science
>>
>> Key: TP:<Type> example: TP:Math
>> Column Family: none
>> Column Entries: none
>>
>>
>> Assume potentially millions+ students & courses (~100s types). common
>> queries:
>>
>> Given a student name -> list all courses
>> Given a course -> list all students
>> Given a course -> list all types
>> list all available course types
>>
>> Open to ideas on alternative designs. Thanks.
>> --
>> View this message in context:
>> http://www.nabble.com/HBase-Data-Model-tp23511426p23528345.html
>> Sent from the HBase User mailing list archive at Nabble.com.
>>
>>
>
Re: HBase Data Model
Posted by Amandeep Khurana <am...@gmail.com>.
I store an entire graph in a single hbase table. This has essentially
many-many relationships. The model is similar to the one you described in
your mail. There are multiple column families and for every "type" of data
point (row id), different set of families contain data. In your design it
seems that a single column family will be filled only for a particular type
of data point, which makes it unnecessary to store in a single table.
Infact, you might be better off not using hbase for it. In my setup, a
column family can contain data for different data points... So that makes it
like a giant sparse matrix of some sort (not exactly though).
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Wed, May 13, 2009 at 12:24 PM, llpind <so...@hotmail.com> wrote:
>
> Thanks. Thats cool, I'm interested in indexes. Here is a classic
> student/course example:
>
> RDBMS
>
> TBL_STUDENT: student_id, student_name, student_address
> TBL_COURSES: course_id, student_id, course_type
> TBL_COURSE_TYPES: course_type, course_desc
>
> 1st shot at HBase (1 HBase table):
>
> Key: ST:<student_id> example: ST:4423
> Column Family: course
> CoulmnEntries: course:<Type>:<Value> example: course:Math:Jon
> Notice that each entity will be a new column and there is not really any
> ‘value’ in the column. The column itself is holding the valuable
> information.
>
> Key: CS:<Type>:<Value> example: CS:Math:Phil
> Column Family: student
> Column Entries: student:<student_id> example: student:4423 or student:5656
>
> Key: VL:<Value> example: VL:John
> Column Family: type
> Column Entries: type:<course> example: type:Math or type:Science
>
> Key: TP:<Type> example: TP:Math
> Column Family: none
> Column Entries: none
>
>
> Assume potentially millions+ students & courses (~100s types). common
> queries:
>
> Given a student name -> list all courses
> Given a course -> list all students
> Given a course -> list all types
> list all available course types
>
> Open to ideas on alternative designs. Thanks.
> --
> View this message in context:
> http://www.nabble.com/HBase-Data-Model-tp23511426p23528345.html
> Sent from the HBase User mailing list archive at Nabble.com.
>
>
Re: HBase Data Model
Posted by llpind <so...@hotmail.com>.
Thanks. Thats cool, I'm interested in indexes. Here is a classic
student/course example:
RDBMS
TBL_STUDENT: student_id, student_name, student_address
TBL_COURSES: course_id, student_id, course_type
TBL_COURSE_TYPES: course_type, course_desc
1st shot at HBase (1 HBase table):
Key: ST:<student_id> example: ST:4423
Column Family: course
CoulmnEntries: course:<Type>:<Value> example: course:Math:Jon
Notice that each entity will be a new column and there is not really any
‘value’ in the column. The column itself is holding the valuable
information.
Key: CS:<Type>:<Value> example: CS:Math:Phil
Column Family: student
Column Entries: student:<student_id> example: student:4423 or student:5656
Key: VL:<Value> example: VL:John
Column Family: type
Column Entries: type:<course> example: type:Math or type:Science
Key: TP:<Type> example: TP:Math
Column Family: none
Column Entries: none
Assume potentially millions+ students & courses (~100s types). common
queries:
Given a student name -> list all courses
Given a course -> list all students
Given a course -> list all types
list all available course types
Open to ideas on alternative designs. Thanks.
--
View this message in context: http://www.nabble.com/HBase-Data-Model-tp23511426p23528345.html
Sent from the HBase User mailing list archive at Nabble.com.
Re: HBase Data Model
Posted by Bob Schulze <b....@ecircle.com>.
llpind,
investigate first how you usually access this data. As HBase works best
when accessing data records through a so-called unique row key, it may
change your whole application design.
Many people also put special indexes on different fields of a Hbase
table, just to figure out the relevant row key of a data set as a first
step in a query.
Would need more details to give better advice.
Bob
llpind schrieb:
> Hello all,
>
> Our company has been looking into Hadoop & HBase, and has decided to put up
> a test cluster. I've got Hadoop (0.19.1) with HBase cluster up and running
> on 4 boxes. Currently we store our data in an Oracle database; I'd like
> ideas on how I can model a specific set of tables into HBase.
>
> The current project has two tables with a many to many relationship. Need a
> way to go back and forth between these tables (e.g. given a record of tableA
> grab many records from tableB. and vice-versa). I would like your opinion
> on how I would go about designing a model for this in HBase. I'm assuming
> it would be most efficient in a single HBase table. Thanks.
>