You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Wei Liu <we...@stellarloyalty.com> on 2014/08/20 00:06:38 UTC
Multiple column families vs Multiple tables
We are doing schema design for our application, One thing we are not so
clear about is multiple column families (more than 3, probably 5 - 8) vs
multiple tables. In our use case, we will have the same number of rows in
all these column families, but some column families may be modified more
often than others, and some column families will have more columns than
others (thousands vs several).
The reason we are thinking about multiple column families is that it
probably can give us better performance if we need to do a search with data
from multiple column families. For example, search for a row with value x
in column family A and with value Y in column family B.
On the other hand, we saw the following paragraph in the user guide which
is scary to us:
"HBase currently does not do well with anything above two or three column
families so keep the number of column families in your schema low.
Currently, flushing and compactions are done on a per Region basis so if
one column family is carrying the bulk of the data bringing on flushes, the
adjacent families will also be flushed though the amount of data they carry
is small. When many column families the flushing and compaction interaction
can make for a bunch of needless i/o loading (To be addressed by changing
flushing and compaction to work on a per column family basis). For more
information on compactions, see Section 9.7.6.7, “Compaction”
<http://hbase.apache.org/book.html#compaction>."
Can any one please shed some light on this topic? Thanks in advance.
Thanks,
Wei
Re: Multiple column families vs Multiple tables
Posted by Wei Liu <we...@stellarloyalty.com>.
Chutium, thanks for your advices. I will check out your links.
I sent the email to the wrong email address! Sorry for the spam.
Wei
On Tue, Aug 19, 2014 at 4:49 PM, chutium <te...@gmail.com> wrote:
> ö_ö you should send this message to hbase user list, not spark user
> list...
>
> but i can give you some personal advice about this, keep column families as
> few as possible!
>
> at least, use some prefix of column qualifier could also be an idea. but
> read performance may be worse for your use case like "search for a row with
> value x in column family A and with value Y in column family B".
>
> so it depends on which workload is important for you, if your use case is
> very read-heavy and you really want to use multi column families to hold a
> good read performance, you should try to disable region split, adjust
> compaction interval carefully, and so on.
>
> there is a good slide for this:
>
> http://photo.weibo.com/1431095941/wbphotos/large/mid/3735178188435939/pid/554cca85gw1eiloddlqa5j20or0ik77z
>
> more slides about hbase + coprocessor, hbase + hive and hbase + spark:
> http://www.weibo.com/1431095941/BeL90zozx
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Multiple-column-families-vs-Multiple-tables-tp12425p12439.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Multiple column families vs Multiple tables
Posted by chutium <te...@gmail.com>.
ö_ö you should send this message to hbase user list, not spark user list...
but i can give you some personal advice about this, keep column families as
few as possible!
at least, use some prefix of column qualifier could also be an idea. but
read performance may be worse for your use case like "search for a row with
value x in column family A and with value Y in column family B".
so it depends on which workload is important for you, if your use case is
very read-heavy and you really want to use multi column families to hold a
good read performance, you should try to disable region split, adjust
compaction interval carefully, and so on.
there is a good slide for this:
http://photo.weibo.com/1431095941/wbphotos/large/mid/3735178188435939/pid/554cca85gw1eiloddlqa5j20or0ik77z
more slides about hbase + coprocessor, hbase + hive and hbase + spark:
http://www.weibo.com/1431095941/BeL90zozx
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Multiple-column-families-vs-Multiple-tables-tp12425p12439.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Multiple column families vs Multiple tables
Posted by Ted Yu <yu...@gmail.com>.
bq. does not do well with anything above two or three column families
Current hbase releases, such as 0.98.x, would do better than the above.
5 column families should be accommodated.
Cheers
On Tue, Aug 19, 2014 at 3:06 PM, Wei Liu <we...@stellarloyalty.com> wrote:
> We are doing schema design for our application, One thing we are not so
> clear about is multiple column families (more than 3, probably 5 - 8) vs
> multiple tables. In our use case, we will have the same number of rows in
> all these column families, but some column families may be modified more
> often than others, and some column families will have more columns than
> others (thousands vs several).
>
> The reason we are thinking about multiple column families is that it
> probably can give us better performance if we need to do a search with data
> from multiple column families. For example, search for a row with value x
> in column family A and with value Y in column family B.
>
> On the other hand, we saw the following paragraph in the user guide which
> is scary to us:
> "HBase currently does not do well with anything above two or three column
> families so keep the number of column families in your schema low.
> Currently, flushing and compactions are done on a per Region basis so if
> one column family is carrying the bulk of the data bringing on flushes, the
> adjacent families will also be flushed though the amount of data they carry
> is small. When many column families the flushing and compaction interaction
> can make for a bunch of needless i/o loading (To be addressed by changing
> flushing and compaction to work on a per column family basis). For more
> information on compactions, see Section 9.7.6.7, “Compaction”
> <http://hbase.apache.org/book.html#compaction>."
>
> Can any one please shed some light on this topic? Thanks in advance.
>
> Thanks,
> Wei
>