You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Shuja Rehman <sh...@gmail.com> on 2011/06/13 15:29:04 UTC
Logical Division of Hbase Cluster
Hi All.
I have a question about logical division of hbase cluster, means dividing
the region servers w.r.t tables. This means that If I have let say 10
computers in cluster then table t1,t2 should be handled by c1,c3,c7 region
servers and all data related to these tables should be placed on these
servers and table t3,t6 should be handled by rest of servers. The goal is to
distribute specific data to specific region servers. SO is it possible to
do? If yes then how? and if No then what other approach can be use?
Thanks
--
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>
Re: Logical Division of Hbase Cluster
Posted by Shuja Rehman <sh...@gmail.com>.
On Tue, Jun 14, 2011 at 11:49 PM, Stack <st...@duboce.net> wrote:
> On Tue, Jun 14, 2011 at 1:41 AM, Shuja Rehman <sh...@gmail.com>
> wrote:
> > Well...There are couple of reasons
> >
> > 1- The data is coming from different regions of country and i want to
> > distribute the data w.r.t regions. e.g
> > RegionServer1-RegsionServer4 contain east region data only.
> > RegionServer2-RegionServer6 contain west region data only.
> >
>
> Can you do this with a table per region? Otherwise, prefix the key w/
> region. This won't be perfect in that the boundary won't be clean but
> perhaps sufficient?
>
> hum...i think table per region will not work as in future, there will be
data coming from different countries and if i use this strategy then it
means i need to create lot of tables for this which does not seem suitable
to me. I also think to prefix the key with region but i have many other
things in the key also e.g timestamp, tags and i am not sure how hbase
distribute the data to region servers in the presence of these things in the
key.
>
> > 2- The cluster is combination of different machines w.r.t hardware (RAM,
> > Processor Speed,Number of Cores). Some tables are access frequently and
> some
> > access for fewer time so i want to place the most accessed tables on the
> > machines with highest RAM and processing speeds. e.g create table1,
> colFam1
> > @10.10.10.2,10.10.10.3,10.10.10.10.4 (list of region servers)
> >
>
>
> In general, a heterogeneous cluster is probably going to cause you
> headache; rare has hbase run on a cluster that was not homogeneous so
> my guess is that you'll run into 'interesting' issues.
>
> Currently the levers are not exposed for manually balancing the
> cluster. Our balancer *should* do this for you factoring in the
> machine resources but currently it does not.
>
> One thing you could do is turn the balancer off and do the balancing
> yourself externally. You can move regions either via the shell or
> script.
> ok,i will look java api to figure out how to move region.
>
> > 4- I need to implement different priority scanning so the highest
> priority
> > query should be serve through good machines and this can be done if i
> able
> > to place the priority data on good machines. e.g if time= busy hours then
> > place data at good region servers.else if time=night then place data at
> > normal servers.
> >
> >
>
> HBase will never let you do this. It won't scale.
>
> St.Ack
>
--
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>
Re: Logical Division of Hbase Cluster
Posted by Stack <st...@duboce.net>.
On Tue, Jun 14, 2011 at 1:41 AM, Shuja Rehman <sh...@gmail.com> wrote:
> Well...There are couple of reasons
>
> 1- The data is coming from different regions of country and i want to
> distribute the data w.r.t regions. e.g
> RegionServer1-RegsionServer4 contain east region data only.
> RegionServer2-RegionServer6 contain west region data only.
>
Can you do this with a table per region? Otherwise, prefix the key w/
region. This won't be perfect in that the boundary won't be clean but
perhaps sufficient?
> 2- The cluster is combination of different machines w.r.t hardware (RAM,
> Processor Speed,Number of Cores). Some tables are access frequently and some
> access for fewer time so i want to place the most accessed tables on the
> machines with highest RAM and processing speeds. e.g create table1, colFam1
> @10.10.10.2,10.10.10.3,10.10.10.10.4 (list of region servers)
>
In general, a heterogeneous cluster is probably going to cause you
headache; rare has hbase run on a cluster that was not homogeneous so
my guess is that you'll run into 'interesting' issues.
Currently the levers are not exposed for manually balancing the
cluster. Our balancer *should* do this for you factoring in the
machine resources but currently it does not.
One thing you could do is turn the balancer off and do the balancing
yourself externally. You can move regions either via the shell or
script.
> 4- I need to implement different priority scanning so the highest priority
> query should be serve through good machines and this can be done if i able
> to place the priority data on good machines. e.g if time= busy hours then
> place data at good region servers.else if time=night then place data at
> normal servers.
>
>
HBase will never let you do this. It won't scale.
St.Ack
Re: Logical Division of Hbase Cluster
Posted by Shuja Rehman <sh...@gmail.com>.
Well...There are couple of reasons
1- The data is coming from different regions of country and i want to
distribute the data w.r.t regions. e.g
RegionServer1-RegsionServer4 contain east region data only.
RegionServer2-RegionServer6 contain west region data only.
2- The cluster is combination of different machines w.r.t hardware (RAM,
Processor Speed,Number of Cores). Some tables are access frequently and some
access for fewer time so i want to place the most accessed tables on the
machines with highest RAM and processing speeds. e.g create table1, colFam1
@10.10.10.2,10.10.10.3,10.10.10.10.4 (list of region servers)
4- I need to implement different priority scanning so the highest priority
query should be serve through good machines and this can be done if i able
to place the priority data on good machines. e.g if time= busy hours then
place data at good region servers.else if time=night then place data at
normal servers.
On Mon, Jun 13, 2011 at 9:38 PM, Stack <st...@duboce.net> wrote:
> On Mon, Jun 13, 2011 at 6:29 AM, Shuja Rehman <sh...@gmail.com>
> wrote:
> > I have a question about logical division of hbase cluster, means dividing
> > the region servers w.r.t tables. This means that If I have let say 10
> > computers in cluster then table t1,t2 should be handled by c1,c3,c7
> region
> > servers and all data related to these tables should be placed on these
> > servers and table t3,t6 should be handled by rest of servers. The goal is
> to
> > distribute specific data to specific region servers. SO is it possible to
> > do? If yes then how? and if No then what other approach can be use?
> >
>
> Not possible currently. I'll spare asking you why you would want to
> do this. You'd need to insert your own balancing algorithm.
> St.Ack
>
--
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>
Re: Logical Division of Hbase Cluster
Posted by Stack <st...@duboce.net>.
On Mon, Jun 13, 2011 at 6:29 AM, Shuja Rehman <sh...@gmail.com> wrote:
> I have a question about logical division of hbase cluster, means dividing
> the region servers w.r.t tables. This means that If I have let say 10
> computers in cluster then table t1,t2 should be handled by c1,c3,c7 region
> servers and all data related to these tables should be placed on these
> servers and table t3,t6 should be handled by rest of servers. The goal is to
> distribute specific data to specific region servers. SO is it possible to
> do? If yes then how? and if No then what other approach can be use?
>
Not possible currently. I'll spare asking you why you would want to
do this. You'd need to insert your own balancing algorithm.
St.Ack
Re: Logical Division of Hbase Cluster
Posted by Jean-Daniel Cryans <jd...@apache.org>.
You'd be fighting with HBase if you want one HBase instance and try
something like you described. You'd be better off splitting your
machines in multiple clusters...
J-D
On Mon, Jun 13, 2011 at 6:29 AM, Shuja Rehman <sh...@gmail.com> wrote:
> Hi All.
>
> I have a question about logical division of hbase cluster, means dividing
> the region servers w.r.t tables. This means that If I have let say 10
> computers in cluster then table t1,t2 should be handled by c1,c3,c7 region
> servers and all data related to these tables should be placed on these
> servers and table t3,t6 should be handled by rest of servers. The goal is to
> distribute specific data to specific region servers. SO is it possible to
> do? If yes then how? and if No then what other approach can be use?
>
> Thanks
> --
> Regards
> Shuja-ur-Rehman Baig
> <http://pk.linkedin.com/in/shujamughal>
>