You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Demai Ni <ni...@gmail.com> on 2016/10/20 21:51:59 UTC
setup two hbase instances on Mac?
hi, folks,
I am trying to setup a simple development environment on my Mac Book. And
like to have multiple instances of HBases, for some testing of replication,
backup. etc. And wondering there is any instruction to setup for multiple
instances(not the VM/container way).
Here is what I did so far. Install one through homebrew, and build another
one from source code.
To make my life easier, I setup the following alias:
/* first HBase, is installed by homebrew and using HDFS as storage, by
specify hbase.rootdir */
alias hDFSHBaseShell='/usr/local/bin/hbase shell'
alias
hstart='/usr/local/Cellar/hadoop/2.7.3/sbin/start-dfs.sh;/usr/local/Cellar/hadoop/2.7.3/sbin/start-yarn.sh'
alias
hstop='/usr/local/Cellar/hadoop/2.7.3/sbin/stop-yarn.sh;/usr/local/Cellar/hadoop/2.7.3/sbin/stop-dfs.sh'
alias startHDFSHBase='/usr/local/bin/start-hbase.sh'
alias stopHDFSHBase='/usr/local/bin/stop-hbase.sh'
/* 2nd HBase, is build locally from git clone, and using localfile system
as storage */
/* changed HBASE_PID_DIR in hbase-env.sh to avoid conflict with the first
instance
alias localHBaseShell='/Users/demai/hbase/bin/hbase shell'
alias startLocalHBase='/Users/demai/hbase/bin/start-hbase.sh'
alias stopLocalHBase='/Users/demai/hbase/bin/stop-hbase.sh'
Still, it is not enough as the start/stop hbase will only bring up on
instance, and bring it down regardless which stop-hbase.sh I used. I guess
more port configuration, like describe there :
http://blog.cloudera.com/blog/2013/07/guide-to-using-apache-hbase-ports/,
need to change in hbase-site.xml?
Before, I go down the manually port configuration route. Just wondering
whether anyone already done it? To save me some time of random shooting....
:-)
Many thanks. BTW, I did a bit google using 'multiple hbase instances', but
search results doesn't exactly match this environment.
Demai
Re: setup two hbase instances on Mac?
Posted by Demai Ni <ni...@gmail.com>.
Dima, thanks.
After a couple hours trying, I got it works now. So will try the docking
method another time.
Just in case someone else plan to do this kind of exercise in the future,
and to keep a closure for myself. Here are my changes:
/** *first HBase*, is installed by homebrew and using HDFS as storage, by
specify hbase.rootdir */
/* modify hbase-site.xml, by add
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>
*/
/* ports are all using default, for example Master UI is 160010 */
/* add following alias for convenience */
alias hDFSHBaseShell='/usr/local/bin/hbase shell'
alias hstart='/usr/local/Cellar/hadoop/2.7.3/sbin/start-dfs.
sh;/usr/local/Cellar/hadoop/2.7.3/sbin/start-yarn.sh'
alias hstop='/usr/local/Cellar/hadoop/2.7.3/sbin/stop-yarn.
sh;/usr/local/Cellar/hadoop/2.7.3/sbin/stop-dfs.sh'
alias startHDFSHBase='/usr/local/bin/start-hbase.sh'
alias stopHDFSHBase='/usr/local/bin/stop-hbase.sh'
/* *2nd HBase* instance, is build locally from git clone, and using
localfile system as storage */
/* modify hbase-env.sh to avoid conflict with the first instance, by
export HBASE_PID_DIR=/Users/demai/hbasetmp */
/* modify hbase-site.xml, to update the following dir and ports to avoid
conflict with the first. */
/*
<property>
<name>hbase.rootdir</name>
<value>file:///Users/demai/hbase_standalone</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/Users/demai/hbase_standalone/zookeeper</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2191</value>
</property>
<property>
<name>hbase.master.port</name>
<value>61000</value>
</property>
<property>
<name>hbase.master.info.port</name>
<value>61010</value>
</property>
<property>
<name>hbase.regionserver.port</name>
<value>61020</value>
</property>
<property>
<name>hbase.regionserver.info.port</name>
<value>61030</value>
</property>
*/
/* add following alias for convenience */
alias localHBaseShell='/Users/demai/hbase/bin/hbase shell'
alias startLocalHBase='/Users/demai/hbase/bin/start-hbase.sh'
alias stopLocalHBase='/Users/demai/hbase/bin/stop-hbase.sh'
On Thu, Oct 20, 2016 at 4:13 PM, Dima Spivak <di...@apache.org> wrote:
> So what you could essentially do is use the Apache HBase topology for
> clusterdock, running the clusterdock_run ./bin/start_cluster command twice,
> once for each cluster you want to start. I can provide specific command
> line arguments if you let me know which version of HBase you're hoping to
> use. The only pre-req would be Docker for Mac preinstalled on your machine.
>
> -Dima
>
> On Thu, Oct 20, 2016 at 4:03 PM, Demai Ni <ni...@gmail.com> wrote:
>
> > Actually I don't have a good reason of 'not use container', except that I
> > already have homebrew install hadoop and hbase on my laptop, hence like
> to
> > just keep using it.
> >
> > Thanks for the instruction through the blog. A quick question to clarify:
> > the blog is for multi-node cluster, instead of setting up two clusters
> on
> > the same machine? If I follow the instruction, which step should I
> revise?
> >
> > At this moment, I don't need the management/monitoring features from CDH,
> > and no need to install other components yet. Hence, looking for a simpler
> > way. I hope(maybe unrealistic) that I can get just a 2nd set for
> > configuration files, such as hbase-site.xml, hbase-env, etcs, then I will
> > be able to run two isolated(not resource wise) hbase instances.
> >
> > Again, thanks a lot for the tip. I may give it a try if simple
> > configuration change not available.
> >
> > Demai
> >
> > Demai
> >
> > On Thu, Oct 20, 2016 at 3:04 PM, Dima Spivak <di...@apache.org>
> > wrote:
> >
> > > Any reason to not use the container way via clusterdock [1]? I do
> > > replication testing on my Mac for this using it and have had pretty
> good
> > > results.
> > >
> > > 1.
> > > http://blog.cloudera.com/blog/2016/08/multi-node-clusters-
> > > with-cloudera-quickstart-for-docker/
> > >
> > > -Dima
> > >
> > > On Thu, Oct 20, 2016 at 2:51 PM, Demai Ni <ni...@gmail.com> wrote:
> > >
> > > > hi, folks,
> > > >
> > > > I am trying to setup a simple development environment on my Mac Book.
> > And
> > > > like to have multiple instances of HBases, for some testing of
> > > replication,
> > > > backup. etc. And wondering there is any instruction to setup for
> > multiple
> > > > instances(not the VM/container way).
> > > >
> > > > Here is what I did so far. Install one through homebrew, and build
> > > another
> > > > one from source code.
> > > >
> > > > To make my life easier, I setup the following alias:
> > > > /* first HBase, is installed by homebrew and using HDFS as storage,
> by
> > > > specify hbase.rootdir */
> > > > alias hDFSHBaseShell='/usr/local/bin/hbase shell'
> > > > alias
> > > > hstart='/usr/local/Cellar/hadoop/2.7.3/sbin/start-dfs.
> > > > sh;/usr/local/Cellar/hadoop/2.7.3/sbin/start-yarn.sh'
> > > > alias
> > > > hstop='/usr/local/Cellar/hadoop/2.7.3/sbin/stop-yarn.
> > > > sh;/usr/local/Cellar/hadoop/2.7.3/sbin/stop-dfs.sh'
> > > > alias startHDFSHBase='/usr/local/bin/start-hbase.sh'
> > > > alias stopHDFSHBase='/usr/local/bin/stop-hbase.sh'
> > > >
> > > >
> > > > /* 2nd HBase, is build locally from git clone, and using localfile
> > system
> > > > as storage */
> > > > /* changed HBASE_PID_DIR in hbase-env.sh to avoid conflict with the
> > first
> > > > instance
> > > > alias localHBaseShell='/Users/demai/hbase/bin/hbase shell'
> > > > alias startLocalHBase='/Users/demai/hbase/bin/start-hbase.sh'
> > > > alias stopLocalHBase='/Users/demai/hbase/bin/stop-hbase.sh'
> > > >
> > > > Still, it is not enough as the start/stop hbase will only bring up on
> > > > instance, and bring it down regardless which stop-hbase.sh I used. I
> > > guess
> > > > more port configuration, like describe there :
> > > > http://blog.cloudera.com/blog/2013/07/guide-to-using-apache-
> > hbase-ports/
> > > ,
> > > > need to change in hbase-site.xml?
> > > >
> > > > Before, I go down the manually port configuration route. Just
> wondering
> > > > whether anyone already done it? To save me some time of random
> > > shooting....
> > > > :-)
> > > >
> > > > Many thanks. BTW, I did a bit google using 'multiple hbase
> instances',
> > > but
> > > > search results doesn't exactly match this environment.
> > > >
> > > > Demai
> > > >
> > >
> >
>
Re: setup two hbase instances on Mac?
Posted by Dima Spivak <di...@apache.org>.
So what you could essentially do is use the Apache HBase topology for
clusterdock, running the clusterdock_run ./bin/start_cluster command twice,
once for each cluster you want to start. I can provide specific command
line arguments if you let me know which version of HBase you're hoping to
use. The only pre-req would be Docker for Mac preinstalled on your machine.
-Dima
On Thu, Oct 20, 2016 at 4:03 PM, Demai Ni <ni...@gmail.com> wrote:
> Actually I don't have a good reason of 'not use container', except that I
> already have homebrew install hadoop and hbase on my laptop, hence like to
> just keep using it.
>
> Thanks for the instruction through the blog. A quick question to clarify:
> the blog is for multi-node cluster, instead of setting up two clusters on
> the same machine? If I follow the instruction, which step should I revise?
>
> At this moment, I don't need the management/monitoring features from CDH,
> and no need to install other components yet. Hence, looking for a simpler
> way. I hope(maybe unrealistic) that I can get just a 2nd set for
> configuration files, such as hbase-site.xml, hbase-env, etcs, then I will
> be able to run two isolated(not resource wise) hbase instances.
>
> Again, thanks a lot for the tip. I may give it a try if simple
> configuration change not available.
>
> Demai
>
> Demai
>
> On Thu, Oct 20, 2016 at 3:04 PM, Dima Spivak <di...@apache.org>
> wrote:
>
> > Any reason to not use the container way via clusterdock [1]? I do
> > replication testing on my Mac for this using it and have had pretty good
> > results.
> >
> > 1.
> > http://blog.cloudera.com/blog/2016/08/multi-node-clusters-
> > with-cloudera-quickstart-for-docker/
> >
> > -Dima
> >
> > On Thu, Oct 20, 2016 at 2:51 PM, Demai Ni <ni...@gmail.com> wrote:
> >
> > > hi, folks,
> > >
> > > I am trying to setup a simple development environment on my Mac Book.
> And
> > > like to have multiple instances of HBases, for some testing of
> > replication,
> > > backup. etc. And wondering there is any instruction to setup for
> multiple
> > > instances(not the VM/container way).
> > >
> > > Here is what I did so far. Install one through homebrew, and build
> > another
> > > one from source code.
> > >
> > > To make my life easier, I setup the following alias:
> > > /* first HBase, is installed by homebrew and using HDFS as storage, by
> > > specify hbase.rootdir */
> > > alias hDFSHBaseShell='/usr/local/bin/hbase shell'
> > > alias
> > > hstart='/usr/local/Cellar/hadoop/2.7.3/sbin/start-dfs.
> > > sh;/usr/local/Cellar/hadoop/2.7.3/sbin/start-yarn.sh'
> > > alias
> > > hstop='/usr/local/Cellar/hadoop/2.7.3/sbin/stop-yarn.
> > > sh;/usr/local/Cellar/hadoop/2.7.3/sbin/stop-dfs.sh'
> > > alias startHDFSHBase='/usr/local/bin/start-hbase.sh'
> > > alias stopHDFSHBase='/usr/local/bin/stop-hbase.sh'
> > >
> > >
> > > /* 2nd HBase, is build locally from git clone, and using localfile
> system
> > > as storage */
> > > /* changed HBASE_PID_DIR in hbase-env.sh to avoid conflict with the
> first
> > > instance
> > > alias localHBaseShell='/Users/demai/hbase/bin/hbase shell'
> > > alias startLocalHBase='/Users/demai/hbase/bin/start-hbase.sh'
> > > alias stopLocalHBase='/Users/demai/hbase/bin/stop-hbase.sh'
> > >
> > > Still, it is not enough as the start/stop hbase will only bring up on
> > > instance, and bring it down regardless which stop-hbase.sh I used. I
> > guess
> > > more port configuration, like describe there :
> > > http://blog.cloudera.com/blog/2013/07/guide-to-using-apache-
> hbase-ports/
> > ,
> > > need to change in hbase-site.xml?
> > >
> > > Before, I go down the manually port configuration route. Just wondering
> > > whether anyone already done it? To save me some time of random
> > shooting....
> > > :-)
> > >
> > > Many thanks. BTW, I did a bit google using 'multiple hbase instances',
> > but
> > > search results doesn't exactly match this environment.
> > >
> > > Demai
> > >
> >
>
Re: setup two hbase instances on Mac?
Posted by Demai Ni <ni...@gmail.com>.
Actually I don't have a good reason of 'not use container', except that I
already have homebrew install hadoop and hbase on my laptop, hence like to
just keep using it.
Thanks for the instruction through the blog. A quick question to clarify:
the blog is for multi-node cluster, instead of setting up two clusters on
the same machine? If I follow the instruction, which step should I revise?
At this moment, I don't need the management/monitoring features from CDH,
and no need to install other components yet. Hence, looking for a simpler
way. I hope(maybe unrealistic) that I can get just a 2nd set for
configuration files, such as hbase-site.xml, hbase-env, etcs, then I will
be able to run two isolated(not resource wise) hbase instances.
Again, thanks a lot for the tip. I may give it a try if simple
configuration change not available.
Demai
Demai
On Thu, Oct 20, 2016 at 3:04 PM, Dima Spivak <di...@apache.org> wrote:
> Any reason to not use the container way via clusterdock [1]? I do
> replication testing on my Mac for this using it and have had pretty good
> results.
>
> 1.
> http://blog.cloudera.com/blog/2016/08/multi-node-clusters-
> with-cloudera-quickstart-for-docker/
>
> -Dima
>
> On Thu, Oct 20, 2016 at 2:51 PM, Demai Ni <ni...@gmail.com> wrote:
>
> > hi, folks,
> >
> > I am trying to setup a simple development environment on my Mac Book. And
> > like to have multiple instances of HBases, for some testing of
> replication,
> > backup. etc. And wondering there is any instruction to setup for multiple
> > instances(not the VM/container way).
> >
> > Here is what I did so far. Install one through homebrew, and build
> another
> > one from source code.
> >
> > To make my life easier, I setup the following alias:
> > /* first HBase, is installed by homebrew and using HDFS as storage, by
> > specify hbase.rootdir */
> > alias hDFSHBaseShell='/usr/local/bin/hbase shell'
> > alias
> > hstart='/usr/local/Cellar/hadoop/2.7.3/sbin/start-dfs.
> > sh;/usr/local/Cellar/hadoop/2.7.3/sbin/start-yarn.sh'
> > alias
> > hstop='/usr/local/Cellar/hadoop/2.7.3/sbin/stop-yarn.
> > sh;/usr/local/Cellar/hadoop/2.7.3/sbin/stop-dfs.sh'
> > alias startHDFSHBase='/usr/local/bin/start-hbase.sh'
> > alias stopHDFSHBase='/usr/local/bin/stop-hbase.sh'
> >
> >
> > /* 2nd HBase, is build locally from git clone, and using localfile system
> > as storage */
> > /* changed HBASE_PID_DIR in hbase-env.sh to avoid conflict with the first
> > instance
> > alias localHBaseShell='/Users/demai/hbase/bin/hbase shell'
> > alias startLocalHBase='/Users/demai/hbase/bin/start-hbase.sh'
> > alias stopLocalHBase='/Users/demai/hbase/bin/stop-hbase.sh'
> >
> > Still, it is not enough as the start/stop hbase will only bring up on
> > instance, and bring it down regardless which stop-hbase.sh I used. I
> guess
> > more port configuration, like describe there :
> > http://blog.cloudera.com/blog/2013/07/guide-to-using-apache-hbase-ports/
> ,
> > need to change in hbase-site.xml?
> >
> > Before, I go down the manually port configuration route. Just wondering
> > whether anyone already done it? To save me some time of random
> shooting....
> > :-)
> >
> > Many thanks. BTW, I did a bit google using 'multiple hbase instances',
> but
> > search results doesn't exactly match this environment.
> >
> > Demai
> >
>
Re: setup two hbase instances on Mac?
Posted by Dima Spivak <di...@apache.org>.
Any reason to not use the container way via clusterdock [1]? I do
replication testing on my Mac for this using it and have had pretty good
results.
1.
http://blog.cloudera.com/blog/2016/08/multi-node-clusters-with-cloudera-quickstart-for-docker/
-Dima
On Thu, Oct 20, 2016 at 2:51 PM, Demai Ni <ni...@gmail.com> wrote:
> hi, folks,
>
> I am trying to setup a simple development environment on my Mac Book. And
> like to have multiple instances of HBases, for some testing of replication,
> backup. etc. And wondering there is any instruction to setup for multiple
> instances(not the VM/container way).
>
> Here is what I did so far. Install one through homebrew, and build another
> one from source code.
>
> To make my life easier, I setup the following alias:
> /* first HBase, is installed by homebrew and using HDFS as storage, by
> specify hbase.rootdir */
> alias hDFSHBaseShell='/usr/local/bin/hbase shell'
> alias
> hstart='/usr/local/Cellar/hadoop/2.7.3/sbin/start-dfs.
> sh;/usr/local/Cellar/hadoop/2.7.3/sbin/start-yarn.sh'
> alias
> hstop='/usr/local/Cellar/hadoop/2.7.3/sbin/stop-yarn.
> sh;/usr/local/Cellar/hadoop/2.7.3/sbin/stop-dfs.sh'
> alias startHDFSHBase='/usr/local/bin/start-hbase.sh'
> alias stopHDFSHBase='/usr/local/bin/stop-hbase.sh'
>
>
> /* 2nd HBase, is build locally from git clone, and using localfile system
> as storage */
> /* changed HBASE_PID_DIR in hbase-env.sh to avoid conflict with the first
> instance
> alias localHBaseShell='/Users/demai/hbase/bin/hbase shell'
> alias startLocalHBase='/Users/demai/hbase/bin/start-hbase.sh'
> alias stopLocalHBase='/Users/demai/hbase/bin/stop-hbase.sh'
>
> Still, it is not enough as the start/stop hbase will only bring up on
> instance, and bring it down regardless which stop-hbase.sh I used. I guess
> more port configuration, like describe there :
> http://blog.cloudera.com/blog/2013/07/guide-to-using-apache-hbase-ports/,
> need to change in hbase-site.xml?
>
> Before, I go down the manually port configuration route. Just wondering
> whether anyone already done it? To save me some time of random shooting....
> :-)
>
> Many thanks. BTW, I did a bit google using 'multiple hbase instances', but
> search results doesn't exactly match this environment.
>
> Demai
>