You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Mohit Anchlia <mo...@gmail.com> on 2012/11/26 23:16:36 UTC
Configuration setup
I have a need to move hbas-site.xml to an external location. So in order to
do that I changed my configuration as shown below. But this doesn't seem to
be working. It picks up the file but I get error, seems like it's going to
the localhost. I checked hbase-site.xml in the directory and the zookeeper
nodes are correctly listed.
[11/26/2012 14:09:31,480] INFO apache.zookeeper.ClientCnxn
[[web-analytics-ci-1.0.0-SNAPSHOT].AsyncFlow.async2.02-SendThread(localhost.localdomain:2181)]():
Opening socket connection to server localhost.localdomain/127.0.0.1:2181
-----
changed from
HBaseConfiguration.create()
to
config = *new* Configuration();
config.addResource(*new* Path(*CONF_FILE_PROP_NAME*));
*log*.info("Config location picked from:" + prop);
Re: Configuration setup
Posted by Mohit Anchlia <mo...@gmail.com>.
I solved it by explicitly setting the property using setProperty method
On Thu, Nov 29, 2012 at 4:46 PM, Suraj Varma <sv...@gmail.com> wrote:
> The directory where hbase-site.xml is located should be in your client
> side classpath. Can you check if this is the case (dump your classpath
> right before you call to check if the directory is present).
>
> Here's the HBaseConfiguration.addHBaseResources code.
>
> http://javasourcecode.org/html/open-source/hbase/hbase-0.90.3/org/apache/hadoop/hbase/HBaseConfiguration.java.html#line.85
> public static Configuration addHbaseResources(Configuration conf) {
> conf.addResource("hbase-default.xml");
> conf.addResource("hbase-site.xml");
>
> checkDefaultsVersion(conf);
> checkForClusterFreeMemoryLimit(conf);
> return conf;
> }
>
> As you can see ... it expects hbase-site.xml to be in the classpath.
>
> If your file is located in some subdirectory of your classpath base,
> you would have to give the full path. Or use getResourceAsStream() to
> get it as an InputStream and then use your
> Configuration.addResource(InputStream) approach to load it.
>
> --Suraj
>
>
> On Mon, Nov 26, 2012 at 5:02 PM, Mohit Anchlia <mo...@gmail.com>
> wrote:
> > Thanks! This is the client code I was referring to. The below code
> doesn't
> > seem to work. Also I tried HBaseConfiguration.addHBaseResrouce and that
> > didn't work either. Is there any other way to make it configurable
> outside
> > the resource?
> >
> > On Mon, Nov 26, 2012 at 2:39 PM, Stack <st...@duboce.net> wrote:
> >
> >> On Mon, Nov 26, 2012 at 2:16 PM, Mohit Anchlia <mo...@gmail.com>
> >> wrote:
> >> > I have a need to move hbas-site.xml to an external location. So in
> order
> >> to
> >> > do that I changed my configuration as shown below. But this doesn't
> seem
> >> to
> >> > be working. It picks up the file but I get error, seems like it's
> going
> >> to
> >> > the localhost. I checked hbase-site.xml in the directory and the
> >> zookeeper
> >> > nodes are correctly listed.
> >> >
> >> >
> >> > [11/26/2012 14:09:31,480] INFO apache.zookeeper.ClientCnxn
> >> >
> >>
> [[web-analytics-ci-1.0.0-SNAPSHOT].AsyncFlow.async2.02-SendThread(localhost.localdomain:2181)]():
> >> > Opening socket connection to server localhost.localdomain/
> 127.0.0.1:2181
> >> >
> >> > -----
> >> >
> >> > changed from
> >> >
> >> > HBaseConfiguration.create()
> >> >
> >> > to
> >> >
> >> >
> >> > config = *new* Configuration();
> >> >
> >> > config.addResource(*new* Path(*CONF_FILE_PROP_NAME*));
> >> >
> >> > *log*.info("Config location picked from:" + prop);
> >>
> >>
> >> The above looks basically right but IIRC, this stuff can be tricky
> >> adding in new resources and making sure stuff is applied in order --
> >> and then there is 'final' configs that are applied after yours.
> >>
> >> You could try copying the hbase conf dir to wherever, amending it to
> >> suit your needs and then when starting hbase, add '--config
> >> ALTERNATE_CONF_DIR'.
> >>
> >> St.Ack
> >>
>
Re: Configuration setup
Posted by Suraj Varma <sv...@gmail.com>.
The directory where hbase-site.xml is located should be in your client
side classpath. Can you check if this is the case (dump your classpath
right before you call to check if the directory is present).
Here's the HBaseConfiguration.addHBaseResources code.
http://javasourcecode.org/html/open-source/hbase/hbase-0.90.3/org/apache/hadoop/hbase/HBaseConfiguration.java.html#line.85
public static Configuration addHbaseResources(Configuration conf) {
conf.addResource("hbase-default.xml");
conf.addResource("hbase-site.xml");
checkDefaultsVersion(conf);
checkForClusterFreeMemoryLimit(conf);
return conf;
}
As you can see ... it expects hbase-site.xml to be in the classpath.
If your file is located in some subdirectory of your classpath base,
you would have to give the full path. Or use getResourceAsStream() to
get it as an InputStream and then use your
Configuration.addResource(InputStream) approach to load it.
--Suraj
On Mon, Nov 26, 2012 at 5:02 PM, Mohit Anchlia <mo...@gmail.com> wrote:
> Thanks! This is the client code I was referring to. The below code doesn't
> seem to work. Also I tried HBaseConfiguration.addHBaseResrouce and that
> didn't work either. Is there any other way to make it configurable outside
> the resource?
>
> On Mon, Nov 26, 2012 at 2:39 PM, Stack <st...@duboce.net> wrote:
>
>> On Mon, Nov 26, 2012 at 2:16 PM, Mohit Anchlia <mo...@gmail.com>
>> wrote:
>> > I have a need to move hbas-site.xml to an external location. So in order
>> to
>> > do that I changed my configuration as shown below. But this doesn't seem
>> to
>> > be working. It picks up the file but I get error, seems like it's going
>> to
>> > the localhost. I checked hbase-site.xml in the directory and the
>> zookeeper
>> > nodes are correctly listed.
>> >
>> >
>> > [11/26/2012 14:09:31,480] INFO apache.zookeeper.ClientCnxn
>> >
>> [[web-analytics-ci-1.0.0-SNAPSHOT].AsyncFlow.async2.02-SendThread(localhost.localdomain:2181)]():
>> > Opening socket connection to server localhost.localdomain/127.0.0.1:2181
>> >
>> > -----
>> >
>> > changed from
>> >
>> > HBaseConfiguration.create()
>> >
>> > to
>> >
>> >
>> > config = *new* Configuration();
>> >
>> > config.addResource(*new* Path(*CONF_FILE_PROP_NAME*));
>> >
>> > *log*.info("Config location picked from:" + prop);
>>
>>
>> The above looks basically right but IIRC, this stuff can be tricky
>> adding in new resources and making sure stuff is applied in order --
>> and then there is 'final' configs that are applied after yours.
>>
>> You could try copying the hbase conf dir to wherever, amending it to
>> suit your needs and then when starting hbase, add '--config
>> ALTERNATE_CONF_DIR'.
>>
>> St.Ack
>>
Re: Configuration setup
Posted by Mohit Anchlia <mo...@gmail.com>.
Thanks! This is the client code I was referring to. The below code doesn't
seem to work. Also I tried HBaseConfiguration.addHBaseResrouce and that
didn't work either. Is there any other way to make it configurable outside
the resource?
On Mon, Nov 26, 2012 at 2:39 PM, Stack <st...@duboce.net> wrote:
> On Mon, Nov 26, 2012 at 2:16 PM, Mohit Anchlia <mo...@gmail.com>
> wrote:
> > I have a need to move hbas-site.xml to an external location. So in order
> to
> > do that I changed my configuration as shown below. But this doesn't seem
> to
> > be working. It picks up the file but I get error, seems like it's going
> to
> > the localhost. I checked hbase-site.xml in the directory and the
> zookeeper
> > nodes are correctly listed.
> >
> >
> > [11/26/2012 14:09:31,480] INFO apache.zookeeper.ClientCnxn
> >
> [[web-analytics-ci-1.0.0-SNAPSHOT].AsyncFlow.async2.02-SendThread(localhost.localdomain:2181)]():
> > Opening socket connection to server localhost.localdomain/127.0.0.1:2181
> >
> > -----
> >
> > changed from
> >
> > HBaseConfiguration.create()
> >
> > to
> >
> >
> > config = *new* Configuration();
> >
> > config.addResource(*new* Path(*CONF_FILE_PROP_NAME*));
> >
> > *log*.info("Config location picked from:" + prop);
>
>
> The above looks basically right but IIRC, this stuff can be tricky
> adding in new resources and making sure stuff is applied in order --
> and then there is 'final' configs that are applied after yours.
>
> You could try copying the hbase conf dir to wherever, amending it to
> suit your needs and then when starting hbase, add '--config
> ALTERNATE_CONF_DIR'.
>
> St.Ack
>
Re: Configuration setup
Posted by Stack <st...@duboce.net>.
On Mon, Nov 26, 2012 at 2:16 PM, Mohit Anchlia <mo...@gmail.com> wrote:
> I have a need to move hbas-site.xml to an external location. So in order to
> do that I changed my configuration as shown below. But this doesn't seem to
> be working. It picks up the file but I get error, seems like it's going to
> the localhost. I checked hbase-site.xml in the directory and the zookeeper
> nodes are correctly listed.
>
>
> [11/26/2012 14:09:31,480] INFO apache.zookeeper.ClientCnxn
> [[web-analytics-ci-1.0.0-SNAPSHOT].AsyncFlow.async2.02-SendThread(localhost.localdomain:2181)]():
> Opening socket connection to server localhost.localdomain/127.0.0.1:2181
>
> -----
>
> changed from
>
> HBaseConfiguration.create()
>
> to
>
>
> config = *new* Configuration();
>
> config.addResource(*new* Path(*CONF_FILE_PROP_NAME*));
>
> *log*.info("Config location picked from:" + prop);
The above looks basically right but IIRC, this stuff can be tricky
adding in new resources and making sure stuff is applied in order --
and then there is 'final' configs that are applied after yours.
You could try copying the hbase conf dir to wherever, amending it to
suit your needs and then when starting hbase, add '--config
ALTERNATE_CONF_DIR'.
St.Ack
Re: Question About Starting and Stopping HADOOP & HBASE Cluster in Secure Mode
Posted by Leonid Fedotov <lf...@hortonworks.com>.
AC,
scripts start-dfs.sh and start-mapred.sh is just wrappers for hadoop-daemon.sh commands.
All the security settings are in the configuration files, so same start procedure should work for both secure and unsecured modes.
Just make sure you have correct configuration files.
Thank you!
Sincerely,
Leonid Fedotov
On Nov 27, 2012, at 3:28 AM, ac@hsk.hk wrote:
> Hi,
>
> I currently use the following steps to start and stop HADOOP & HBASE cluster:
>
> 1) Without Kerberos Security
> (start zookeepers)
> start the cluster from Master:
> {$HADOOP_HOME}/bin/start-dfs.sh // one command start all servers
> {$HADOOP_HOME}/bin/start-mapred.sh
> {$HBASE_HOME}/bin/start-hbase.sh
>
> stop the cluster from Master:
> {$HBASE_HOME}/bin/stop-hbase.sh
> {$HADOOP_HOME}/bin/stop-mapred.sh
> {$HADOOP_HOME}/bin/stop-dfs.sh
> (stop zookeepers)
>
>
> 2) With Kerberos and in Secure Mode
> start the HADOOP Namenode
> {$HADOOP_HOME}/bin/hadoop-daemon.sh start namenode
> then for each datanode:
> {$HADOOP_HOME}/bin/hadoop-daemon.sh start datanode
> then HBASE Master:
> {$HBASE_HOME}/bin/hbase-daemon.sh start master (as root)
> then for each HBASE regionserver:
> {$HBASE_HOME}/bin/hbase-daemon.sh start regionserver
>
>
> QUESTION: As I can see from 2) there are more steps to start the entire cluster in secure node, are there any existing commands which simplify start/stop HADOOP & HBASE in Secure mode?
>
> Thanks
> ac
Question About Starting and Stopping HADOOP & HBASE Cluster in Secure Mode
Posted by "ac@hsk.hk" <ac...@hsk.hk>.
Hi,
I currently use the following steps to start and stop HADOOP & HBASE cluster:
1) Without Kerberos Security
(start zookeepers)
start the cluster from Master:
{$HADOOP_HOME}/bin/start-dfs.sh // one command start all servers
{$HADOOP_HOME}/bin/start-mapred.sh
{$HBASE_HOME}/bin/start-hbase.sh
stop the cluster from Master:
{$HBASE_HOME}/bin/stop-hbase.sh
{$HADOOP_HOME}/bin/stop-mapred.sh
{$HADOOP_HOME}/bin/stop-dfs.sh
(stop zookeepers)
2) With Kerberos and in Secure Mode
start the HADOOP Namenode
{$HADOOP_HOME}/bin/hadoop-daemon.sh start namenode
then for each datanode:
{$HADOOP_HOME}/bin/hadoop-daemon.sh start datanode
then HBASE Master:
{$HBASE_HOME}/bin/hbase-daemon.sh start master (as root)
then for each HBASE regionserver:
{$HBASE_HOME}/bin/hbase-daemon.sh start regionserver
QUESTION: As I can see from 2) there are more steps to start the entire cluster in secure node, are there any existing commands which simplify start/stop HADOOP & HBASE in Secure mode?
Thanks
ac