You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Dennis Kubes <ku...@apache.org> on 2007/10/18 19:55:52 UTC

Getting started with HBase

I am just getting started with HBase.  Thinking about using it for 
future Nutch development.  Have successfully build with ant scripts.  In 
the src I see conf and bin directories similar to Hadoop.  But in the 
build I don't see those.  Is there a build that I can drop into  a 
directory that contains the conf, webapps, libs, etc. or do I need to 
pull all of that together myself?

Dennis

Re: Getting started with HBase

Posted by Dennis Kubes <ku...@apache.org>.

Michael Stack wrote:
> Andrzej Bialecki wrote:
>> That's excellent news - I just looked at the code, I think it would 
>> require only minimal tweaks to use it together with other Hadoop 
>> services running in "local" mode - e.g. it would be more convenient 
>> to have the MiniHBaseCluster (or its modified version, let's call it 
>> LocalHBaseCluster) handle the startup / shutdown itself, so that the 
>> user applications could assume that all necessary services are 
>> already running. I'm also going to check what is the startup time of 
>> MiniHBaseCluster.
>>
> MiniHBaseCluster already has a shutdown method.  Construction of a 
> MiniHBaseCluster starts up the cluster instance.  Would you like a 
> different startup mechanism Andrzej?  If so, what would you suggest?
>
> Otherwise, I"m thinking we'd move MiniHBaseCluster from src/test to 
> src/java so it makes it into the hadoop-*-hbase.jar renaming it 
> LocalHBaseCluster to follow the LocalJobRunner precedent.  We should 
> also make the hbase.master value default "local" rather than 
> "0.0.0.0:60000" and start a LocalHBaseCluster .  How does that sound?
>
> See the content of src/test/hbase-default.xml if you want to tweak 
> things to make startup faster.  The properties therein are the list to 
> tune if you want to make startup and shutdown snappier.

That sounds great to me.
>>> If the default mode -- i.e. if the hbase.master was set to 'local' 
>>> in hbase-default.xml -- was to run a MiniHBaseCluster instance, 
>>> would this suffice Andrzej?  Or do you need master and regionservers 
>>> talking to each via direct in-process method invocations rather than 
>>> over sockets as is done in "local" mapreduce?
>>>
>>
>> Direct in-process pseudo-protocol would be probably more efficient 
>> and it would reduce the number of sockets in use, but we could 
>> implement it as a future enhancement if needed. For now I'm happy 
>> with them using sockets.
>
> Agreed.  I do not think it would be hard having the master invoke 
> region server methods directly (and vice versa) rather than go via the 
> RPC but that we can be done later.
>
> St.Ack

I think the RPC method is fine for right now.  Good stuff.

Dennis Kubes

Re: Getting started with HBase

Posted by Michael Stack <st...@duboce.net>.
Thanks for fixing hudson lads.
St.Ack

Nigel Daley wrote:
> Ok, with Allen Wittenauer's help, I've got the patch process going 
> again.  Thanks Allen!
>
> Looks like a sendmail upgrade over the weekend put sendmail into 
> localhost mode.
>
> I've resubmitted to Hudson all the Jira's that are currently "Patch 
> Available".
>
> Cheers,
> Nige
>
> On Oct 23, 2007, at 11:26 PM, Nigel Daley wrote:
>
>> (moving to hadoop-dev)
>>
>> Hmmm, our Solaris zones machine doesn't seem to be accepting 
>> connections on port 25 (sendmail).  I've restarted sendmail to no 
>> avail.  I suspect this problem started when the system was rebooted 
>> on Oct 21 (Sunday) since there have been no test runs since Oct 20.  
>> I'll try to look at this more when I have time (I'm away this week).  
>> In the meantime, feel free to send me suggestions on what to try next 
>> (Allen W?).
>>
>> Cheers,
>> Nige
>>
>> On Oct 23, 2007, at 1:35 PM, Michael Stack wrote:
>>
>>> Andrzej Bialecki wrote:
>>>> Michael Stack wrote:
>>>>> ...
>>>>> Otherwise, I"m thinking we'd move MiniHBaseCluster from src/test 
>>>>> to src/java so it makes it into the hadoop-*-hbase.jar renaming it 
>>>>> LocalHBaseCluster to follow the LocalJobRunner precedent.  We 
>>>>> should also make the hbase.master value default "local" rather 
>>>>> than "0.0.0.0:60000" and start a LocalHBaseCluster .  How does 
>>>>> that sound?
>>>>
>>>> Sounds great, just what we need.
>>> Hudson for some reason has stopped running hadoop patch builds.  
>>> We're trying to figure why. Unfortunately, his obstinacy is in the 
>>> way of my committing the above (2084) pseudo 'local' support and the 
>>> hbase-running-out-of-build-dir patch for Dennis (2088).  Hopefully, 
>>> he'll soon come around.
>>>
>>> St.Ack
>>>
>>
>


RE: Getting started with HBase

Posted by Jim Kellerman <ji...@powerset.com>.
Gracias!

---
Jim Kellerman, Senior Engineer; Powerset
jim@powerset.com


> -----Original Message-----
> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
> Sent: Wednesday, October 24, 2007 5:35 PM
> To: hadoop-dev@lucene.apache.org
> Subject: Re: Getting started with HBase
>
> Ok, with Allen Wittenauer's help, I've got the patch process
> going again.  Thanks Allen!
>
> Looks like a sendmail upgrade over the weekend put sendmail
> into localhost mode.
>
> I've resubmitted to Hudson all the Jira's that are currently
> "Patch Available".
>
> Cheers,
> Nige
>
> On Oct 23, 2007, at 11:26 PM, Nigel Daley wrote:
>
> > (moving to hadoop-dev)
> >
> > Hmmm, our Solaris zones machine doesn't seem to be accepting
> > connections on port 25 (sendmail).  I've restarted sendmail to no
> > avail.  I suspect this problem started when the system was
> rebooted on
> > Oct 21 (Sunday) since there have been no test runs since
> Oct 20.  I'll
> > try to look at this more when I have time (I'm away this week).  In
> > the meantime, feel free to send me suggestions on what to try next
> > (Allen W?).
> >
> > Cheers,
> > Nige
> >
> > On Oct 23, 2007, at 1:35 PM, Michael Stack wrote:
> >
> >> Andrzej Bialecki wrote:
> >>> Michael Stack wrote:
> >>>> ...
> >>>> Otherwise, I"m thinking we'd move MiniHBaseCluster from
> src/test to
> >>>> src/java so it makes it into the hadoop-*-hbase.jar renaming it
> >>>> LocalHBaseCluster to follow the LocalJobRunner precedent.  We
> >>>> should also make the hbase.master value default "local"
> rather than
> >>>> "0.0.0.0:60000" and start a LocalHBaseCluster .  How does that
> >>>> sound?
> >>>
> >>> Sounds great, just what we need.
> >> Hudson for some reason has stopped running hadoop patch builds.
> >> We're trying to figure why. Unfortunately, his obstinacy is in the
> >> way of my committing the above (2084) pseudo 'local'
> support and the
> >> hbase-running-out-of-build-dir patch for Dennis (2088).
> >> Hopefully, he'll soon come around.
> >>
> >> St.Ack
> >>
> >
>
>

Re: Getting started with HBase

Posted by Nigel Daley <nd...@yahoo-inc.com>.
Ok, with Allen Wittenauer's help, I've got the patch process going  
again.  Thanks Allen!

Looks like a sendmail upgrade over the weekend put sendmail into  
localhost mode.

I've resubmitted to Hudson all the Jira's that are currently "Patch  
Available".

Cheers,
Nige

On Oct 23, 2007, at 11:26 PM, Nigel Daley wrote:

> (moving to hadoop-dev)
>
> Hmmm, our Solaris zones machine doesn't seem to be accepting  
> connections on port 25 (sendmail).  I've restarted sendmail to no  
> avail.  I suspect this problem started when the system was rebooted  
> on Oct 21 (Sunday) since there have been no test runs since Oct  
> 20.  I'll try to look at this more when I have time (I'm away this  
> week).  In the meantime, feel free to send me suggestions on what  
> to try next (Allen W?).
>
> Cheers,
> Nige
>
> On Oct 23, 2007, at 1:35 PM, Michael Stack wrote:
>
>> Andrzej Bialecki wrote:
>>> Michael Stack wrote:
>>>> ...
>>>> Otherwise, I"m thinking we'd move MiniHBaseCluster from src/test  
>>>> to src/java so it makes it into the hadoop-*-hbase.jar renaming  
>>>> it LocalHBaseCluster to follow the LocalJobRunner precedent.  We  
>>>> should also make the hbase.master value default "local" rather  
>>>> than "0.0.0.0:60000" and start a LocalHBaseCluster .  How does  
>>>> that sound?
>>>
>>> Sounds great, just what we need.
>> Hudson for some reason has stopped running hadoop patch builds.   
>> We're trying to figure why. Unfortunately, his obstinacy is in the  
>> way of my committing the above (2084) pseudo 'local' support and  
>> the hbase-running-out-of-build-dir patch for Dennis (2088).   
>> Hopefully, he'll soon come around.
>>
>> St.Ack
>>
>


Re: Getting started with HBase

Posted by Nigel Daley <nd...@yahoo-inc.com>.
(moving to hadoop-dev)

Hmmm, our Solaris zones machine doesn't seem to be accepting  
connections on port 25 (sendmail).  I've restarted sendmail to no  
avail.  I suspect this problem started when the system was rebooted  
on Oct 21 (Sunday) since there have been no test runs since Oct 20.   
I'll try to look at this more when I have time (I'm away this week).   
In the meantime, feel free to send me suggestions on what to try next  
(Allen W?).

Cheers,
Nige

On Oct 23, 2007, at 1:35 PM, Michael Stack wrote:

> Andrzej Bialecki wrote:
>> Michael Stack wrote:
>>> ...
>>> Otherwise, I"m thinking we'd move MiniHBaseCluster from src/test  
>>> to src/java so it makes it into the hadoop-*-hbase.jar renaming  
>>> it LocalHBaseCluster to follow the LocalJobRunner precedent.  We  
>>> should also make the hbase.master value default "local" rather  
>>> than "0.0.0.0:60000" and start a LocalHBaseCluster .  How does  
>>> that sound?
>>
>> Sounds great, just what we need.
> Hudson for some reason has stopped running hadoop patch builds.   
> We're trying to figure why. Unfortunately, his obstinacy is in the  
> way of my committing the above (2084) pseudo 'local' support and  
> the hbase-running-out-of-build-dir patch for Dennis (2088).   
> Hopefully, he'll soon come around.
>
> St.Ack
>


Re: Getting started with HBase

Posted by Michael Stack <st...@duboce.net>.
Andrzej Bialecki wrote:
> Michael Stack wrote:
>> ...
>> Otherwise, I"m thinking we'd move MiniHBaseCluster from src/test to 
>> src/java so it makes it into the hadoop-*-hbase.jar renaming it 
>> LocalHBaseCluster to follow the LocalJobRunner precedent.  We should 
>> also make the hbase.master value default "local" rather than 
>> "0.0.0.0:60000" and start a LocalHBaseCluster .  How does that sound?
>
> Sounds great, just what we need.
Hudson for some reason has stopped running hadoop patch builds.  We're 
trying to figure why. Unfortunately, his obstinacy is in the way of my 
committing the above (2084) pseudo 'local' support and the 
hbase-running-out-of-build-dir patch for Dennis (2088).  Hopefully, 
he'll soon come around.

St.Ack


Re: Getting started with HBase

Posted by Andrzej Bialecki <ab...@getopt.org>.
Michael Stack wrote:
> Andrzej Bialecki wrote:
>> That's excellent news - I just looked at the code, I think it would 
>> require only minimal tweaks to use it together with other Hadoop 
>> services running in "local" mode - e.g. it would be more convenient to 
>> have the MiniHBaseCluster (or its modified version, let's call it 
>> LocalHBaseCluster) handle the startup / shutdown itself, so that the 
>> user applications could assume that all necessary services are already 
>> running. I'm also going to check what is the startup time of 
>> MiniHBaseCluster.
>>
> MiniHBaseCluster already has a shutdown method.  Construction of a 
> MiniHBaseCluster starts up the cluster instance.  Would you like a 
> different startup mechanism Andrzej?  If so, what would you suggest?
> 
> Otherwise, I"m thinking we'd move MiniHBaseCluster from src/test to 
> src/java so it makes it into the hadoop-*-hbase.jar renaming it 
> LocalHBaseCluster to follow the LocalJobRunner precedent.  We should 
> also make the hbase.master value default "local" rather than 
> "0.0.0.0:60000" and start a LocalHBaseCluster .  How does that sound?

Sounds great, just what we need.

-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Re: Getting started with HBase

Posted by Michael Stack <st...@duboce.net>.
Andrzej Bialecki wrote:
> That's excellent news - I just looked at the code, I think it would 
> require only minimal tweaks to use it together with other Hadoop 
> services running in "local" mode - e.g. it would be more convenient to 
> have the MiniHBaseCluster (or its modified version, let's call it 
> LocalHBaseCluster) handle the startup / shutdown itself, so that the 
> user applications could assume that all necessary services are already 
> running. I'm also going to check what is the startup time of 
> MiniHBaseCluster.
>
MiniHBaseCluster already has a shutdown method.  Construction of a 
MiniHBaseCluster starts up the cluster instance.  Would you like a 
different startup mechanism Andrzej?  If so, what would you suggest?

Otherwise, I"m thinking we'd move MiniHBaseCluster from src/test to 
src/java so it makes it into the hadoop-*-hbase.jar renaming it 
LocalHBaseCluster to follow the LocalJobRunner precedent.  We should 
also make the hbase.master value default "local" rather than 
"0.0.0.0:60000" and start a LocalHBaseCluster .  How does that sound?

See the content of src/test/hbase-default.xml if you want to tweak 
things to make startup faster.  The properties therein are the list to 
tune if you want to make startup and shutdown snappier. 

>> If the default mode -- i.e. if the hbase.master was set to 'local' in 
>> hbase-default.xml -- was to run a MiniHBaseCluster instance, would 
>> this suffice Andrzej?  Or do you need master and regionservers 
>> talking to each via direct in-process method invocations rather than 
>> over sockets as is done in "local" mapreduce?
>>
>
> Direct in-process pseudo-protocol would be probably more efficient and 
> it would reduce the number of sockets in use, but we could implement 
> it as a future enhancement if needed. For now I'm happy with them 
> using sockets.

Agreed.  I do not think it would be hard having the master invoke region 
server methods directly (and vice versa) rather than go via the RPC but 
that we can be done later.

St.Ack

Re: Getting started with HBase

Posted by Andrzej Bialecki <ab...@getopt.org>.
Michael Stack wrote:
> Andrzej Bialecki wrote:
>> If I'm not mistaken, there is no way right now to use HBase in a 
>> "local" mode similar to the Hadoop "local" mode, where we don't have 
>> to start any daemons and all necessary infrastructure runs inside a 
>> single JVM. What would it take to implement such mode? Would it 
>> require big changes to the codebase?
> Checkout MinHBaseCluster.  Its used in the bulk of the hbase unit 
> tests.  It runs a master and a configurable amount of region servers 
> each to its own thread.  Its modeled on MiniDFSCluster. 

That's excellent news - I just looked at the code, I think it would 
require only minimal tweaks to use it together with other Hadoop 
services running in "local" mode - e.g. it would be more convenient to 
have the MiniHBaseCluster (or its modified version, let's call it 
LocalHBaseCluster) handle the startup / shutdown itself, so that the 
user applications could assume that all necessary services are already 
running. I'm also going to check what is the startup time of 
MiniHBaseCluster.

> If the default 
> mode -- i.e. if the hbase.master was set to 'local' in hbase-default.xml 
> -- was to run a MiniHBaseCluster instance, would this suffice Andrzej?  
> Or do you need master and regionservers talking to each via direct 
> in-process method invocations rather than over sockets as is done in 
> "local" mapreduce?
> 

Direct in-process pseudo-protocol would be probably more efficient and 
it would reduce the number of sockets in use, but we could implement it 
as a future enhancement if needed. For now I'm happy with them using 
sockets.

-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Re: Getting started with HBase

Posted by Michael Stack <st...@duboce.net>.
Andrzej Bialecki wrote:
> If I'm not mistaken, there is no way right now to use HBase in a 
> "local" mode similar to the Hadoop "local" mode, where we don't have 
> to start any daemons and all necessary infrastructure runs inside a 
> single JVM. What would it take to implement such mode? Would it 
> require big changes to the codebase?
Checkout MinHBaseCluster.  Its used in the bulk of the hbase unit 
tests.  It runs a master and a configurable amount of region servers 
each to its own thread.  Its modeled on MiniDFSCluster.  If the default 
mode -- i.e. if the hbase.master was set to 'local' in hbase-default.xml 
-- was to run a MiniHBaseCluster instance, would this suffice Andrzej?  
Or do you need master and regionservers talking to each via direct 
in-process method invocations rather than over sockets as is done in 
"local" mapreduce?

Thanks,
St.Ack

Re: Getting started with HBase

Posted by Andrzej Bialecki <ab...@getopt.org>.
Dennis Kubes wrote:
> I am just getting started with HBase.  Thinking about using it for 
> future Nutch development.  Have successfully build with ant scripts.  In 
> the src I see conf and bin directories similar to Hadoop.  But in the 
> build I don't see those.  Is there a build that I can drop into  a 
> directory that contains the conf, webapps, libs, etc. or do I need to 
> pull all of that together myself?

I'm also planning to start experimenting with HBase.

If I'm not mistaken, there is no way right now to use HBase in a "local" 
mode similar to the Hadoop "local" mode, where we don't have to start 
any daemons and all necessary infrastructure runs inside a single JVM. 
What would it take to implement such mode? Would it require big changes 
to the codebase?

The reason I'm asking is that we're considering using HBase in Nutch. 
However, Nutch is quite often used in small single machine installation, 
where it uses the "local" mode, i.e. LocalFileSystem and 
LocalJobTracker. This simplified mode of operation is attractive in case 
of such small installations, where the ease of use outweighs scalability 
and performance concerns.

-- 
Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com


Re: Getting started with HBase

Posted by Michael Stack <st...@duboce.net>.
It doesn't have the hadoop*hbase.jar but it has the hbase classes.  
Pattern for contribs seems to be that contrib jars gets built into the 
contrib directory (See product of the package build).  Presumption 
to-date  is that hbase is hosted inside of hadoop (The start scripts 
will look in dirs above for hadoop jars and libs, scripts and configs).

Have a go at the build script Dennis.  A patch is probably best way of 
you getting your point across.

Good stuff,
St.Ack


Dennis Kubes wrote:
> Yeah, except for that still doesn't have the 
> hadoop-0.16.0-dev-hbase.jar , would those be dropped into the hadoop 
> installation or the root hbase directory?  From this build it looks 
> like we are keeping it hbase separate from the hadoop build, kinda.  
> If we are keeping the HBase install separate then it would be best to 
> have a runnable HBase drop such as copy and go.  If you want I can 
> modify the ant script for this?
>
> Dennis Kubes
>
> Michael Stack wrote:
>> Should we be making a runnable hbase at 
>> $HADOOP_HOME/build/contrib/hbase Dennis?
>>
>> If you run the package target from $HADOOP_HOME/build.xml, under 
>> $HADOOP_HOME/build it makes a hadoop-X.X.X directory.  In its src, 
>> there is a contrib/hbase with the lib, bin, etc.  You can run hbase 
>> from here.
>>
>> Let us know and we'll fix...
>> St.Ack
>>
>>
>> Dennis Kubes wrote:
>>> I am just getting started with HBase.  Thinking about using it for 
>>> future Nutch development.  Have successfully build with ant 
>>> scripts.  In the src I see conf and bin directories similar to 
>>> Hadoop.  But in the build I don't see those.  Is there a build that 
>>> I can drop into  a directory that contains the conf, webapps, libs, 
>>> etc. or do I need to pull all of that together myself?
>>>
>>> Dennis
>>


Re: Getting started with HBase

Posted by Dennis Kubes <ku...@apache.org>.
Yeah, except for that still doesn't have the hadoop-0.16.0-dev-hbase.jar 
, would those be dropped into the hadoop installation or the root hbase 
directory?  From this build it looks like we are keeping it hbase 
separate from the hadoop build, kinda.  If we are keeping the HBase 
install separate then it would be best to have a runnable HBase drop 
such as copy and go.  If you want I can modify the ant script for this?

Dennis Kubes

Michael Stack wrote:
> Should we be making a runnable hbase at 
> $HADOOP_HOME/build/contrib/hbase Dennis?
>
> If you run the package target from $HADOOP_HOME/build.xml, under 
> $HADOOP_HOME/build it makes a hadoop-X.X.X directory.  In its src, 
> there is a contrib/hbase with the lib, bin, etc.  You can run hbase 
> from here.
>
> Let us know and we'll fix...
> St.Ack
>
>
> Dennis Kubes wrote:
>> I am just getting started with HBase.  Thinking about using it for 
>> future Nutch development.  Have successfully build with ant scripts.  
>> In the src I see conf and bin directories similar to Hadoop.  But in 
>> the build I don't see those.  Is there a build that I can drop into  
>> a directory that contains the conf, webapps, libs, etc. or do I need 
>> to pull all of that together myself?
>>
>> Dennis
>

Re: Getting started with HBase

Posted by Michael Stack <st...@duboce.net>.
Should we be making a runnable hbase at $HADOOP_HOME/build/contrib/hbase 
Dennis?

If you run the package target from $HADOOP_HOME/build.xml, under 
$HADOOP_HOME/build it makes a hadoop-X.X.X directory.  In its src, there 
is a contrib/hbase with the lib, bin, etc.  You can run hbase from here.

Let us know and we'll fix...
St.Ack


Dennis Kubes wrote:
> I am just getting started with HBase.  Thinking about using it for 
> future Nutch development.  Have successfully build with ant scripts.  
> In the src I see conf and bin directories similar to Hadoop.  But in 
> the build I don't see those.  Is there a build that I can drop into  a 
> directory that contains the conf, webapps, libs, etc. or do I need to 
> pull all of that together myself?
>
> Dennis