Is TaskTracker required for hbase?

[Michał Podsiadłowski <po...@gmail.com>]

Hi all!

Can someone tell me if tasktracker and jobtracker are required by hbase for
running? We are trying to minimize memory consumption due to sharing
machines for dataNodes/regions servers with other applications.

Thanks,
Michael

Re: Is TaskTracker required for hbase?

[stack <st...@duboce.net>]

Hey Michael:

Combining regionserver and datanode daemons is an interesting idea.  An
initial cut shouldn't be too hard.  You'd have the two daemons communicate
via RPC as they do now.  For an example, look at hbase tests where we spin
up minidfscluster and a minihbasecluster all in the one JVM.

Integrating ZooKeeper makes less sense.  It still has System.exit calls
embedded.  Also, no matter how you slice it, its in the nature of the
zookeeper cluster to be orthogonal to the running of the hbase+datanode
cluster.

I am with Edward, both that its a cool idea, but also that this is not
something to be rushed.  I'd suggest that the  effort conflating processes
would be better spent on study of the running system understanding how they
interact in your environment, developing a necessary balance between all the
running parts.  We can help you out if you go this path.  If you make a
frankenstein of hbase + datanodes, while it'd be cute, we'll be less able to
assist especially when time is short.

Good luck Michael,
St.Ack




On Fri, Dec 11, 2009 at 6:47 AM, Edward Capriolo <ed...@gmail.com>wrote:

> If you do not have much time to experiment, you prob do not have time
> for this, because you are going to have to build your own code to run
> all the components in one jvm. If you are unfamiliar with the
> codebases this takes getting used to, also running the ant test
> targets take some time. so I can't immagine such a drastic recoding
> would save you time.
>
> On 12/11/09, Michał Podsiadłowski <po...@gmail.com> wrote:
> > 2009/12/11 Ryan Rawson <ry...@gmail.com>
> >
> >> They are not necessary.
> >>
> >> HBase only depends on HDFS for storage, so you will need datanode,
> >> namenode and the hbase processes.
> >>
> >>
> >>
> >>
> > Hi Ryan
> > Thanks for answer. This will make our lives easier.
> > I've got another question which I've posted yesterday about combining
> > datanode and hbase processes (including zookeeper) into single JVM
> process.
> > Would you be so kind and drop me a few words about this since your our
> hbase
> > guru. I really need to know how this can behave - we don't have much time
> > for experiments and any clues can occure indispensable for us.
> >
> > Thanks,
> > Michael
> >
>

Re: Is TaskTracker required for hbase?

[Edward Capriolo <ed...@gmail.com>]

If you do not have much time to experiment, you prob do not have time
for this, because you are going to have to build your own code to run
all the components in one jvm. If you are unfamiliar with the
codebases this takes getting used to, also running the ant test
targets take some time. so I can't immagine such a drastic recoding
would save you time.

On 12/11/09, Michał Podsiadłowski <po...@gmail.com> wrote:
> 2009/12/11 Ryan Rawson <ry...@gmail.com>
>
>> They are not necessary.
>>
>> HBase only depends on HDFS for storage, so you will need datanode,
>> namenode and the hbase processes.
>>
>>
>>
>>
> Hi Ryan
> Thanks for answer. This will make our lives easier.
> I've got another question which I've posted yesterday about combining
> datanode and hbase processes (including zookeeper) into single JVM process.
> Would you be so kind and drop me a few words about this since your our hbase
> guru. I really need to know how this can behave - we don't have much time
> for experiments and any clues can occure indispensable for us.
>
> Thanks,
> Michael
>

Re: Is TaskTracker required for hbase?

[Michał Podsiadłowski <po...@gmail.com>]

2009/12/11 Ryan Rawson <ry...@gmail.com>

> They are not necessary.
>
> HBase only depends on HDFS for storage, so you will need datanode,
> namenode and the hbase processes.
>
>
>
>
Hi Ryan
Thanks for answer. This will make our lives easier.
I've got another question which I've posted yesterday about combining
datanode and hbase processes (including zookeeper) into single JVM process.
Would you be so kind and drop me a few words about this since your our hbase
guru. I really need to know how this can behave - we don't have much time
for experiments and any clues can occure indispensable for us.

Thanks,
Michael

Re: Is TaskTracker required for hbase?

[Ryan Rawson <ry...@gmail.com>]

They are not necessary.

HBase only depends on HDFS for storage, so you will need datanode,
namenode and the hbase processes.



2009/12/11 Michał Podsiadłowski <po...@gmail.com>:
> Hi all!
>
> Can someone tell me if tasktracker and jobtracker are required by hbase for
> running? We are trying to minimize memory consumption due to sharing
> machines for dataNodes/regions servers with other applications.
>
> Thanks,
> Michael
>