You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by kaveh minooie <ka...@plutoz.com> on 2013/02/22 20:15:49 UTC
does hbase master need to be a hadoop datanode as well?
Hi everyone,
I had a quick question. do I need to be running datanode on the server
that is running hbase master or datanode should be running only on
servers that are running as regionserver?
thanks,
--
Kaveh Minooie
Re: does hbase master need to be a hadoop datanode as well?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hi Harish,
HBase still run ZooKeeper even in standalone mode:
http://hbase.apache.org/book/standalone_dist.html
So you still can face same kind of issues. But since it's not using Hadoop,
there is more memory available for you HBase, which might reduce GCs
"issues".
JM
2013/2/22 Harish Krishnan <ha...@gmail.com>
> So in case of a stand alone set up, how come we are not experiencing those
> issues?
> Is it because of the limited data set that we are operating on?
>
> Thanks & Regards,
> Harish.T.K
>
>
> On Fri, Feb 22, 2013 at 1:19 PM, Harsh J <ha...@cloudera.com> wrote:
>
> > While this general guideline of not requiring a DN on HM is correct, I
> > just wanted to point out that the HM is hardly high on memory
> > usage/requirements (as possibly stated below), as all it does is
> > maintain the cluster and not serve nor hold any client-useful data or
> > frequent requests.
> >
> > On Sat, Feb 23, 2013 at 1:13 AM, Mohammad Tariq <do...@gmail.com>
> > wrote:
> > > Hello Kaven,
> > >
> > > It's not all necessary to run HM on server running DN. Actually
> > it's
> > > not advisable to do so, as HM has high memory requirements and running
> > > other daemons on the same machine may lead you to problems. The basic
> > > cluster setup, which people normally use looks somewhat like this :
> > >
> > > NN+JT(Hadoop master)
> > > SNN
> > > HM(Hbase master)
> > > DN+TT+RS
> > >
> > >
> > > Warm Regards,
> > > Tariq
> > > https://mtariq.jux.com/
> > > cloudfront.blogspot.com
> > >
> > >
> > > On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com>
> > wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> I had a quick question. do I need to be running datanode on the server
> > >> that is running hbase master or datanode should be running only on
> > servers
> > >> that are running as regionserver?
> > >>
> > >> thanks,
> > >> --
> > >> Kaveh Minooie
> > >>
> >
> >
> >
> > --
> > Harsh J
> >
>
Re: does hbase master need to be a hadoop datanode as well?
Posted by Harish Krishnan <ha...@gmail.com>.
So in case of a stand alone set up, how come we are not experiencing those
issues?
Is it because of the limited data set that we are operating on?
Thanks & Regards,
Harish.T.K
On Fri, Feb 22, 2013 at 1:19 PM, Harsh J <ha...@cloudera.com> wrote:
> While this general guideline of not requiring a DN on HM is correct, I
> just wanted to point out that the HM is hardly high on memory
> usage/requirements (as possibly stated below), as all it does is
> maintain the cluster and not serve nor hold any client-useful data or
> frequent requests.
>
> On Sat, Feb 23, 2013 at 1:13 AM, Mohammad Tariq <do...@gmail.com>
> wrote:
> > Hello Kaven,
> >
> > It's not all necessary to run HM on server running DN. Actually
> it's
> > not advisable to do so, as HM has high memory requirements and running
> > other daemons on the same machine may lead you to problems. The basic
> > cluster setup, which people normally use looks somewhat like this :
> >
> > NN+JT(Hadoop master)
> > SNN
> > HM(Hbase master)
> > DN+TT+RS
> >
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com>
> wrote:
> >
> >> Hi everyone,
> >>
> >> I had a quick question. do I need to be running datanode on the server
> >> that is running hbase master or datanode should be running only on
> servers
> >> that are running as regionserver?
> >>
> >> thanks,
> >> --
> >> Kaveh Minooie
> >>
>
>
>
> --
> Harsh J
>
Re: does hbase master need to be a hadoop datanode as well?
Posted by Harsh J <ha...@cloudera.com>.
While this general guideline of not requiring a DN on HM is correct, I
just wanted to point out that the HM is hardly high on memory
usage/requirements (as possibly stated below), as all it does is
maintain the cluster and not serve nor hold any client-useful data or
frequent requests.
On Sat, Feb 23, 2013 at 1:13 AM, Mohammad Tariq <do...@gmail.com> wrote:
> Hello Kaven,
>
> It's not all necessary to run HM on server running DN. Actually it's
> not advisable to do so, as HM has high memory requirements and running
> other daemons on the same machine may lead you to problems. The basic
> cluster setup, which people normally use looks somewhat like this :
>
> NN+JT(Hadoop master)
> SNN
> HM(Hbase master)
> DN+TT+RS
>
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com> wrote:
>
>> Hi everyone,
>>
>> I had a quick question. do I need to be running datanode on the server
>> that is running hbase master or datanode should be running only on servers
>> that are running as regionserver?
>>
>> thanks,
>> --
>> Kaveh Minooie
>>
--
Harsh J
Re: does hbase master need to be a hadoop datanode as well?
Posted by Mohammad Tariq <do...@gmail.com>.
Yeah..Good point JM.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Sat, Feb 23, 2013 at 1:19 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Just to add to Mohammad's advices, you should avoid to run ZK on the same
> servers as you are running HBase.
>
> Reason is, if you are running in long GCs, ZK might miss the heartbeats and
> thinks servers are down. So safer to run same separately if you can.
>
> JM
>
> 2013/2/22 Mohammad Tariq <do...@gmail.com>
>
> > Hello Kaven,
> >
> > It's not all necessary to run HM on server running DN. Actually
> it's
> > not advisable to do so, as HM has high memory requirements and running
> > other daemons on the same machine may lead you to problems. The basic
> > cluster setup, which people normally use looks somewhat like this :
> >
> > NN+JT(Hadoop master)
> > SNN
> > HM(Hbase master)
> > DN+TT+RS
> >
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com>
> wrote:
> >
> > > Hi everyone,
> > >
> > > I had a quick question. do I need to be running datanode on the server
> > > that is running hbase master or datanode should be running only on
> > servers
> > > that are running as regionserver?
> > >
> > > thanks,
> > > --
> > > Kaveh Minooie
> > >
> >
>
Re: does hbase master need to be a hadoop datanode as well?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
"as long as machine is powerful"
That's the key ;)
If you run you standalone HBase in a virtual machine, it might not be
powerful enought and you might miss the heartbeats. Same if you run on a
computer without enought memory and it's starting to swap.
JM
2013/2/23 anil gupta <an...@gmail.com>
> On Feb 22, 2013 11:50 AM, "Jean-Marc Spaggiari" <je...@spaggiari.org>
> wrote:
> >
> > Just to add to Mohammad's advices, you should avoid to run ZK on the same
> > servers as you are running HBase.
> >
> > Reason is, if you are running in long GCs, ZK might miss the heartbeats
> and
> > thinks servers are down. So safer to run same separately if you can.
> >
> Zk and HM run in seperate JVM; so how would running them on the same
> machine will cause ZK missing hearbeats?
> As far as I know as long as machine is powerful enough to handle the load
> of ZK and HM, then it should be fine. One thing to note is that Zk will
> work well if you have dedicated disk for it.
> > JM
> >
> > 2013/2/22 Mohammad Tariq <do...@gmail.com>
> >
> > > Hello Kaven,
> > >
> > > It's not all necessary to run HM on server running DN. Actually
> it's
> > > not advisable to do so, as HM has high memory requirements and running
> > > other daemons on the same machine may lead you to problems. The basic
> > > cluster setup, which people normally use looks somewhat like this :
> > >
> > > NN+JT(Hadoop master)
> > > SNN
> > > HM(Hbase master)
> > > DN+TT+RS
> > >
> > >
> > > Warm Regards,
> > > Tariq
> > > https://mtariq.jux.com/
> > > cloudfront.blogspot.com
> > >
> > >
> > > On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com>
> wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I had a quick question. do I need to be running datanode on the
> server
> > > > that is running hbase master or datanode should be running only on
> > > servers
> > > > that are running as regionserver?
> > > >
> > > > thanks,
> > > > --
> > > > Kaveh Minooie
> > > >
> > >
>
Re: does hbase master need to be a hadoop datanode as well?
Posted by anil gupta <an...@gmail.com>.
On Feb 22, 2013 11:50 AM, "Jean-Marc Spaggiari" <je...@spaggiari.org>
wrote:
>
> Just to add to Mohammad's advices, you should avoid to run ZK on the same
> servers as you are running HBase.
>
> Reason is, if you are running in long GCs, ZK might miss the heartbeats
and
> thinks servers are down. So safer to run same separately if you can.
>
Zk and HM run in seperate JVM; so how would running them on the same
machine will cause ZK missing hearbeats?
As far as I know as long as machine is powerful enough to handle the load
of ZK and HM, then it should be fine. One thing to note is that Zk will
work well if you have dedicated disk for it.
> JM
>
> 2013/2/22 Mohammad Tariq <do...@gmail.com>
>
> > Hello Kaven,
> >
> > It's not all necessary to run HM on server running DN. Actually
it's
> > not advisable to do so, as HM has high memory requirements and running
> > other daemons on the same machine may lead you to problems. The basic
> > cluster setup, which people normally use looks somewhat like this :
> >
> > NN+JT(Hadoop master)
> > SNN
> > HM(Hbase master)
> > DN+TT+RS
> >
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com>
wrote:
> >
> > > Hi everyone,
> > >
> > > I had a quick question. do I need to be running datanode on the server
> > > that is running hbase master or datanode should be running only on
> > servers
> > > that are running as regionserver?
> > >
> > > thanks,
> > > --
> > > Kaveh Minooie
> > >
> >
Re: does hbase master need to be a hadoop datanode as well?
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Just to add to Mohammad's advices, you should avoid to run ZK on the same
servers as you are running HBase.
Reason is, if you are running in long GCs, ZK might miss the heartbeats and
thinks servers are down. So safer to run same separately if you can.
JM
2013/2/22 Mohammad Tariq <do...@gmail.com>
> Hello Kaven,
>
> It's not all necessary to run HM on server running DN. Actually it's
> not advisable to do so, as HM has high memory requirements and running
> other daemons on the same machine may lead you to problems. The basic
> cluster setup, which people normally use looks somewhat like this :
>
> NN+JT(Hadoop master)
> SNN
> HM(Hbase master)
> DN+TT+RS
>
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com> wrote:
>
> > Hi everyone,
> >
> > I had a quick question. do I need to be running datanode on the server
> > that is running hbase master or datanode should be running only on
> servers
> > that are running as regionserver?
> >
> > thanks,
> > --
> > Kaveh Minooie
> >
>
Re: does hbase master need to be a hadoop datanode as well?
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Kaven,
It's not all necessary to run HM on server running DN. Actually it's
not advisable to do so, as HM has high memory requirements and running
other daemons on the same machine may lead you to problems. The basic
cluster setup, which people normally use looks somewhat like this :
NN+JT(Hadoop master)
SNN
HM(Hbase master)
DN+TT+RS
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Sat, Feb 23, 2013 at 12:45 AM, kaveh minooie <ka...@plutoz.com> wrote:
> Hi everyone,
>
> I had a quick question. do I need to be running datanode on the server
> that is running hbase master or datanode should be running only on servers
> that are running as regionserver?
>
> thanks,
> --
> Kaveh Minooie
>