You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by André Oriani <ao...@gmail.com> on 2013/03/08 02:37:31 UTC
Heartbeat interval and timeout: why 3 secs and 10 min?
Hi,
Is there any particular reason why the default heartbeat interval is 3
seconds and the timeout is 10 minutes? Everywhere I looked (code, Google,
..) only mentions the values but no clue on why those values were chosen.
Thanks in advance,
André Oriani
Re: Heartbeat interval and timeout: why 3 secs and 10 min?
Posted by Suresh Srinivas <su...@hortonworks.com>.
You are right, in heartbeat response namenode sends commands to the
datanode. Commands sent this way include deletion of blocks, replication,
block recovery secret key updates etc. Increasing the heartbeat interval
results
in namenode not being able to quickly act on the events in the cluster and
send commands to datanodes.
Declaring a datanode dead is done with a much conservative value of 10
minutes to avoid unnecessarily replicating the data, which will affect
both storage and cluster bandwidth.
On Tue, Mar 12, 2013 at 5:29 PM, André Oriani <ao...@gmail.com> wrote:
> No take on this one?
>
> In Zookeeper the heartbeats happen on every third of the timeout. If I am
> not mistaken, recomended timeout is more than 2 minutes to avoid false
> positives.
>
> But I still cannot see the relationship on HDFS between heartbeat interval
> and timeout. Okay 10 minutes seems to be a conservative value to avoid
> false positives in a big cluster. But that means 200 hearbeats. Heartbeats
> on HDFS are not only used for liveness detection but also to send
> information about free space and load and to receive commands from
> NameNode. So they are also essential for block placement decisions and for
> ensuring the replication levels. Would that then be reason why heartbeats
> are so frequent? A lot can happen to a DataNode in just three seconds?
>
>
> Thanks,
> André Oriani
>
>
>
> On Thu, Mar 7, 2013 at 10:37 PM, André Oriani <ao...@gmail.com> wrote:
>
> > Hi,
> >
> > Is there any particular reason why the default heartbeat interval is 3
> > seconds and the timeout is 10 minutes? Everywhere I looked (code, Google,
> > ..) only mentions the values but no clue on why those values were
> chosen.
> >
> >
> > Thanks in advance,
> > André Oriani
> >
>
--
http://hortonworks.com/download/
Re: Heartbeat interval and timeout: why 3 secs and 10 min?
Posted by André Oriani <ao...@gmail.com>.
Thanks Colin and Suresh!
On Wed, Mar 13, 2013 at 3:08 PM, Colin McCabe <cm...@alumni.cmu.edu>wrote:
> My understanding is that the 10 minute timeout helps to avoid replication
> storms, especially during startup.
>
> You might be interested in HDFS-3703, which adds a "stale" state which
> datanodes are placed into after 30 seconds of missing heartbeats. (This is
> an optional feature controlled by dfs.namenode.check.stale.datanode )
>
> best,
> Colin
>
>
> On Tue, Mar 12, 2013 at 5:29 PM, André Oriani <ao...@gmail.com> wrote:
>
> > No take on this one?
> >
> > In Zookeeper the heartbeats happen on every third of the timeout. If I
> am
> > not mistaken, recomended timeout is more than 2 minutes to avoid false
> > positives.
> >
> > But I still cannot see the relationship on HDFS between heartbeat
> interval
> > and timeout. Okay 10 minutes seems to be a conservative value to avoid
> > false positives in a big cluster. But that means 200 hearbeats.
> Heartbeats
> > on HDFS are not only used for liveness detection but also to send
> > information about free space and load and to receive commands from
> > NameNode. So they are also essential for block placement decisions and
> for
> > ensuring the replication levels. Would that then be reason why heartbeats
> > are so frequent? A lot can happen to a DataNode in just three seconds?
> >
> >
> > Thanks,
> > André Oriani
> >
> >
> >
> > On Thu, Mar 7, 2013 at 10:37 PM, André Oriani <ao...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Is there any particular reason why the default heartbeat interval is 3
> > > seconds and the timeout is 10 minutes? Everywhere I looked (code,
> Google,
> > > ..) only mentions the values but no clue on why those values were
> > chosen.
> > >
> > >
> > > Thanks in advance,
> > > André Oriani
> > >
> >
>
Re: Heartbeat interval and timeout: why 3 secs and 10 min?
Posted by Colin McCabe <cm...@alumni.cmu.edu>.
My understanding is that the 10 minute timeout helps to avoid replication
storms, especially during startup.
You might be interested in HDFS-3703, which adds a "stale" state which
datanodes are placed into after 30 seconds of missing heartbeats. (This is
an optional feature controlled by dfs.namenode.check.stale.datanode )
best,
Colin
On Tue, Mar 12, 2013 at 5:29 PM, André Oriani <ao...@gmail.com> wrote:
> No take on this one?
>
> In Zookeeper the heartbeats happen on every third of the timeout. If I am
> not mistaken, recomended timeout is more than 2 minutes to avoid false
> positives.
>
> But I still cannot see the relationship on HDFS between heartbeat interval
> and timeout. Okay 10 minutes seems to be a conservative value to avoid
> false positives in a big cluster. But that means 200 hearbeats. Heartbeats
> on HDFS are not only used for liveness detection but also to send
> information about free space and load and to receive commands from
> NameNode. So they are also essential for block placement decisions and for
> ensuring the replication levels. Would that then be reason why heartbeats
> are so frequent? A lot can happen to a DataNode in just three seconds?
>
>
> Thanks,
> André Oriani
>
>
>
> On Thu, Mar 7, 2013 at 10:37 PM, André Oriani <ao...@gmail.com> wrote:
>
> > Hi,
> >
> > Is there any particular reason why the default heartbeat interval is 3
> > seconds and the timeout is 10 minutes? Everywhere I looked (code, Google,
> > ..) only mentions the values but no clue on why those values were
> chosen.
> >
> >
> > Thanks in advance,
> > André Oriani
> >
>
Re: Heartbeat interval and timeout: why 3 secs and 10 min?
Posted by André Oriani <ao...@gmail.com>.
No take on this one?
In Zookeeper the heartbeats happen on every third of the timeout. If I am
not mistaken, recomended timeout is more than 2 minutes to avoid false
positives.
But I still cannot see the relationship on HDFS between heartbeat interval
and timeout. Okay 10 minutes seems to be a conservative value to avoid
false positives in a big cluster. But that means 200 hearbeats. Heartbeats
on HDFS are not only used for liveness detection but also to send
information about free space and load and to receive commands from
NameNode. So they are also essential for block placement decisions and for
ensuring the replication levels. Would that then be reason why heartbeats
are so frequent? A lot can happen to a DataNode in just three seconds?
Thanks,
André Oriani
On Thu, Mar 7, 2013 at 10:37 PM, André Oriani <ao...@gmail.com> wrote:
> Hi,
>
> Is there any particular reason why the default heartbeat interval is 3
> seconds and the timeout is 10 minutes? Everywhere I looked (code, Google,
> ..) only mentions the values but no clue on why those values were chosen.
>
>
> Thanks in advance,
> André Oriani
>