You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ChingShen <ch...@gmail.com> on 2010/02/09 17:52:10 UTC
Why does the default HEARTBEAT_INTERVAL value is 3?
Hi,
I have a question about HEARTBEAT_INTERVAL.
Why does the default HEARTBEAT_INTERVAL value is 3 rather than 2 or 1? any
resources?
Thanks.
Shen
Re: Why does the default HEARTBEAT_INTERVAL value is 3?
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, Feb 9, 2010 at 10:22 PM, Tomer Shiran <ts...@maprtech.com> wrote:
> Just curious. What limit are you referring to?
>
See the JIRA:
https://issues.apache.org/jira/browse/MAPREDUCE-1266
The limit is the HEARTBEAT_SCALING_FACTOR stuff, and the clusterSize factor
(second comment down)
-Todd
> On Tue, Feb 9, 2010 at 11:42 AM, Todd Lipcon <to...@cloudera.com> wrote:
>
> > On Tue, Feb 9, 2010 at 11:37 AM, Edward Capriolo <ed...@gmail.com>
> > wrote:
> >
> > > With the setting of 5 each tasktracker checks into the jobtracker
> > > every 5 seconds. The concept is that with enough TaskTrackers , say a
> > > 1000 node cluster 1000/5= 200 will be checking in at and given second.
> >
> > Actually there's already a limit that automatically scales up the
> > heartbeat interval as the cluster size increases. So reducing the
> > minimum won't break scalability on large clusters in any way.
> >
> > -Todd
> >
>
>
>
> --
> Tomer Shiran
> Director of Product Management | MapR Technologies (www.mapr.com) |
> 650-804-8657
>
Re: Why does the default HEARTBEAT_INTERVAL value is 3?
Posted by Tomer Shiran <ts...@maprtech.com>.
Just curious. What limit are you referring to?
On Tue, Feb 9, 2010 at 11:42 AM, Todd Lipcon <to...@cloudera.com> wrote:
> On Tue, Feb 9, 2010 at 11:37 AM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
> > With the setting of 5 each tasktracker checks into the jobtracker
> > every 5 seconds. The concept is that with enough TaskTrackers , say a
> > 1000 node cluster 1000/5= 200 will be checking in at and given second.
>
> Actually there's already a limit that automatically scales up the
> heartbeat interval as the cluster size increases. So reducing the
> minimum won't break scalability on large clusters in any way.
>
> -Todd
>
--
Tomer Shiran
Director of Product Management | MapR Technologies (www.mapr.com) |
650-804-8657
Re: Why does the default HEARTBEAT_INTERVAL value is 3?
Posted by Todd Lipcon <to...@cloudera.com>.
On Tue, Feb 9, 2010 at 11:37 AM, Edward Capriolo <ed...@gmail.com> wrote:
> With the setting of 5 each tasktracker checks into the jobtracker
> every 5 seconds. The concept is that with enough TaskTrackers , say a
> 1000 node cluster 1000/5= 200 will be checking in at and given second.
Actually there's already a limit that automatically scales up the
heartbeat interval as the cluster size increases. So reducing the
minimum won't break scalability on large clusters in any way.
-Todd
Re: Why does the default HEARTBEAT_INTERVAL value is 3?
Posted by Edward Capriolo <ed...@gmail.com>.
On Tue, Feb 9, 2010 at 2:29 PM, Todd Lipcon <to...@cloudera.com> wrote:
> On a small cluster, I'm of the opinion that a value less than 3 would
> actually be useful in reducing job startup time a little bit.
>
> https://issues.apache.org/jira/browse/MAPREDUCE-1266
>
> The issue got stalled a bit. If you want it, pipe up on the JIRA :)
> Especially if you have hard data indicating this is a good idea (I
> never had the time to really prove it)
>
> -Todd
>
> On Tue, Feb 9, 2010 at 10:21 AM, E. Sammer <er...@lifeless.net> wrote:
>> On 2/9/10 11:52 AM, ChingShen wrote:
>>>
>>> Hi,
>>>
>>> I have a question about HEARTBEAT_INTERVAL.
>>> Why does the default HEARTBEAT_INTERVAL value is 3 rather than 2 or 1?
>>> any
>>> resources?
>>
>> Shen:
>>
>> While I don't have a good answer for why the number 3 was chosen (actually,
>> I think it's 5 seconds on heartbeats and the 3 seconds is how often a task
>> tracker thread checks if progress is being made or something like that), I
>> can tell you that there's network chatter caused by the heartbeat. You
>> wouldn't want heartbeat to be any faster as you would unnecessarily cause
>> network congestion and force the job tracker to do additional (possibly
>> unnecessary) work. As the cluster grows, the heartbeat interval is increased
>> leading to even less frequent check-ins to attempt to mitigate the
>> congestion / high concurrency on the JT.
>>
>> One of the down sides to this is that tasks aren't given to task trackers as
>> quickly as they could be, but there are probably better ways of decreasing
>> the amount of time required to hand out work rather than simply increasing
>> the heartbeat rate.
>>
>> Keep in mind that most Hadoop jobs run for long periods of time, so the
>> slight delay in handing out tasks isn't a huge problem and 3 to 5 seconds is
>> more than sufficient to know that a task tracker is alive and healthy.
>>
>> Hope this helps.
>> --
>> Eric Sammer
>> eric@lifeless.net
>> http://esammer.blogspot.com
>>
>
With the setting of 5 each tasktracker checks into the jobtracker
every 5 seconds. The concept is that with enough TaskTrackers , say a
1000 node cluster 1000/5= 200 will be checking in at and given second.
As Todd mentions, if your cluster is small say 2 nodes 2/5 = .4 nodes
will be checking in for a given second and this results in a small
delay. Configurable would be nice, then again there is already a good
amount of things to configure :)
Re: Why does the default HEARTBEAT_INTERVAL value is 3?
Posted by Todd Lipcon <to...@cloudera.com>.
On a small cluster, I'm of the opinion that a value less than 3 would
actually be useful in reducing job startup time a little bit.
https://issues.apache.org/jira/browse/MAPREDUCE-1266
The issue got stalled a bit. If you want it, pipe up on the JIRA :)
Especially if you have hard data indicating this is a good idea (I
never had the time to really prove it)
-Todd
On Tue, Feb 9, 2010 at 10:21 AM, E. Sammer <er...@lifeless.net> wrote:
> On 2/9/10 11:52 AM, ChingShen wrote:
>>
>> Hi,
>>
>> I have a question about HEARTBEAT_INTERVAL.
>> Why does the default HEARTBEAT_INTERVAL value is 3 rather than 2 or 1?
>> any
>> resources?
>
> Shen:
>
> While I don't have a good answer for why the number 3 was chosen (actually,
> I think it's 5 seconds on heartbeats and the 3 seconds is how often a task
> tracker thread checks if progress is being made or something like that), I
> can tell you that there's network chatter caused by the heartbeat. You
> wouldn't want heartbeat to be any faster as you would unnecessarily cause
> network congestion and force the job tracker to do additional (possibly
> unnecessary) work. As the cluster grows, the heartbeat interval is increased
> leading to even less frequent check-ins to attempt to mitigate the
> congestion / high concurrency on the JT.
>
> One of the down sides to this is that tasks aren't given to task trackers as
> quickly as they could be, but there are probably better ways of decreasing
> the amount of time required to hand out work rather than simply increasing
> the heartbeat rate.
>
> Keep in mind that most Hadoop jobs run for long periods of time, so the
> slight delay in handing out tasks isn't a huge problem and 3 to 5 seconds is
> more than sufficient to know that a task tracker is alive and healthy.
>
> Hope this helps.
> --
> Eric Sammer
> eric@lifeless.net
> http://esammer.blogspot.com
>
Re: Why does the default HEARTBEAT_INTERVAL value is 3?
Posted by "E. Sammer" <er...@lifeless.net>.
On 2/9/10 11:52 AM, ChingShen wrote:
> Hi,
>
> I have a question about HEARTBEAT_INTERVAL.
> Why does the default HEARTBEAT_INTERVAL value is 3 rather than 2 or 1? any
> resources?
Shen:
While I don't have a good answer for why the number 3 was chosen
(actually, I think it's 5 seconds on heartbeats and the 3 seconds is how
often a task tracker thread checks if progress is being made or
something like that), I can tell you that there's network chatter caused
by the heartbeat. You wouldn't want heartbeat to be any faster as you
would unnecessarily cause network congestion and force the job tracker
to do additional (possibly unnecessary) work. As the cluster grows, the
heartbeat interval is increased leading to even less frequent check-ins
to attempt to mitigate the congestion / high concurrency on the JT.
One of the down sides to this is that tasks aren't given to task
trackers as quickly as they could be, but there are probably better ways
of decreasing the amount of time required to hand out work rather than
simply increasing the heartbeat rate.
Keep in mind that most Hadoop jobs run for long periods of time, so the
slight delay in handing out tasks isn't a huge problem and 3 to 5
seconds is more than sufficient to know that a task tracker is alive and
healthy.
Hope this helps.
--
Eric Sammer
eric@lifeless.net
http://esammer.blogspot.com