You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amareshwari Sri Ramadasu (JIRA)" <ji...@apache.org> on 2007/11/13 12:57:54 UTC
[jira] Issue Comment Edited: (HADOOP-1900) the heartbeat and task
event queries interval should be set dynamically by the JobTracker
[ https://issues.apache.org/jira/browse/HADOOP-1900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12542122 ]
amareshwari edited comment on HADOOP-1900 at 11/13/07 3:57 AM:
----------------------------------------------------------------------------
With the patch attached, I ran sort benchmarks on 390 node cluster and 120 node cluster. The performance is almost the same as with the trunk.
To simulate busyness at the job tracker, I ran the sort benchmarks on 120 node cluster with number of handlers=4 and with max queue size per handler =10, but there are drops and lost task trackers with the patch and without.
Thus Cluster size factor as (clusterSize/50+1) is fine. but the busy factor has to be tuned more.
I propose the following to tune busy factor:
We have threshouldDropCount = clusterSize/10;
We increment busyFactor by HEARTBEAT_BUSY_FACTOR (say 2secs) for 10% cluster size drops.
if(dropCount > threshouldDropCount) {
busyFactor += (dropCount/threshouldDropCount)*HEARTBEAT_BUSY_FACTOR;
}
if job tracker is not busy for '_notBusyPeriod_' , then we will decrement busyFactor by HEARTBEAT_BUSY_FACTOR;
We have, 2 rpcs to be processed as at the jobtracker i.e. heartbeat and task completion events. let processing time for rpc be 2 seconds.
Here, notBusyPeriod is calculated as:
notBusyPeriod = (clusterSize/#handlers)*processingTime*2;
To stabilize,
consider we have drops at t, and we increment heartbeat interval by busyfactor,_b_. And we notice that _b_ is small enough to get decremented and we dont see drops at the new interval, then we stabilize.
Thoughts?
was (Author: amareshwari):
With the patch attached, I ran sort benchmarks on 390 node cluster and 120 node cluster. The performance is almost the same as with the trunk.
To simulate busyness at the job tracker, I ran the sort benchmarks on 120 node cluster with number of handlers=4 and with max queue size per handler =10, but there are drops and lost task trackers with the patch and wothout.
Thus Cluster size factor as (clusterSize/50+1) is fine. but the busy factor has to be tuned more.
I propose the following to tune busy factor:
threshouldDropCount = clusterSize/10;
if(dropCount > threshouldDropCount) {
busyFactor += (dropCount/threshouldDropCount)*HEARTBEAT_BUSY_FACTOR;
}
if job tracker is not busy for '_notBusyPeriod_' , then we will decrement busyFactor by HEARTBEAT_BUSY_FACTOR;
We have, 2 rpcs to be processed as at the jobtracker i.e. heartbeat and task completion events. let processing time for rpc be 2 seconds.
Here, notBusyPeriod is calculated as:
notBusyPeriod = (clusterSize/#handlers)*processingTime*2;
To stabilize,
consider we have drops at t, and we increment heartbeat interval by busyfactor,_b_. And we notice that _b_ is small enough and we dont see drops at the new interval, then we stabilize.
Thoughts?
> the heartbeat and task event queries interval should be set dynamically by the JobTracker
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-1900
> URL: https://issues.apache.org/jira/browse/HADOOP-1900
> Project: Hadoop
> Issue Type: Improvement
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sri Ramadasu
> Attachments: patch-1900.txt, patch-1900.txt
>
>
> The JobTracker should scale the intervals that the TaskTrackers use to contact it dynamically, based on how the busy it is and the size of the cluster.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.