You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Leitao Guo (JIRA)" <ji...@apache.org> on 2008/07/01 10:07:45 UTC
[jira] Created: (HADOOP-3674) dynamic heartbeat interval for the
locality-aware task scheduling
dynamic heartbeat interval for the locality-aware task scheduling
-----------------------------------------------------------------
Key: HADOOP-3674
URL: https://issues.apache.org/jira/browse/HADOOP-3674
Project: Hadoop Core
Issue Type: Wish
Components: mapred
Reporter: Leitao Guo
Priority: Minor
In current hadoop release (0.17.0), there is no special scheduling policy for those tasktrackers who have no data for some jobs. So, there would be inefficient in some senarios. For example, tasktracker A has the data for a job, but tasktracker B, which has no data for this job, sends the heartbeat message to the jobtracker for a new task before tasktrack A. The task may be scheduled to B instead of A. While Jobtracker has to find a new task for tasktracker A when A ask for a new task.
In this situation, if jobtracker has some reservation policy, such as reserve the task for tasktracker A and let B ask for new task in the next heartbeat message, that would be more efficient. Because before tasktracker B asking for new task the second time, tasktracker A has applied for a new task and jobtracker has scheduled the task to A.
Here is a rough idea to deal with the senario above:
(1) Jobtracker receives the heartbeat message sent by tasktracker B, which has no data for any job.
(2) Jobtracker send response message to tasktracker B with a new heartbeat message interval, but does not schedule new task to B. The new heartbeat interval should be shorter the current heartbeat interval, for example, current_heartbeat_interval/2.
(3) Tasktracker B receive the response from jobtracker, and sends another heartbeat message for a new task after a period of current_heartbeat_interval/2 .
(4) Jobtracker then find a new task for tasktracker B.
This is just an primary idea for the improvement of the locality-aware scheduling. Any comments are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3674) dynamic heartbeat interval for the
locality-aware task scheduling
Posted by "Amar Kamat (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609490#action_12609490 ]
Amar Kamat commented on HADOOP-3674:
------------------------------------
Leitao,
Generally we try to make sure that we dont waste any compute cycle of tasktrackers. If this is a big performance hit then we might need to rethink on this. Although we can bias the decision of what to give based on various parameters. Check HADOOP-2812 and HADOOP-2014 that are somewhat related. Let us know why you feel that not having a greedy approach works better here.
> dynamic heartbeat interval for the locality-aware task scheduling
> -----------------------------------------------------------------
>
> Key: HADOOP-3674
> URL: https://issues.apache.org/jira/browse/HADOOP-3674
> Project: Hadoop Core
> Issue Type: Wish
> Components: mapred
> Reporter: Leitao Guo
> Priority: Minor
>
> In current hadoop release (0.17.0), there is no special scheduling policy for those tasktrackers who have no data for some jobs. So, there would be inefficient in some senarios. For example, tasktracker A has the data for a job, but tasktracker B, which has no data for this job, sends the heartbeat message to the jobtracker for a new task before tasktrack A. The task may be scheduled to B instead of A. While Jobtracker has to find a new task for tasktracker A when A ask for a new task.
> In this situation, if jobtracker has some reservation policy, such as reserve the task for tasktracker A and let B ask for new task in the next heartbeat message, that would be more efficient. Because before tasktracker B asking for new task the second time, tasktracker A has applied for a new task and jobtracker has scheduled the task to A.
> Here is a rough idea to deal with the senario above:
> (1) Jobtracker receives the heartbeat message sent by tasktracker B, which has no data for any job.
> (2) Jobtracker send response message to tasktracker B with a new heartbeat message interval, but does not schedule new task to B. The new heartbeat interval should be shorter the current heartbeat interval, for example, current_heartbeat_interval/2.
> (3) Tasktracker B receive the response from jobtracker, and sends another heartbeat message for a new task after a period of current_heartbeat_interval/2 .
> (4) Jobtracker then find a new task for tasktracker B.
> This is just an primary idea for the improvement of the locality-aware scheduling. Any comments are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3674) dynamic heartbeat interval for the
locality-aware task scheduling
Posted by "Vivek Ratan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640074#action_12640074 ]
Vivek Ratan commented on HADOOP-3674:
-------------------------------------
There's definitely an advantage, in some cases, to looking ahead and not necessarily always assigning a task to a free TT. This is, however, hard to do, as the JT currently has no idea of the order in which the TTs will ask for tasks. There have been many discussions on basing scheduling decisions on a more global view of the system (see HADOOP-3136, for example).
Do you have any ideas on how the JT can predict the order of the TTs?
> dynamic heartbeat interval for the locality-aware task scheduling
> -----------------------------------------------------------------
>
> Key: HADOOP-3674
> URL: https://issues.apache.org/jira/browse/HADOOP-3674
> Project: Hadoop Core
> Issue Type: Wish
> Components: mapred
> Reporter: Leitao Guo
> Priority: Minor
>
> In current hadoop release (0.17.0), there is no special scheduling policy for those tasktrackers who have no data for some jobs. So, there would be inefficient in some senarios. For example, tasktracker A has the data for a job, but tasktracker B, which has no data for this job, sends the heartbeat message to the jobtracker for a new task before tasktrack A. The task may be scheduled to B instead of A. While Jobtracker has to find a new task for tasktracker A when A ask for a new task.
> In this situation, if jobtracker has some reservation policy, such as reserve the task for tasktracker A and let B ask for new task in the next heartbeat message, that would be more efficient. Because before tasktracker B asking for new task the second time, tasktracker A has applied for a new task and jobtracker has scheduled the task to A.
> Here is a rough idea to deal with the senario above:
> (1) Jobtracker receives the heartbeat message sent by tasktracker B, which has no data for any job.
> (2) Jobtracker send response message to tasktracker B with a new heartbeat message interval, but does not schedule new task to B. The new heartbeat interval should be shorter the current heartbeat interval, for example, current_heartbeat_interval/2.
> (3) Tasktracker B receive the response from jobtracker, and sends another heartbeat message for a new task after a period of current_heartbeat_interval/2 .
> (4) Jobtracker then find a new task for tasktracker B.
> This is just an primary idea for the improvement of the locality-aware scheduling. Any comments are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3674) dynamic heartbeat interval for the
locality-aware task scheduling
Posted by "Leitao Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609809#action_12609809 ]
Leitao Guo commented on HADOOP-3674:
------------------------------------
check this senario:
H1 has the data of Task1
H2 has the data of Task2
H3 has the data of Task3
H4 has the data of Task4
H5 has no data
And now, tasktrackers ask for new tasks in this order: H5, H4, H3, H2, H1.
If there is no special task scheduling policy for those tasktrackers who have no data, there would be like this:
Task4 is scheduled to H5,
Task3 is scheduled to H4,
Task2 is scheduled to H3,
Task1 is scheduled to H2,
no task is scheduled to H1.
This has caused the IO stress for data movement, although no compute cycies are wasted.
So, if there is a kind of reservation policy for task scheduling, and let those tasktrackers have no data wait for a litter more time, the situation would be like this:
Task4 is scheduled to H4,
Task3 is scheduled to H3,
Task2 is scheduled to H2,
Task1 is scheduled to H1,
no task is scheduled to H5.
This may be more efficient.
> dynamic heartbeat interval for the locality-aware task scheduling
> -----------------------------------------------------------------
>
> Key: HADOOP-3674
> URL: https://issues.apache.org/jira/browse/HADOOP-3674
> Project: Hadoop Core
> Issue Type: Wish
> Components: mapred
> Reporter: Leitao Guo
> Priority: Minor
>
> In current hadoop release (0.17.0), there is no special scheduling policy for those tasktrackers who have no data for some jobs. So, there would be inefficient in some senarios. For example, tasktracker A has the data for a job, but tasktracker B, which has no data for this job, sends the heartbeat message to the jobtracker for a new task before tasktrack A. The task may be scheduled to B instead of A. While Jobtracker has to find a new task for tasktracker A when A ask for a new task.
> In this situation, if jobtracker has some reservation policy, such as reserve the task for tasktracker A and let B ask for new task in the next heartbeat message, that would be more efficient. Because before tasktracker B asking for new task the second time, tasktracker A has applied for a new task and jobtracker has scheduled the task to A.
> Here is a rough idea to deal with the senario above:
> (1) Jobtracker receives the heartbeat message sent by tasktracker B, which has no data for any job.
> (2) Jobtracker send response message to tasktracker B with a new heartbeat message interval, but does not schedule new task to B. The new heartbeat interval should be shorter the current heartbeat interval, for example, current_heartbeat_interval/2.
> (3) Tasktracker B receive the response from jobtracker, and sends another heartbeat message for a new task after a period of current_heartbeat_interval/2 .
> (4) Jobtracker then find a new task for tasktracker B.
> This is just an primary idea for the improvement of the locality-aware scheduling. Any comments are welcome.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.