You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Ferdy (JIRA)" <ji...@apache.org> on 2009/06/19 15:55:07 UTC
[jira] Created: (HADOOP-6093) Capacity Scheduler does not kill
reduce tasks if no running reducers have made any progress at all.
Capacity Scheduler does not kill reduce tasks if no running reducers have made any progress at all.
---------------------------------------------------------------------------------------------------
Key: HADOOP-6093
URL: https://issues.apache.org/jira/browse/HADOOP-6093
Project: Hadoop Core
Issue Type: Bug
Components: contrib/capacity-sched
Affects Versions: 0.20.0
Reporter: Ferdy
Priority: Minor
If the Capacity Scheduler decides to kill a reduce job then it selects the task that made the least progress. In my test setup I created a dummy reduce task that does nothing but waiting indefinitely. All reduce progresses are "1". Now the "getRunningTaskWithLeastProgress(tip)" will return null, so no task is killed.
Although not very likely this will occur in a production setup (timeout killing would kick in anyway) but it may be a bit unexpecting.
I will attach a patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6093) Capacity Scheduler does not kill
reduce tasks if all reducers are in the progress of their last record.
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12721797#action_12721797 ]
Hemanth Yamijala commented on HADOOP-6093:
------------------------------------------
In HADOOP-5726, we removed code related to pre-emption in an effort to simplify development of the capacity scheduler. The method mentioned in the description is no longer there in the Hadoop 0.20 code base. So, I suppose this issue is no longer applicable to the branch / trunk.
> Capacity Scheduler does not kill reduce tasks if all reducers are in the progress of their last record.
> -------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6093
> URL: https://issues.apache.org/jira/browse/HADOOP-6093
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Ferdy
> Priority: Minor
> Attachments: PatchHadoop6093v1.patch
>
>
> If the Capacity Scheduler decides to kill a reduce job then it selects the task that made the least progress. In my test setup I created a dummy reduce task that does nothing but waiting indefinitely. All reduce progresses are "1" because all reducers are in the progress of their last record. Now the "getRunningTaskWithLeastProgress(tip)" will return null, so no task is killed.
> Although not very likely this will occur in a production setup (timeout killing would kick in anyway) but it may be a bit unexpecting.
> I will attach a patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6093) Capacity Scheduler does not kill
reduce tasks if no running reducers have made any progress at all.
Posted by "Ferdy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ferdy updated HADOOP-6093:
--------------------------
Attachment: PatchHadoop6093v1.patch
> Capacity Scheduler does not kill reduce tasks if no running reducers have made any progress at all.
> ---------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6093
> URL: https://issues.apache.org/jira/browse/HADOOP-6093
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Ferdy
> Priority: Minor
> Attachments: PatchHadoop6093v1.patch
>
>
> If the Capacity Scheduler decides to kill a reduce job then it selects the task that made the least progress. In my test setup I created a dummy reduce task that does nothing but waiting indefinitely. All reduce progresses are "1". Now the "getRunningTaskWithLeastProgress(tip)" will return null, so no task is killed.
> Although not very likely this will occur in a production setup (timeout killing would kick in anyway) but it may be a bit unexpecting.
> I will attach a patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6093) Capacity Scheduler does not kill
reduce tasks if all reducers are in the progress of their last record.
Posted by "Ferdy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ferdy updated HADOOP-6093:
--------------------------
Description:
If the Capacity Scheduler decides to kill a reduce job then it selects the task that made the least progress. In my test setup I created a dummy reduce task that does nothing but waiting indefinitely. All reduce progresses are "1" because all reducers are in the progress of their last record. Now the "getRunningTaskWithLeastProgress(tip)" will return null, so no task is killed.
Although not very likely this will occur in a production setup (timeout killing would kick in anyway) but it may be a bit unexpecting.
I will attach a patch.
was:
If the Capacity Scheduler decides to kill a reduce job then it selects the task that made the least progress. In my test setup I created a dummy reduce task that does nothing but waiting indefinitely. All reduce progresses are "1". Now the "getRunningTaskWithLeastProgress(tip)" will return null, so no task is killed.
Although not very likely this will occur in a production setup (timeout killing would kick in anyway) but it may be a bit unexpecting.
I will attach a patch.
Summary: Capacity Scheduler does not kill reduce tasks if all reducers are in the progress of their last record. (was: Capacity Scheduler does not kill reduce tasks if no running reducers have made any progress at all.)
> Capacity Scheduler does not kill reduce tasks if all reducers are in the progress of their last record.
> -------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6093
> URL: https://issues.apache.org/jira/browse/HADOOP-6093
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Ferdy
> Priority: Minor
> Attachments: PatchHadoop6093v1.patch
>
>
> If the Capacity Scheduler decides to kill a reduce job then it selects the task that made the least progress. In my test setup I created a dummy reduce task that does nothing but waiting indefinitely. All reduce progresses are "1" because all reducers are in the progress of their last record. Now the "getRunningTaskWithLeastProgress(tip)" will return null, so no task is killed.
> Although not very likely this will occur in a production setup (timeout killing would kick in anyway) but it may be a bit unexpecting.
> I will attach a patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6093) Capacity Scheduler does not kill
reduce tasks if all reducers are in the progress of their last record.
Posted by "Ferdy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ferdy updated HADOOP-6093:
--------------------------
Status: Patch Available (was: Open)
> Capacity Scheduler does not kill reduce tasks if all reducers are in the progress of their last record.
> -------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-6093
> URL: https://issues.apache.org/jira/browse/HADOOP-6093
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/capacity-sched
> Affects Versions: 0.20.0
> Reporter: Ferdy
> Priority: Minor
> Attachments: PatchHadoop6093v1.patch
>
>
> If the Capacity Scheduler decides to kill a reduce job then it selects the task that made the least progress. In my test setup I created a dummy reduce task that does nothing but waiting indefinitely. All reduce progresses are "1" because all reducers are in the progress of their last record. Now the "getRunningTaskWithLeastProgress(tip)" will return null, so no task is killed.
> Although not very likely this will occur in a production setup (timeout killing would kick in anyway) but it may be a bit unexpecting.
> I will attach a patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.