You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2008/09/06 05:58:44 UTC
[jira] Created: (HADOOP-4100) Scheduler.assignTasks should not be
dealing with cleanupTask
Scheduler.assignTasks should not be dealing with cleanupTask
------------------------------------------------------------
Key: HADOOP-4100
URL: https://issues.apache.org/jira/browse/HADOOP-4100
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.19.0
Reporter: Devaraj Das
Assignee: Amareshwari Sriramadasu
Fix For: 0.19.0
HADOOP-3150 introduced the notion of a cleanupTask. The implementation of the scheduling of the cleanupTask is such that the schedulers have to be aware of the cleanupTask. It would be very nice if schedulers didn't have to be aware of this special task. There is a discussion thread - https://issues.apache.org/jira/browse/HADOOP-3150?focusedCommentId=12628635#action_12628635
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4100) Scheduler.assignTasks should not be
dealing with cleanupTask
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628838#action_12628838 ]
Hemanth Yamijala commented on HADOOP-4100:
------------------------------------------
I am assuming that since this JIRA is filed, further discussions on the approach and implementation of this issue will happen here.
In the discussion thread mentioned in the description, it is suggested that the new cleanupTask introduced b HADOOP-3150 is scheduled from the framework directly, as opposed to having it scheduled from schedulers. In general, this seems the right approach. So, +1.
One question I have (and possibly this was indirectly brought up by Matei as well), should the running of the cleanupTask count as slots used by the Job - as this could impact fairness and capacity limits ? From what I've seen, it appears that each Job has atmost 1 cleanup task - correct ? In that case, it might not be *that* critical to count these for quotas etc. Comments ?
> Scheduler.assignTasks should not be dealing with cleanupTask
> ------------------------------------------------------------
>
> Key: HADOOP-4100
> URL: https://issues.apache.org/jira/browse/HADOOP-4100
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Devaraj Das
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
>
> HADOOP-3150 introduced the notion of a cleanupTask. The implementation of the scheduling of the cleanupTask is such that the schedulers have to be aware of the cleanupTask. It would be very nice if schedulers didn't have to be aware of this special task. There is a discussion thread - https://issues.apache.org/jira/browse/HADOOP-3150?focusedCommentId=12628635#action_12628635
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4100) Scheduler.assignTasks should not be
dealing with cleanupTask
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628877#action_12628877 ]
Devaraj Das commented on HADOOP-4100:
-------------------------------------
Hemanth, that's right. Every job will have at most one cleanup task (multiple attempts might happen due to failures just like a regular task). I don't think it is critical to keep track of this from the point of view of fairness/quotas, etc.
> Scheduler.assignTasks should not be dealing with cleanupTask
> ------------------------------------------------------------
>
> Key: HADOOP-4100
> URL: https://issues.apache.org/jira/browse/HADOOP-4100
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Devaraj Das
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
>
> HADOOP-3150 introduced the notion of a cleanupTask. The implementation of the scheduling of the cleanupTask is such that the schedulers have to be aware of the cleanupTask. It would be very nice if schedulers didn't have to be aware of this special task. There is a discussion thread - https://issues.apache.org/jira/browse/HADOOP-3150?focusedCommentId=12628635#action_12628635
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4100) Scheduler.assignTasks should not be
dealing with cleanupTask
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4100:
--------------------------------------------
Status: Patch Available (was: Open)
> Scheduler.assignTasks should not be dealing with cleanupTask
> ------------------------------------------------------------
>
> Key: HADOOP-4100
> URL: https://issues.apache.org/jira/browse/HADOOP-4100
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Devaraj Das
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4100.txt
>
>
> HADOOP-3150 introduced the notion of a cleanupTask. The implementation of the scheduling of the cleanupTask is such that the schedulers have to be aware of the cleanupTask. It would be very nice if schedulers didn't have to be aware of this special task. There is a discussion thread - https://issues.apache.org/jira/browse/HADOOP-3150?focusedCommentId=12628635#action_12628635
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4100) Scheduler.assignTasks should not be
dealing with cleanupTask
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629175#action_12629175 ]
Hadoop QA commented on HADOOP-4100:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389658/patch-4100.txt
against trunk revision 693048.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3206/console
This message is automatically generated.
> Scheduler.assignTasks should not be dealing with cleanupTask
> ------------------------------------------------------------
>
> Key: HADOOP-4100
> URL: https://issues.apache.org/jira/browse/HADOOP-4100
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Devaraj Das
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4100.txt
>
>
> HADOOP-3150 introduced the notion of a cleanupTask. The implementation of the scheduling of the cleanupTask is such that the schedulers have to be aware of the cleanupTask. It would be very nice if schedulers didn't have to be aware of this special task. There is a discussion thread - https://issues.apache.org/jira/browse/HADOOP-3150?focusedCommentId=12628635#action_12628635
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4100) Scheduler.assignTasks should not be
dealing with cleanupTask
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-4100:
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Amareshwari!
> Scheduler.assignTasks should not be dealing with cleanupTask
> ------------------------------------------------------------
>
> Key: HADOOP-4100
> URL: https://issues.apache.org/jira/browse/HADOOP-4100
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Devaraj Das
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4100.txt
>
>
> HADOOP-3150 introduced the notion of a cleanupTask. The implementation of the scheduling of the cleanupTask is such that the schedulers have to be aware of the cleanupTask. It would be very nice if schedulers didn't have to be aware of this special task. There is a discussion thread - https://issues.apache.org/jira/browse/HADOOP-3150?focusedCommentId=12628635#action_12628635
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4100) Scheduler.assignTasks should not be
dealing with cleanupTask
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629182#action_12629182 ]
Matei Zaharia commented on HADOOP-4100:
---------------------------------------
Looks good to me.
> Scheduler.assignTasks should not be dealing with cleanupTask
> ------------------------------------------------------------
>
> Key: HADOOP-4100
> URL: https://issues.apache.org/jira/browse/HADOOP-4100
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Devaraj Das
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4100.txt
>
>
> HADOOP-3150 introduced the notion of a cleanupTask. The implementation of the scheduling of the cleanupTask is such that the schedulers have to be aware of the cleanupTask. It would be very nice if schedulers didn't have to be aware of this special task. There is a discussion thread - https://issues.apache.org/jira/browse/HADOOP-3150?focusedCommentId=12628635#action_12628635
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4100) Scheduler.assignTasks should not be
dealing with cleanupTask
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-4100:
--------------------------------------------
Attachment: patch-4100.txt
Here is a patch which removes the dependency to assign a cleanup task from scheduler. Now, the JobTracker gets cleanup task for the jobs if any and launches them. If there are no cleanup tasks to be launched, it calls TaskScheduler.assignTasks().
> Scheduler.assignTasks should not be dealing with cleanupTask
> ------------------------------------------------------------
>
> Key: HADOOP-4100
> URL: https://issues.apache.org/jira/browse/HADOOP-4100
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Devaraj Das
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4100.txt
>
>
> HADOOP-3150 introduced the notion of a cleanupTask. The implementation of the scheduling of the cleanupTask is such that the schedulers have to be aware of the cleanupTask. It would be very nice if schedulers didn't have to be aware of this special task. There is a discussion thread - https://issues.apache.org/jira/browse/HADOOP-3150?focusedCommentId=12628635#action_12628635
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4100) Scheduler.assignTasks should not be
dealing with cleanupTask
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630184#action_12630184 ]
Hudson commented on HADOOP-4100:
--------------------------------
Integrated in Hadoop-trunk #600 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/600/])
> Scheduler.assignTasks should not be dealing with cleanupTask
> ------------------------------------------------------------
>
> Key: HADOOP-4100
> URL: https://issues.apache.org/jira/browse/HADOOP-4100
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Devaraj Das
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-4100.txt
>
>
> HADOOP-3150 introduced the notion of a cleanupTask. The implementation of the scheduling of the cleanupTask is such that the schedulers have to be aware of the cleanupTask. It would be very nice if schedulers didn't have to be aware of this special task. There is a discussion thread - https://issues.apache.org/jira/browse/HADOOP-3150?focusedCommentId=12628635#action_12628635
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.