You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2008/10/15 23:06:44 UTC
[jira] Created: (HADOOP-4421) Rework job-setup and job-cleanup
tasks
Rework job-setup and job-cleanup tasks
--------------------------------------
Key: HADOOP-4421
URL: https://issues.apache.org/jira/browse/HADOOP-4421
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.19.0
Reporter: Arun C Murthy
Fix For: 0.20.0
Currently we have a notion of map-{setup|cleanup} TIP and reduce-{setup|cleanup} TIP of which only 1 setup and cleanup tasks are picked by the JobInProgress. Also a lot of state-maintenence of these TIPs are done by the JobInProgress itself, outside of the more logical place i.e. TaskInProgress.
We really should rework this to have a single setup and cleanup task which isn't associated with a map or reduce task i.e. into separate task _types_. What we have currently is quite ungainly and hard to maintain.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4421) Rework job-setup and job-cleanup
tasks
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12639991#action_12639991 ]
Arun C Murthy commented on HADOOP-4421:
---------------------------------------
I propose we bite the bullet and introduce an enum TaskType {MAP, REDUCE, SETUP, CLEANUP} while we fix this...
> Rework job-setup and job-cleanup tasks
> --------------------------------------
>
> Key: HADOOP-4421
> URL: https://issues.apache.org/jira/browse/HADOOP-4421
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Arun C Murthy
> Fix For: 0.20.0
>
>
> Currently we have a notion of map-{setup|cleanup} TIP and reduce-{setup|cleanup} TIP of which only 1 setup and cleanup tasks are picked by the JobInProgress. Also a lot of state-maintenence of these TIPs are done by the JobInProgress itself, outside of the more logical place i.e. TaskInProgress.
> We really should rework this to have a single setup and cleanup task which isn't associated with a map or reduce task i.e. into separate task _types_. What we have currently is quite ungainly and hard to maintain.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4421) Rework job-setup and job-cleanup
tasks
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12698358#action_12698358 ]
Devaraj Das commented on HADOOP-4421:
-------------------------------------
I am +1 for the enum for TaskTypes.
There are two approaches to having slots for the setup/cleanup tasks -
1) Have a special slot in the TT for all the special tasks - JobSetup, JobCleanup and TaskCleanup. This special slot can run any of the special tasks. With this setup, we have three types of slots in the TT - Map, Reduce, and Setup/Cleanup slot.
2) Have the Map slots in the TT be floating between the actual maps and the special tasks. In this model, one would configure #Reduce slots in the TT, and "#other slots".
The other thing that I am thinking of removing is the reuse of taskattemptID for the cleanup tasks (as implemented in HADOOP-4759). Made the code hard to read/maintain.
Thoughts?
> Rework job-setup and job-cleanup tasks
> --------------------------------------
>
> Key: HADOOP-4421
> URL: https://issues.apache.org/jira/browse/HADOOP-4421
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Arun C Murthy
> Fix For: 0.20.0
>
>
> Currently we have a notion of map-{setup|cleanup} TIP and reduce-{setup|cleanup} TIP of which only 1 setup and cleanup tasks are picked by the JobInProgress. Also a lot of state-maintenence of these TIPs are done by the JobInProgress itself, outside of the more logical place i.e. TaskInProgress.
> We really should rework this to have a single setup and cleanup task which isn't associated with a map or reduce task i.e. into separate task _types_. What we have currently is quite ungainly and hard to maintain.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4421) Rework job-setup and job-cleanup
tasks
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12640051#action_12640051 ]
Amareshwari Sriramadasu commented on HADOOP-4421:
-------------------------------------------------
+1 for separate map, reduce, cleanup and setup TIPs.
Now the cleanup and setup tips occupy map/reduce slot whichever is free. I think we should have separate slot for cleanup and setup tasks sothat they can be launched asap.
> Rework job-setup and job-cleanup tasks
> --------------------------------------
>
> Key: HADOOP-4421
> URL: https://issues.apache.org/jira/browse/HADOOP-4421
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Arun C Murthy
> Fix For: 0.20.0
>
>
> Currently we have a notion of map-{setup|cleanup} TIP and reduce-{setup|cleanup} TIP of which only 1 setup and cleanup tasks are picked by the JobInProgress. Also a lot of state-maintenence of these TIPs are done by the JobInProgress itself, outside of the more logical place i.e. TaskInProgress.
> We really should rework this to have a single setup and cleanup task which isn't associated with a map or reduce task i.e. into separate task _types_. What we have currently is quite ungainly and hard to maintain.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4421) Rework job-setup and job-cleanup
tasks
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das reassigned HADOOP-4421:
-----------------------------------
Assignee: Devaraj Das
> Rework job-setup and job-cleanup tasks
> --------------------------------------
>
> Key: HADOOP-4421
> URL: https://issues.apache.org/jira/browse/HADOOP-4421
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Arun C Murthy
> Assignee: Devaraj Das
> Fix For: 0.20.0
>
>
> Currently we have a notion of map-{setup|cleanup} TIP and reduce-{setup|cleanup} TIP of which only 1 setup and cleanup tasks are picked by the JobInProgress. Also a lot of state-maintenence of these TIPs are done by the JobInProgress itself, outside of the more logical place i.e. TaskInProgress.
> We really should rework this to have a single setup and cleanup task which isn't associated with a map or reduce task i.e. into separate task _types_. What we have currently is quite ungainly and hard to maintain.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4421) Rework job-setup and job-cleanup
tasks
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12698808#action_12698808 ]
Devaraj Das commented on HADOOP-4421:
-------------------------------------
One more thing - schedulers would be made aware of these special tasks..
> Rework job-setup and job-cleanup tasks
> --------------------------------------
>
> Key: HADOOP-4421
> URL: https://issues.apache.org/jira/browse/HADOOP-4421
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Arun C Murthy
> Assignee: Devaraj Das
> Fix For: 0.20.0
>
>
> Currently we have a notion of map-{setup|cleanup} TIP and reduce-{setup|cleanup} TIP of which only 1 setup and cleanup tasks are picked by the JobInProgress. Also a lot of state-maintenence of these TIPs are done by the JobInProgress itself, outside of the more logical place i.e. TaskInProgress.
> We really should rework this to have a single setup and cleanup task which isn't associated with a map or reduce task i.e. into separate task _types_. What we have currently is quite ungainly and hard to maintain.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.