You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Hemanth Yamijala (JIRA)" <ji...@apache.org> on 2009/08/13 06:41:14 UTC
[jira] Created: (MAPREDUCE-853) Support a hierarchy of queues in
the Map/Reduce framework
Support a hierarchy of queues in the Map/Reduce framework
---------------------------------------------------------
Key: MAPREDUCE-853
URL: https://issues.apache.org/jira/browse/MAPREDUCE-853
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: jobtracker
Reporter: Hemanth Yamijala
Fix For: 0.21.0
In MAPREDUCE-824, we proposed introducing a hierarchy of queues in the capacity scheduler. Currently, the M/R framework provides the notion of job queues and handles some functionality related to queues in a scheduler-agnostic manner. This functionality includes:
- Managing the list of ACLs for queues
- Managing the run state of queues - running or stopped
- Displaying scheduling information about queues in the jobtracker web UI and job client CLI
- Displaying list of jobs in a queue in the jobtracker web UI and job client CLI
- Providing APIs for list queues and queue information in JobClient.
Since it would be beneficial to extend this functionality to hierarchical queues, this JIRA is proposing introducing the concept into the map/reduce framework as well. We could treat this as an umbrella JIRA and file additional tasks for each of the changes involved, sticking to the high level approach in this JIRA.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-853) Support a hierarchy of queues in
the Map/Reduce framework
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hemanth Yamijala resolved MAPREDUCE-853.
----------------------------------------
Resolution: Fixed
All the three sub tasks associated to this JIRA are resolved. So, resolving the ticket.
> Support a hierarchy of queues in the Map/Reduce framework
> ---------------------------------------------------------
>
> Key: MAPREDUCE-853
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-853
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Fix For: 0.21.0
>
>
> In MAPREDUCE-824, we proposed introducing a hierarchy of queues in the capacity scheduler. Currently, the M/R framework provides the notion of job queues and handles some functionality related to queues in a scheduler-agnostic manner. This functionality includes:
> - Managing the list of ACLs for queues
> - Managing the run state of queues - running or stopped
> - Displaying scheduling information about queues in the jobtracker web UI and job client CLI
> - Displaying list of jobs in a queue in the jobtracker web UI and job client CLI
> - Providing APIs for list queues and queue information in JobClient.
> Since it would be beneficial to extend this functionality to hierarchical queues, this JIRA is proposing introducing the concept into the map/reduce framework as well. We could treat this as an umbrella JIRA and file additional tasks for each of the changes involved, sticking to the high level approach in this JIRA.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-853) Support a hierarchy of queues in
the Map/Reduce framework
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742804#action_12742804 ]
Hemanth Yamijala commented on MAPREDUCE-853:
--------------------------------------------
In summary, the following three areas would be impacted by this change:
- APIs in JobClient (public), JobSubmissionProtocol and QueueManager
- Configuration in mapred-queues.xml and mapred-site.xml
- UI - the hadoop queue family of commands in the CLI and the Scheduling information shown in the Jobtracker's web UI and Queue details pages.
It is a goal to be able to keep this feature backwards compatible. So sites not interested in hierarchical queues, but using the current single level of queues should be able to continue to function as before.
To keep details manageable and not confuse with the high level picture, I will add tasks under this JIRA for each of these areas.
> Support a hierarchy of queues in the Map/Reduce framework
> ---------------------------------------------------------
>
> Key: MAPREDUCE-853
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-853
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Fix For: 0.21.0
>
>
> In MAPREDUCE-824, we proposed introducing a hierarchy of queues in the capacity scheduler. Currently, the M/R framework provides the notion of job queues and handles some functionality related to queues in a scheduler-agnostic manner. This functionality includes:
> - Managing the list of ACLs for queues
> - Managing the run state of queues - running or stopped
> - Displaying scheduling information about queues in the jobtracker web UI and job client CLI
> - Displaying list of jobs in a queue in the jobtracker web UI and job client CLI
> - Providing APIs for list queues and queue information in JobClient.
> Since it would be beneficial to extend this functionality to hierarchical queues, this JIRA is proposing introducing the concept into the map/reduce framework as well. We could treat this as an umbrella JIRA and file additional tasks for each of the changes involved, sticking to the high level approach in this JIRA.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-853) Support a hierarchy of queues in
the Map/Reduce framework
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742796#action_12742796 ]
Hemanth Yamijala commented on MAPREDUCE-853:
--------------------------------------------
The basic proposal is to define the concept of sub-queues. Sub-queues are queues that are contained in other queues. Sub-queues can be nested. The last level of queues in the hierarchy, the leaf level queues, are called job queues, as they contain jobs. By that definition, all the queues that are defined in the present system are job queues.
An example organization could be:
{noformat}
grid {
org1 {
priority {
production
proj1
proj2
proj3
}
miscellaneous
}
org2
}
{noformat}
In this example, grid, org1, and priority are container queues. And production, proj1, proj2, proj3, miscellaneous and org2 are job queues. An example of how policies such as capacity can be assigned to this hierarchy and how it benefits them is described in MAPREDUCE-824.
> Support a hierarchy of queues in the Map/Reduce framework
> ---------------------------------------------------------
>
> Key: MAPREDUCE-853
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-853
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Fix For: 0.21.0
>
>
> In MAPREDUCE-824, we proposed introducing a hierarchy of queues in the capacity scheduler. Currently, the M/R framework provides the notion of job queues and handles some functionality related to queues in a scheduler-agnostic manner. This functionality includes:
> - Managing the list of ACLs for queues
> - Managing the run state of queues - running or stopped
> - Displaying scheduling information about queues in the jobtracker web UI and job client CLI
> - Displaying list of jobs in a queue in the jobtracker web UI and job client CLI
> - Providing APIs for list queues and queue information in JobClient.
> Since it would be beneficial to extend this functionality to hierarchical queues, this JIRA is proposing introducing the concept into the map/reduce framework as well. We could treat this as an umbrella JIRA and file additional tasks for each of the changes involved, sticking to the high level approach in this JIRA.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-853) Support a hierarchy of queues in
the Map/Reduce framework
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742713#action_12742713 ]
Hemanth Yamijala commented on MAPREDUCE-853:
--------------------------------------------
The motivation for hierarchical queues is discussed in the [proposal on MAPREDUCE-824|https://issues.apache.org/jira/browse/MAPREDUCE-824?focusedCommentId=12738975&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12738975], for those interested.
Briefly, hierarchical queues allow administrators to have a greater control over how capacity (or conceivably other policies) associated with a queue can be used. They also allow delegation of control. Large clusters could have root level queues set up for organizations, and then operators from the organizations could be given access to manage queues under the root level queues (which we are calling sub-queues in MAPREDUCE-824).
> Support a hierarchy of queues in the Map/Reduce framework
> ---------------------------------------------------------
>
> Key: MAPREDUCE-853
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-853
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: jobtracker
> Reporter: Hemanth Yamijala
> Fix For: 0.21.0
>
>
> In MAPREDUCE-824, we proposed introducing a hierarchy of queues in the capacity scheduler. Currently, the M/R framework provides the notion of job queues and handles some functionality related to queues in a scheduler-agnostic manner. This functionality includes:
> - Managing the list of ACLs for queues
> - Managing the run state of queues - running or stopped
> - Displaying scheduling information about queues in the jobtracker web UI and job client CLI
> - Displaying list of jobs in a queue in the jobtracker web UI and job client CLI
> - Providing APIs for list queues and queue information in JobClient.
> Since it would be beneficial to extend this functionality to hierarchical queues, this JIRA is proposing introducing the concept into the map/reduce framework as well. We could treat this as an umbrella JIRA and file additional tasks for each of the changes involved, sticking to the high level approach in this JIRA.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.