You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Mayank Bansal (JIRA)" <ji...@apache.org> on 2012/06/21 23:43:43 UTC
[jira] [Created] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honur the max capacity of container queue
Mayank Bansal created MAPREDUCE-4360:
----------------------------------------
Summary: Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
Key: MAPREDUCE-4360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.22.1, trunk
Reporter: Mayank Bansal
Assignee: Mayank Bansal
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honur the max capacity of container queue
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402271#comment-13402271 ]
Jason Lowe commented on MAPREDUCE-4360:
---------------------------------------
This JIRA indicates that trunk is affected, but I believe this has already been addressed in trunk (and branch-2 and branch-0.23) by MAPREDUCE-3683.
> Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1, trunk
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Attachments: MAPREDUCE-4360-22.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honur the max capacity of container queue
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404132#comment-13404132 ]
Konstantin Shvachko commented on MAPREDUCE-4360:
------------------------------------------------
Sorry my mistake. I had something else running on my box on port 50030, preventing SiulatorJobTracker to start for mumak. It's fine now. I'll commit the patch.
Will remove unused variable counter in the new test.
> Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Attachments: MAPREDUCE-4360-22-1.patch, MAPREDUCE-4360-22.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honur the max capacity of container queue
Posted by "Mayank Bansal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mayank Bansal updated MAPREDUCE-4360:
-------------------------------------
Affects Version/s: (was: trunk)
> Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Attachments: MAPREDUCE-4360-22.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honur the max capacity of container queue
Posted by "Mayank Bansal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mayank Bansal updated MAPREDUCE-4360:
-------------------------------------
Attachment: MAPREDUCE-4360-22.patch
Attaching the initial patch for 22.
Thanks,
Mayank
> Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1, trunk
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Attachments: MAPREDUCE-4360-22.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honur the max capacity of container queue
Posted by "Mayank Bansal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402426#comment-13402426 ]
Mayank Bansal commented on MAPREDUCE-4360:
------------------------------------------
Jason,
I did not realize that it is already fixed in trunk will update the JIRA. Thanks for pointing this out.
Konst,
Thats already been done in when tasks been assigned any queue.
Thanks,
Mayank
> Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1, trunk
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Attachments: MAPREDUCE-4360-22.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honur the max capacity of container queue
Posted by "Mayank Bansal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mayank Bansal updated MAPREDUCE-4360:
-------------------------------------
Attachment: MAPREDUCE-4360-22-1.patch
Thanks Konst for your comments.
Updated the patch with formatting issues.
Thanks,
Mayank
> Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Attachments: MAPREDUCE-4360-22-1.patch, MAPREDUCE-4360-22.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honur the max capacity of container queue
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403731#comment-13403731 ]
Konstantin Shvachko commented on MAPREDUCE-4360:
------------------------------------------------
Patch looks good. Wanted to commit it, but mumak tests are failing.
Run test-contrib target, you will see.
> Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Attachments: MAPREDUCE-4360-22-1.patch, MAPREDUCE-4360-22.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honur the max capacity of container queue
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko resolved MAPREDUCE-4360.
--------------------------------------------
Resolution: Fixed
Fix Version/s: 0.22.1
Target Version/s: 0.22.1
Hadoop Flags: Reviewed
I just committed this. Thank you Mayank.
> Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Fix For: 0.22.1
>
> Attachments: MAPREDUCE-4360-22-1.patch, MAPREDUCE-4360-22.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honur the max capacity of container queue
Posted by "Mayank Bansal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13398892#comment-13398892 ]
Mayank Bansal commented on MAPREDUCE-4360:
------------------------------------------
Maximum capacity set on a container queue is not getting honored. For ex: If Queue A has Queue A1 and Queue A2 as child queues (leaf queues), and if max capacity is not set for the child queues and set only to the container queue (Queue A), then all the slots occupied by the jobs submitted to Queue A1 and Queue A2 should not exceed the maximum capacity configured for Queue A. However, it is not the case now. Tasks gets scheduled beyond Queue A's configured max capacity.
> Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1, trunk
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honur the max capacity of container queue
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401849#comment-13401849 ]
Konstantin Shvachko commented on MAPREDUCE-4360:
------------------------------------------------
You should work on usual formatting issues.
So the fix as I understood is to build a hierarchy of queues by setting the parent member. Then find the parent that has maxCapacity set, and verify it is not exceeded. Sounds like the right approach.
Do you need to update the current capacity for the parent, when you allocate new slots for a child queue?
> Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1, trunk
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Attachments: MAPREDUCE-4360-22.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honor the max capacity of container queue
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404363#comment-13404363 ]
Hudson commented on MAPREDUCE-4360:
-----------------------------------
Integrated in Hadoop-Mapreduce-22-branch #108 (See [https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/108/])
MAPREDUCE-4360. Capacity Scheduler Hierarchical leaf queue does not honor the max capacity of container queue. Contributed by Mayank Bansal. (Revision 1355514)
Result = SUCCESS
shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1355514
Files :
* /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
* /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred/CapacityTaskScheduler.java
* /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred/QueueHierarchyBuilder.java
* /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/java/org/apache/hadoop/mapred/QueueSchedulingContext.java
* /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/capacity-scheduler/src/test/org/apache/hadoop/mapred/TestContainerQueue.java
> Capacity Scheduler Hierarchical leaf queue does not honor the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Fix For: 0.22.1
>
> Attachments: MAPREDUCE-4360-22-1.patch, MAPREDUCE-4360-22.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4360) Capacity Scheduler Hierarchical
leaf queue does not honor the max capacity of container queue
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated MAPREDUCE-4360:
-------------------------------------------
Summary: Capacity Scheduler Hierarchical leaf queue does not honor the max capacity of container queue (was: Capacity Scheduler Hierarchical leaf queue does not honur the max capacity of container queue)
> Capacity Scheduler Hierarchical leaf queue does not honor the max capacity of container queue
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4360
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4360
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.22.1
> Reporter: Mayank Bansal
> Assignee: Mayank Bansal
> Fix For: 0.22.1
>
> Attachments: MAPREDUCE-4360-22-1.patch, MAPREDUCE-4360-22.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira