You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Robert Joseph Evans (JIRA)" <ji...@apache.org> on 2011/07/22 18:17:57 UTC
[jira] [Created] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to
mrv2
MR-279: port MAPREDUCE-2324 to mrv2
-----------------------------------
Key: MAPREDUCE-2723
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 0.23.0
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans
Fix For: 0.23.0
MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to
mrv2
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069602#comment-13069602 ]
Robert Joseph Evans commented on MAPREDUCE-2723:
------------------------------------------------
I have been looking at MR-279 and I want to do something similar to MR-2324 it is just not really set up to do it easily. The scheduling is split up between the resource manager and the application master. And in fact the resource manager/application master are completely ignoring disk utilization at this point.
The plan is to add in disk utilization to the resources that the RM uses, and then have AM request both disk and RAM space for reduces with disk space based off of the size estimate currently used. Then inside the scheduler, which is the right place in my opinion to decide if a request is being starved or not, it would do just what MR-2324 does now but more generalized for all resource constraints, not just disk. This means that all schedulers would have to be modified to support this, but I can make the code generic so it should be fairly simple to do. I just need to dig into the MR-279 code to decide exactly how I want to insert this in. I should hopefully have a patch by mid next week.
> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
> Key: MAPREDUCE-2723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to
mrv2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071557#comment-13071557 ]
Hudson commented on MAPREDUCE-2723:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #757 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/757/])
MAPREDUCE-2723. Remove directly accessing FSNamesystem.LOG from TestCopyFiles and TestDistCh.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151349
Files :
* /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestCopyFiles.java
* /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestDistCh.java
* /hadoop/common/trunk/mapreduce/CHANGES.txt
> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
> Key: MAPREDUCE-2723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to
mrv2
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069695#comment-13069695 ]
Arun C Murthy commented on MAPREDUCE-2723:
------------------------------------------
I think this is the right direction...
The AM should request for disk in addition to memory via ResourceRequest.
In terms of timelines, I think it's a little early to right *now*. I'd rather wait for a .1 release to get this feature in to ensure we spend enough time stabilizing the core of the RM right now.
Does that make sense? Thanks.
> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
> Key: MAPREDUCE-2723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to
mrv2
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071712#comment-13071712 ]
Robert Joseph Evans commented on MAPREDUCE-2723:
------------------------------------------------
It looks like someone got the CHANGES.txt and checkin comments wrong. Those changes do not look like they have anything to do with this JIRA.
> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
> Key: MAPREDUCE-2723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to
mrv2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071768#comment-13071768 ]
Hudson commented on MAPREDUCE-2723:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #750 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/750/])
MAPREDUCE-2723. Remove directly accessing FSNamesystem.LOG from TestCopyFiles and TestDistCh.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151349
Files :
* /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestCopyFiles.java
* /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/tools/TestDistCh.java
* /hadoop/common/trunk/mapreduce/CHANGES.txt
> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
> Key: MAPREDUCE-2723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to
mrv2
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071730#comment-13071730 ]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-2723:
---------------------------------------------------
> It looks like someone got the CHANGES.txt and checkin comments wrong. ...
Oops, it was my mistake. Will fix CHANGE.txt. Thanks Robert for pointing it out.
> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
> Key: MAPREDUCE-2723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to
mrv2
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071717#comment-13071717 ]
Robert Joseph Evans commented on MAPREDUCE-2723:
------------------------------------------------
Those notes are for MAPREDUCE-2732 (I just found it)
> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
> Key: MAPREDUCE-2723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to
mrv2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13071745#comment-13071745 ]
Hudson commented on MAPREDUCE-2723:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #759 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/759/])
Fix a typo in CHANGE.txt: MAPREDUCE-2723 should be MAPREDUCE-2732.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151464
Files :
* /hadoop/common/trunk/mapreduce/CHANGES.txt
> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
> Key: MAPREDUCE-2723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to
mrv2
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082072#comment-13082072 ]
Hudson commented on MAPREDUCE-2723:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #751 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/751/])
Fix a typo in CHANGE.txt: MAPREDUCE-2723 should be MAPREDUCE-2732.
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151464
Files :
* /hadoop/common/trunk/mapreduce/CHANGES.txt
> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
> Key: MAPREDUCE-2723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to
mrv2
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2723:
-------------------------------------
Fix Version/s: (was: 0.23.1)
0.24.0
> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
> Key: MAPREDUCE-2723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.24.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to schedule a container. For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira