You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ahmed Radwan (JIRA)" <ji...@apache.org> on 2011/08/08 21:22:27 UTC
[jira] [Created] (MAPREDUCE-2787) MR-279: Performance improvement
in running Uber MapTasks
MR-279: Performance improvement in running Uber MapTasks
--------------------------------------------------------
Key: MAPREDUCE-2787
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2787
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
The runUberMapTasks() in org.apache.hadoop.mapred.UberTask obtains the local fileSystem and local job configuration for every task attempt. This will have a negative performance impact.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2787) MR-279: Performance improvement
in running Uber MapTasks
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan resolved MAPREDUCE-2787.
-------------------------------------
Resolution: Won't Fix
Thanks Arun and Vinod for the clarification. I am closing the ticket.
> MR-279: Performance improvement in running Uber MapTasks
> --------------------------------------------------------
>
> Key: MAPREDUCE-2787
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2787
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2787.patch
>
>
> The runUberMapTasks() in org.apache.hadoop.mapred.UberTask obtains the local fileSystem and local job configuration for every task attempt. This will have a negative performance impact.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2787) MR-279: Performance improvement
in running Uber MapTasks
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2787:
------------------------------------
Attachment: MAPREDUCE-2787.patch
The attached patch fixes the described issue by only creating the FileSystem and Configuration once for all task attempts.
All mapreduce unit tests ran successfully.
> MR-279: Performance improvement in running Uber MapTasks
> --------------------------------------------------------
>
> Key: MAPREDUCE-2787
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2787
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2787.patch
>
>
> The runUberMapTasks() in org.apache.hadoop.mapred.UberTask obtains the local fileSystem and local job configuration for every task attempt. This will have a negative performance impact.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2787) MR-279: Performance improvement
in running Uber MapTasks
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083010#comment-13083010 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2787:
----------------------------------------------------
Just to give more clarity to what Arun said: even though there are patches on MR-279 branch which implement the uber-task feature for the classic runtime (JT/TT), those patches are not going to be ported to trunk when we merge MR-279 to trunk. OTOH, the uber-task feature for yarn+MR runtime is implemented via LocalContainerAllocator and LocalContainerLauncher which is what you should look at.
bq. Should we close this as won't fix?
+1. Ahmed, please close this once you are convinced. Thanks!
> MR-279: Performance improvement in running Uber MapTasks
> --------------------------------------------------------
>
> Key: MAPREDUCE-2787
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2787
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2787.patch
>
>
> The runUberMapTasks() in org.apache.hadoop.mapred.UberTask obtains the local fileSystem and local job configuration for every task attempt. This will have a negative performance impact.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2787) MR-279: Performance improvement
in running Uber MapTasks
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082823#comment-13082823 ]
Arun C Murthy commented on MAPREDUCE-2787:
------------------------------------------
Ahmed - UberTask isn't used anymore in MR-279. The AM uses a different LocalContainerLauncher and LocalContainerAllocator in-lieu of UberTask. Please do take a look at the newer impl. Thanks.
Should we close this as won't fix?
> MR-279: Performance improvement in running Uber MapTasks
> --------------------------------------------------------
>
> Key: MAPREDUCE-2787
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2787
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2787.patch
>
>
> The runUberMapTasks() in org.apache.hadoop.mapred.UberTask obtains the local fileSystem and local job configuration for every task attempt. This will have a negative performance impact.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira