You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Mahadev konar (JIRA)" <ji...@apache.org> on 2011/03/26 02:41:05 UTC
[jira] [Created] (MAPREDUCE-2405) MR-279: Implement uber-AppMaster
(in-cluster LocalJobRunner for MRv2)
MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
---------------------------------------------------------------------
Key: MAPREDUCE-2405
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2405
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: Mahadev konar
Assignee: Greg Roelofs
Fix For: 0.23.0
"Port" MAPREDUCE-1220 to MRv2. This is an optimization for small jobs wherein all tasks run on the same node in the same JVM/container.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2405) MR-279: Implement
uber-AppMaster (in-cluster LocalJobRunner for MRv2)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017249#comment-13017249 ]
Greg Roelofs commented on MAPREDUCE-2405:
-----------------------------------------
Thanks, Mahadev! You're clearly a person of outstanding character, intelligence, and better-than-average looks. :-)
(I'll just note here that there's further work/fixes to be done, particularly including counters, UI, the memory and input-size uber-decision criteria, and AM-restart on task-attempt failures, but I'll file a follow-up JIRA for that.)
> MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-2405
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2405
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Mahadev konar
> Assignee: Greg Roelofs
> Fix For: 0.23.0
>
> Attachments: MR-2405-MR-1220-yarn.v10.MR-279-hadoop-yarn.patch.txt, MR-2405-MR-1220-yarn.v11.MR-279-hadoop-yarn.patch.txt, MR-2405-MR-1220-yarn.v8.MR-279-hadoop-yarn.patch.txt
>
>
> "Port" MAPREDUCE-1220 to MRv2. This is an optimization for small jobs wherein all tasks run on the same node in the same JVM/container.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2405) MR-279: Implement uber-AppMaster
(in-cluster LocalJobRunner for MRv2)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-2405:
------------------------------------
Attachment: MR-2405-MR-1220-yarn.v11.MR-279-hadoop-yarn.patch.txt
Merged with MAPREDUCE-2414, boilerplate added, additional minor cleanups. This should be ready for commit to the MR-279 branch.
> MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-2405
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2405
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Mahadev konar
> Assignee: Greg Roelofs
> Fix For: 0.23.0
>
> Attachments: MR-2405-MR-1220-yarn.v10.MR-279-hadoop-yarn.patch.txt, MR-2405-MR-1220-yarn.v11.MR-279-hadoop-yarn.patch.txt, MR-2405-MR-1220-yarn.v8.MR-279-hadoop-yarn.patch.txt
>
>
> "Port" MAPREDUCE-1220 to MRv2. This is an optimization for small jobs wherein all tasks run on the same node in the same JVM/container.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2405) MR-279: Implement uber-AppMaster
(in-cluster LocalJobRunner for MRv2)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-2405:
------------------------------------
Attachment: MR-2405-MR-1220-yarn.v8.MR-279-hadoop-yarn.patch.txt
Preliminary patch: still full of debug noise, FIXMEs, TODOs, etc., but reasonably functional at the level of current testing.
This passes all current unit tests except testFailingMapper() in TestMRJobs, and even that one basically works except for an Avro NPE after the job completes and the AM shuts down.
> MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-2405
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2405
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Mahadev konar
> Assignee: Greg Roelofs
> Fix For: 0.23.0
>
> Attachments: MR-2405-MR-1220-yarn.v8.MR-279-hadoop-yarn.patch.txt
>
>
> "Port" MAPREDUCE-1220 to MRv2. This is an optimization for small jobs wherein all tasks run on the same node in the same JVM/container.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2405) MR-279: Implement
uber-AppMaster (in-cluster LocalJobRunner for MRv2)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011568#comment-13011568 ]
Greg Roelofs commented on MAPREDUCE-2405:
-----------------------------------------
As with MAPREDUCE-1220's "UberTask", which essentially runs small jobs sequentially within a single Task (and therefore in a single JVM), the MRv2 version does so within an "UberAppMaster"--which is really just the regular MRAppMaster with two container services overridden (allocator, launcher). The analogue to UberTask's run() method is LocalContainerLauncher's SubtaskRunner.run(), which hooks into the state machines and executes the subtasks sequentially.
This design is much cleaner than that in MR-1220 since the subtasks are "real" and can communicate directly with external entities, and the uber-AM is the regular AM and requires no special UI handling (though we'll want to flag uberized AMs and jobs in the UI somehow). UberTask, on the other hand, had to translate its subtasks' status-updates and was very awkward to integrate with the UI (is it a ReduceTask or an UberTask? -- the cascade of required changes was huge and never completed).
Kudos go to Sharad Agarwal for this design.
> MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-2405
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2405
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Mahadev konar
> Assignee: Greg Roelofs
> Fix For: 0.23.0
>
>
> "Port" MAPREDUCE-1220 to MRv2. This is an optimization for small jobs wherein all tasks run on the same node in the same JVM/container.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2405) MR-279: Implement
uber-AppMaster (in-cluster LocalJobRunner for MRv2)
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011941#comment-13011941 ]
Sharad Agarwal commented on MAPREDUCE-2405:
-------------------------------------------
The overall direction looks good.
Noticed in JobImpl:
{code}
.addTransition(JobState.INITED, JobState.KILL_WAIT,
JobEventType.JOB_KILL,
KILL_NEW_JOB_TRANSITION)
{code}
Kill event on INITED state should directly get to KILLED state. Also it should not use KILL_NEW_JOB_TRANSITION because it does not call the abort logic. Here we need a new Transition which executes abort.
> MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-2405
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2405
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Mahadev konar
> Assignee: Greg Roelofs
> Fix For: 0.23.0
>
> Attachments: MR-2405-MR-1220-yarn.v8.MR-279-hadoop-yarn.patch.txt
>
>
> "Port" MAPREDUCE-1220 to MRv2. This is an optimization for small jobs wherein all tasks run on the same node in the same JVM/container.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2405) MR-279: Implement uber-AppMaster
(in-cluster LocalJobRunner for MRv2)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-2405:
------------------------------------
Attachment: MR-2405-MR-1220-yarn.v10.MR-279-hadoop-yarn.patch.txt
De-personalized, cleaned up, and a better unit test.
Unfortunately, the git mirror is hosed, so I can't merge with the recently committed MAPREDUCE-2414 (i.e., this won't apply). And I just remembered that I forgot to add boilerplate to the three or four new files in here. But other than those caveats (and keeping in mind that this is a first cut, as previously noted), this is ready for pre-commit review.
> MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-2405
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2405
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Mahadev konar
> Assignee: Greg Roelofs
> Fix For: 0.23.0
>
> Attachments: MR-2405-MR-1220-yarn.v10.MR-279-hadoop-yarn.patch.txt, MR-2405-MR-1220-yarn.v8.MR-279-hadoop-yarn.patch.txt
>
>
> "Port" MAPREDUCE-1220 to MRv2. This is an optimization for small jobs wherein all tasks run on the same node in the same JVM/container.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-2405) MR-279: Implement uber-AppMaster
(in-cluster LocalJobRunner for MRv2)
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar resolved MAPREDUCE-2405.
--------------------------------------
Resolution: Fixed
I just pushed this to MR-279 branch. thanks greg.
> MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-2405
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2405
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Mahadev konar
> Assignee: Greg Roelofs
> Fix For: 0.23.0
>
> Attachments: MR-2405-MR-1220-yarn.v10.MR-279-hadoop-yarn.patch.txt, MR-2405-MR-1220-yarn.v11.MR-279-hadoop-yarn.patch.txt, MR-2405-MR-1220-yarn.v8.MR-279-hadoop-yarn.patch.txt
>
>
> "Port" MAPREDUCE-1220 to MRv2. This is an optimization for small jobs wherein all tasks run on the same node in the same JVM/container.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2405) MR-279: Implement uber-AppMaster
(in-cluster LocalJobRunner for MRv2)
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2405:
-------------------------------------
Release Note: An efficient implementation of small jobs by running all tasks in the MR ApplicationMaster JVM, there-by affecting lower latency.
> MR-279: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
> ---------------------------------------------------------------------
>
> Key: MAPREDUCE-2405
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2405
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Reporter: Mahadev konar
> Assignee: Greg Roelofs
> Fix For: 0.23.0
>
> Attachments: MR-2405-MR-1220-yarn.v10.MR-279-hadoop-yarn.patch.txt, MR-2405-MR-1220-yarn.v11.MR-279-hadoop-yarn.patch.txt, MR-2405-MR-1220-yarn.v8.MR-279-hadoop-yarn.patch.txt
>
>
> "Port" MAPREDUCE-1220 to MRv2. This is an optimization for small jobs wherein all tasks run on the same node in the same JVM/container.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira