[jira] [Commented] (MAPREDUCE-2424) MR-279: counters/UI/etc. for uber-AppMaster (in-cluster LocalJobRunner for MRv2)

["Greg Roelofs (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020521#comment-13020521 ] 

Greg Roelofs commented on MAPREDUCE-2424:
-----------------------------------------

Thanks, Mahadev!

We discussed the laundry list offline; each one really should be a separate JIRA, but I don't have the time to do that at the moment.  The (now committed) patch above addressed only the first item and the final third of the second item.  Which is to say, the following remain to be done:

* GUI indicators
** RM all-containers level
** multi-job app level [if exists]
* fix uber-decision ("is this a small job?"):
** memory criterion
** input-bytes criterion
* disable speculation
* isUber() method (somewhere) for unit tests to use
* delete (most of) old UberTask code (MAPREDUCE-1220; came in with initial MR-279 branch)
* implement a non-RPC, local version of umbilical
* AM restart (default 4 tries) on another node on any task-attempt failure
* uber-specific metrics?
* rename configurables? (still "ubertask"-based)

Mahadev also indicated a desire to put the subtasks back into their own JVMs (for better isolation), assuming the performance cost isn't too great.  And as noted above, a better approach for the counters would be job-level counters (not task-level as implemented), since uber-AM is really a job-level feature.

> MR-279: counters/UI/etc. for uber-AppMaster (in-cluster LocalJobRunner for MRv2)
> --------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2424
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2424
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>            Reporter: Greg Roelofs
>            Assignee: Greg Roelofs
>             Fix For: 0.23.0
>
>         Attachments: MR-2424-MR-1220-yarn.v1.MR-279-hadoop-yarn.patch.txt, MR-2424-MR-1220-yarn.v2.MR-279-hadoop-yarn.patch.txt
>
>
> Polish uber-AM (MAPREDUCE-2405).  Specifically:
> * uber-specific counters ("command-line UI")
> * GUI indicators
> ** RM all-containers level
> ** multi-job app level [if exists]
> ** single-job level
> * fix uber-decision ("is this a small job?"):
> ** memory criterion
> ** input-bytes criterion
> * disable speculation
> * isUber() method (somewhere) for unit tests to use
> * delete (most of) old UberTask code (MAPREDUCE-1220; came in with initial MR-279 branch)
> * implement non-RPC, local version of umbilical
> * AM restart (default 4 tries) on another node on any task-attempt failure
> * uber-specific metrics?
> * rename configurables? (still "ubertask"-based)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira