You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2011/07/06 19:56:16 UTC

[jira] [Created] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

MR279: Fate of finished Applications on RM
------------------------------------------

                 Key: MAPREDUCE-2649
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: mrv2
            Reporter: Thomas Graves
            Assignee: Thomas Graves


Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
the user experience saner. Users can revisit RM UI and check the status of their apps.

We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated MAPREDUCE-2649:
-------------------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)

I just committed this. Thanks Thomas!

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088851#comment-13088851 ] 

Arun C Murthy commented on MAPREDUCE-2649:
------------------------------------------

bq. Is this better in MR2 because the state on the RM is much smaller? Or is there still per-container state being retained?

It's better in MR2 since state-per-app is constant and very small, unlike in MR1 where state-per-app is unpredictable (depends on #tasks, Counters etc.)


> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated MAPREDUCE-2649:
-------------------------------------

    Status: Open  (was: Patch Available)

Thomas, sorry to come in late.

As you put it, it's much easier to do with the RMApp state-machine now, could you please use that? This way we don't need to write a lot of redundant code.

Also, seems to me that we could do with a much simpler model:
Just define a maximum number of in-memory completed jobs (say 10k) and not bother with jobs-per-user etc.

Thoughts? Do you see any issues with the simpler model?

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089193#comment-13089193 ] 

Arun C Murthy commented on MAPREDUCE-2649:
------------------------------------------

bq. I see no good way around this.

One option is to do this check in ClientRMService to shorten the window...

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089197#comment-13089197 ] 

Thomas Graves commented on MAPREDUCE-2649:
------------------------------------------

To be more clear its in the patch already..

+      ApplicationId applicationId = submissionContext.getApplicationId();
+      if (rmContext.getRMApps().get(applicationId) != null) {
         throw new IOException("Application with id " + applicationId
             + " is already present! Cannot add a duplicate!");
       }



> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081711#comment-13081711 ] 

Thomas Graves commented on MAPREDUCE-2649:
------------------------------------------

It might be a nice enhancement to have this more event driven - which appears easier now with RM reworked. When app finishes event could be sent to the expire monitor and it handles it. I think that can happen as a follow on change though.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072353#comment-13072353 ] 

Thomas Graves commented on MAPREDUCE-2649:
------------------------------------------

I've added a thread to the application manager that will retire the applications either when you hit a configurable time limit after app has finished or when the hit a max number per user.  How often the thread runs to check is also configurable.  The applications are simply removed from the RM application map and thus removed from the UI.

The new configs added are:

+  // time(in ms) between when the retire applications thread checks
<name>yarn.server.resourcemanager.retireApps.check.interval</name>

+  // the length of time(in ms) the RM keeps a completed application
<name>yarn.server.resourcemanager.retireApps.keep.period</name>

+  // the maximum number of completed applications per user RM keeps
 <name>yarn.server.resourcemanager.retireApps.user.completed.max</name>



> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>         Attachments: MAPREDUCE-2649-patch-mr279.txt
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Release Note: 
New config added:

   // the maximum number of completed applications the RM keeps <name>yarn.server.resourcemanager.expire.applications.completed.max</name>

  was:
The new configs added are:

  // time(in ms) between when the expire applications thread checks      <name>yarn.server.resourcemanager.expire.applications.monitor.interval</name>
 
  // the length of time(in ms) the RM keeps a completed application
<name>yarn.server.resourcemanager.expire.applications.interval</name>

   // the maximum number of completed applications per user RM keeps <name>yarn.server.resourcemanager.expire.applications.user.completed.max</name>

          Status: Patch Available  (was: Open)

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089208#comment-13089208 ] 

Arun C Murthy commented on MAPREDUCE-2649:
------------------------------------------

Good point, sorry I missed it.

I'll commit this asap. Thanks.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Attachment: MAPREDUCE-2649-v5.patch

removed some uneeded code.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13078308#comment-13078308 ] 

Robert Joseph Evans commented on MAPREDUCE-2649:
------------------------------------------------

Just a few comments about the patch.  

This is very minor but it would be good to make APPLICATION_FINISHTIME_COMPARATOR a static inner class of ApplicationsManagerImpl instead of an anonymous class.  It reduces the amount of unused data in the class, even though it is only instantiated once.

There is no error handling on RetireApplicationsThread.  If an Error or RuntimeException occurs then the thread will die silently.  However I cannot find an example of how it should be handled in the YARN code, so it might be fine the way it is.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083540#comment-13083540 ] 

Thomas Graves commented on MAPREDUCE-2649:
------------------------------------------

Reworked the patch to send an event when the RMapp completes and have the expirer handle it. It expires apps based on a maximum number of completed jobs.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089478#comment-13089478 ] 

Hudson commented on MAPREDUCE-2649:
-----------------------------------

Integrated in Hadoop-Hdfs-trunk #763 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/763/])
    MAPREDUCE-2649. Handling of finished applications in RM. Contributed by Thomas Graves.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521
Files : 
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java


> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089196#comment-13089196 ] 

Thomas Graves commented on MAPREDUCE-2649:
------------------------------------------

Actually I left that check in ClientRMService already..

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Attachment: MAPREDUCE-2649-v3.patch

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Status: Open  (was: Patch Available)

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086385#comment-13086385 ] 

Thomas Graves commented on MAPREDUCE-2649:
------------------------------------------

Thanks for the review, sorry for my delay responding, I was out.  

I agree, it makes sense to have an RMAppManager entity doing all the manipulations.  I'll make the changes here.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Attachment: MAPREDUCE-2649-v4.patch

updated the yarn-default.xml to include new config.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Attachment: MAPREDUCE-2649-patch-mr279.txt

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>         Attachments: MAPREDUCE-2649-patch-mr279.txt
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088090#comment-13088090 ] 

Todd Lipcon commented on MAPREDUCE-2649:
----------------------------------------

Coming late to see this patch. One question from an MR2 n00b: in MR1, these configs were very error prone since different jobs took up wildly variant amounts of RAM. So a value of 100 retained jobs would work fine for small jobs but fail miserably with OOME if bigger jobs were submitted. Then we added all of the limits to try to curtail this, but it's a bit of a stopgap rather than a real solution.

Is this better in MR2 because the state on the RM is much smaller? Or is there still per-container state being retained?

One thing we should consider is to adopt an interface like HBase's "HeapSize" -- objects that are going to be retained/cached need to implement it, and then the limit scan be set based on memory usage rather than strict counts.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Release Note: 
The new configs added are:

  // time(in ms) between when the expire applications thread checks      <name>yarn.server.resourcemanager.expire.applications.monitor.interval</name>
 
  // the length of time(in ms) the RM keeps a completed application
<name>yarn.server.resourcemanager.expire.applications.interval</name>

   // the maximum number of completed applications per user RM keeps <name>yarn.server.resourcemanager.expire.applications.user.completed.max</name>

  was:
The new configs added are:

+ // time(in ms) between when the retire applications thread checks
<name>yarn.server.resourcemanager.retireApps.check.interval</name>

+ // the length of time(in ms) the RM keeps a completed application
<name>yarn.server.resourcemanager.retireApps.keep.period</name>

+ // the maximum number of completed applications per user RM keeps
<name>yarn.server.resourcemanager.retireApps.user.completed.max</name>

          Status: Patch Available  (was: Open)

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089212#comment-13089212 ] 

Hudson commented on MAPREDUCE-2649:
-----------------------------------

Integrated in Hadoop-Common-trunk-Commit #771 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/771/])
    MAPREDUCE-2649. Handling of finished applications in RM. Contributed by Thomas Graves.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521
Files : 
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java


> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083548#comment-13083548 ] 

Mahadev konar commented on MAPREDUCE-2649:
------------------------------------------

Thomas, could you please add the property to yarn-default.xml as well. We need to populate our yarn-default.xml which currently is missing quite a few of the config knobs.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082900#comment-13082900 ] 

Thomas Graves commented on MAPREDUCE-2649:
------------------------------------------


I'm fine with the simpler model.  This algorithm was based on how the jobtracker does it now.  It will simplify things a lot and we won't need a separate thread. 

If no one else has objects I'll make the changes.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Status: Open  (was: Patch Available)

making changes to have common RMAppManager.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13088938#comment-13088938 ] 

Thomas Graves commented on MAPREDUCE-2649:
------------------------------------------

Since there isn't any test-patch script yet I ran all the tests.  They all passed:



Ran:
mvn clean install assembly:assembly 

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:[INFO]
[INFO] hadoop-yarn-api ................................... SUCCESS [21.917s]
[INFO] hadoop-yarn-common ................................ SUCCESS [9.999s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [7.904s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [35.353s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [38.603s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [7.361s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.069s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.079s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [6.682s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [5.401s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [2.287s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [1:39.490s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [5.548s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [3:24.055s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.052s]
[INFO] hadoop-mapreduce .................................. SUCCESS [58.003s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8:23.395s
[INFO] Finished at: Mon Aug 22 19:05:14 UTC 2011
[INFO] Final Memory: 41M/297M
[INFO] ------------------------------------------------------------------------


> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084569#comment-13084569 ] 

Arun C Murthy commented on MAPREDUCE-2649:
------------------------------------------

Thomas, the patch looks good, thanks!

One thing that bothers me, nothing related to your patch of course, is that RMContextImpl.applications is manipulated in ClientRMService (submitApplication adds an RMApp to the list) and now, with your patch, RMAppExpirer removes it from the same list.

Should we consider renaming RMAppExpirer to something like RMAppManager which does both the add and delete? Thus ClientRMService should send an event to RMAppManager which then creates the RMApp, adds to list and sends other notifications. Similarly APP_COMPLETED comes over to RMAppManager which then deletes RMApps as in your patch. 
Thoughts? I'm fine doing this in another jira too... but it might improve the overall system rightaway. Thanks.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Release Note: 
The new configs added are:

+ // time(in ms) between when the retire applications thread checks
<name>yarn.server.resourcemanager.retireApps.check.interval</name>

+ // the length of time(in ms) the RM keeps a completed application
<name>yarn.server.resourcemanager.retireApps.keep.period</name>

+ // the maximum number of completed applications per user RM keeps
<name>yarn.server.resourcemanager.retireApps.user.completed.max</name>
          Status: Patch Available  (was: Open)

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Fix Version/s: 0.23.0

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089487#comment-13089487 ] 

Hudson commented on MAPREDUCE-2649:
-----------------------------------

Integrated in Hadoop-Mapreduce-trunk #779 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/779/])
    MAPREDUCE-2649. Handling of finished applications in RM. Contributed by Thomas Graves.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521
Files : 
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java


> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Status: Patch Available  (was: Open)

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Status: Open  (was: Patch Available)

reworking patch since RM got changed a lot.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Status: Patch Available  (was: Open)

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Attachment: MAPREDUCE-2649-v2.patch

patch reworked. 

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Attachment: MAPREDUCE-2649-v6.patch

Updates to have RMAppManager.

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089191#comment-13089191 ] 

Arun C Murthy commented on MAPREDUCE-2649:
------------------------------------------

Thomas the patch looks fine, I fixed a minor package name problem for TestAppManager and ran all the tests.

One minor issue: looks like there is no way for the job-client to understand that it submitted the same app twice since the error is silently lost in RMAppManager with:

{code}

      if (rmContext.getRMApps().putIfAbsent(applicationId, application) != null) {
        LOG.info("Application with id " + applicationId + 
            " is already present! Cannot add a duplicate!");
        // don't send event through dispatcher as it will be handled by app already
        // present with this id.
        application.handle(new RMAppRejectedEvent(applicationId,
            "Application with this id is already present! Cannot add a duplicate!"));
        } else {
          // ...
{code}

Previously it was caught by the ClientRMService who then threw an exception...

I see no good way around this.

Thoughts?

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Updated] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Graves updated MAPREDUCE-2649:
-------------------------------------

    Attachment: MAPREDUCE-2649-v7.patch

add test for new state transition in RMAppImpl.java

> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

[jira] [Commented] (MAPREDUCE-2649) MR279: Fate of finished Applications on RM

Posted by "Hudson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/MAPREDUCE-2649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089211#comment-13089211 ] 

Hudson commented on MAPREDUCE-2649:
-----------------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #848 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/848/])
    MAPREDUCE-2649. Handling of finished applications in RM. Contributed by Thomas Graves.

acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1160521
Files : 
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-mr-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEvent.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMConfig.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerEventType.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManagerSubmitEvent.java
* /hadoop/common/trunk/hadoop-mapreduce/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java


> MR279: Fate of finished Applications on RM
> ------------------------------------------
>
>                 Key: MAPREDUCE-2649
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2649
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Thomas Graves
>            Assignee: Thomas Graves
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2649-patch-mr279.txt, MAPREDUCE-2649-v2.patch, MAPREDUCE-2649-v3.patch, MAPREDUCE-2649-v4.patch, MAPREDUCE-2649-v5.patch, MAPREDUCE-2649-v6.patch, MAPREDUCE-2649-v7.patch
>
>
> Today RM keeps the references of finished application for ever. Though this is not sustainable long term, it keeps
> the user experience saner. Users can revisit RM UI and check the status of their apps.
> We need to think of purging old references yet keeping the UX sane.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira