You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ramya Sunil (JIRA)" <ji...@apache.org> on 2011/08/01 20:10:09 UTC
[jira] [Created] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
[MR-279] - Cleanup staging dir after job completion
---------------------------------------------------
Key: MAPREDUCE-2762
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Fix For: 0.23.0
The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2762:
-------------------------------------
Attachment: MAPREDUCE-2762.patch
Changed the patch to do cleanup after calling stop on all services in MRAppMaster.
- Added a job cleanup event which is synchronously called from the MRAppMaster Jobfinished handler.
We could also add a job api for cleanupo but either of those are not very clean implementation. The more I think, the earlier suggestion on sending events for de registering and then cleanup is probably much cleaner.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130568#comment-13130568 ]
Hudson commented on MAPREDUCE-2762:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #44 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/44/])
Merge -c 1185880 from trunk to branch-0.23 to complete fix for MAPREDUCE-2762.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185881
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130048#comment-13130048 ]
Hudson commented on MAPREDUCE-2762:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #16 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/16/])
Merge -c 1185880 from trunk to branch-0.23 to complete fix for MAPREDUCE-2762.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185881
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117002#comment-13117002 ]
Hadoop QA commented on MAPREDUCE-2762:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496967/MAPREDUCE-2762.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/889//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/889//console
This message is automatically generated.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130041#comment-13130041 ]
Hudson commented on MAPREDUCE-2762:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #17 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/17/])
Merge -c 1185880 from trunk to branch-0.23 to complete fix for MAPREDUCE-2762.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185881
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2762:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Mahadev!
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130043#comment-13130043 ]
Hudson commented on MAPREDUCE-2762:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1102 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1102/])
MAPREDUCE-2762. Cleanup MR staging directory on completion. Contributed by Mahadev Konar.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185880
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129004#comment-13129004 ]
Mahadev konar commented on MAPREDUCE-2762:
------------------------------------------
Yeah had thought abt that. I was trying to keep the MRAppMaster code clean and did not want to inject any code from Jobs in the MRAppMaster.
As Vinod tells me, the code is already tainted with the uber AM, so I'll go ahead and make the quick fix for cleaning the staging directories.
We'll have to rethink the MRAppMaster code when we do DAG of jobs in the AM.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2762:
-------------------------------------
Attachment: MAPREDUCE-2762.patch
Updated patch. Tested it on a cluster.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117304#comment-13117304 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2762:
----------------------------------------------------
This works for most of the cases, but there is a corner condition - if the staging files are deleted and then AM crashes for whatever reason before it unregisters with the RM, AM restart is affected. RM will happily restart the AM but the AM cannot recover because of absent job-files.
I think we should do the cleanup as the last step of the MR AM(main() ? ).
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117063#comment-13117063 ]
Arun C Murthy commented on MAPREDUCE-2762:
------------------------------------------
Mahadev - there are 2 calls to cleanupStagingDir - should we move the call to JobImpl.finished(JobState)? That way we do the cleanup in one place... ideally we'd have a single 'base transition' which called this.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2762:
-------------------------------------
Status: Patch Available (was: Open)
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128945#comment-13128945 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2762:
----------------------------------------------------
Why can't we simply move the {{cleanupStagingDir()}} operation into MRAppMaster itself from the Job. In that case, we don't need this complex event handling from Job into MRAppMaster and then back into Job. Essentially
{code}
class MRAppMasster {
.....
@Override
public void stop() {
super.stop(); // Stop all services
cleanupStagingDir(); // Cleanup the dir now.
}
.....
}
{code}
I get a feeling that there is a disconnect here and I may be missing something, am I?
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130004#comment-13130004 ]
Arun C Murthy commented on MAPREDUCE-2762:
------------------------------------------
+1
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2762:
-------------------------------------
Status: Open (was: Patch Available)
Forgot to add a class :(
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2762:
-------------------------------------
Status: Patch Available (was: Open)
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117738#comment-13117738 ]
Mahadev konar commented on MAPREDUCE-2762:
------------------------------------------
You are right Vinod.
Thinking abt this more, I think we should move the unregister call out of the stop() methods. We could make sure that as soon as unregister is called, we send an event back to JobImpl saying done with the RM, and then we can go ahead to delete the staging dir.
So, there'd be a state transition from
RUNNING->SUCCEEDED that would sent an unregister event to RMCommunicator.
and then
SUCCEEDED -> SUCCEEDED in JobImpl, that would happen on an event triggered after unregister is called, and the same for Killed and Failed, which would delete the staging directory.
Do you guys see any issues with that?
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2762:
-----------------------------------------------
Status: Open (was: Patch Available)
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2762:
-------------------------------------
Status: Patch Available (was: Open)
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130594#comment-13130594 ]
Hudson commented on MAPREDUCE-2762:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #865 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/865/])
MAPREDUCE-2762. Cleanup MR staging directory on completion. Contributed by Mahadev Konar.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185880
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2762:
-------------------------------------
Attachment: MAPREDUCE-2762.patch
Patch that cleans up the staging dir.
Added tests to make sure staging dir is cleaned up on kill/fail/success. Sadly took longer than expected time on writing/fixing the tests.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117069#comment-13117069 ]
Mahadev konar commented on MAPREDUCE-2762:
------------------------------------------
good idea. will take a look tomm morning.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2762:
-------------------------------------
Attachment: MAPREDUCE-2762.patch
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130058#comment-13130058 ]
Hudson commented on MAPREDUCE-2762:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1121 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1121/])
MAPREDUCE-2762. Cleanup MR staging directory on completion. Contributed by Mahadev Konar.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185880
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2762:
-------------------------------------
Priority: Blocker (was: Major)
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130046#comment-13130046 ]
Hudson commented on MAPREDUCE-2762:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1181 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1181/])
MAPREDUCE-2762. Cleanup MR staging directory on completion. Contributed by Mahadev Konar.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185880
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130602#comment-13130602 ]
Hudson commented on MAPREDUCE-2762:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #56 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/56/])
Merge -c 1185880 from trunk to branch-0.23 to complete fix for MAPREDUCE-2762.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185881
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2762:
-------------------------------------
Attachment: MAPREDUCE-2762.patch
Attached missing file.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13116188#comment-13116188 ]
Mahadev konar commented on MAPREDUCE-2762:
------------------------------------------
Preliminary patch that fixes the issue. Doing a little more manual testing and adding test cases.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129569#comment-13129569 ]
Hadoop QA commented on MAPREDUCE-2762:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12499513/MAPREDUCE-2762.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1713 javac compiler warnings (more than the trunk's current 1711 warnings).
-1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapreduce.TestJobMonitorAndPrint
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1053//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1053//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1053//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1053//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1053//console
This message is automatically generated.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122527#comment-13122527 ]
Mahadev konar commented on MAPREDUCE-2762:
------------------------------------------
@Vinod,
Do you see any issues with the proposed soln?
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2762:
-------------------------------------
Attachment: MAPREDUCE-2762.patch
Updated patch will cleanup in MRAppMaster. Still doing some testing.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130070#comment-13130070 ]
Hudson commented on MAPREDUCE-2762:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #18 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/18/])
Merge -c 1185880 from trunk to branch-0.23 to complete fix for MAPREDUCE-2762.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185881
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117010#comment-13117010 ]
Hadoop QA commented on MAPREDUCE-2762:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496969/MAPREDUCE-2762.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/890//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/890//console
This message is automatically generated.
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar reassigned MAPREDUCE-2762:
----------------------------------------
Assignee: Mahadev konar
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Fix For: 0.23.0
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122734#comment-13122734 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2762:
----------------------------------------------------
It is simpler than that. Cleanup should always be the last step so that recovery is possible. The control always returns to MRAppMaster, we can override stop() in MRAppMaster and do the cleanup after stopping all the components (super.stop()) without adding any new events.
What do you think?
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2762) [MR-279] - Cleanup staging dir
after job completion
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130621#comment-13130621 ]
Hudson commented on MAPREDUCE-2762:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #835 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/835/])
MAPREDUCE-2762. Cleanup MR staging directory on completion. Contributed by Mahadev Konar.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1185880
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationCleanup.java
> [MR-279] - Cleanup staging dir after job completion
> ---------------------------------------------------
>
> Key: MAPREDUCE-2762
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2762
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch, MAPREDUCE-2762.patch
>
>
> The files created under the staging dir have to be deleted after job completion. Currently, all job.* files remain forever in the ${yarn.apps.stagingDir}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira