You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2012/05/30 17:47:24 UTC
[jira] [Created] (MAPREDUCE-4296) remove old aggregated logs
Thomas Graves created MAPREDUCE-4296:
----------------------------------------
Summary: remove old aggregated logs
Key: MAPREDUCE-4296
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.3
Reporter: Thomas Graves
Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4296:
-------------------------------------------
Attachment: MR-4296.txt
OK I updated the tests, and I updated the code to just log the smaller error.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4296:
-------------------------------------------
Attachment: MR-4296.txt
Hopefully all issues are now addressed
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4296:
-------------------------------------------
Release Note:
When upgrading if log aggregation is enabled and yarn.log-aggregation.retain-seconds is set so that logs are going to be deleted, a cluster administrator should change the permissions on all user's logs to be 770.
hadoop fs -chmod -R 770 <log-dir>/*
Please be sure that <log-dir> itself is not modified.
Added in new config yarn.log-aggregation.retain-seconds.
How long to keep aggregation logs before deleting them. -1 disables. Be careful set this too small and you will spam the name node.
was:
When upgrading if log aggregation is enabled and yarn.log-aggregation.retain-seconds is set so that logs are going to be deleted, a cluster administrator should change the permissions on all user's logs to be 770.
hadoop fs -chmod -R 770 <log-dir>/*
Please be sure that <log-dir> itself is not modified.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4296:
-------------------------------------------
Target Version/s: 0.23.3, 2.1.0-alpha (was: 0.23.3)
Release Note:
When upgrading if log aggregation is enabled and yarn.log-aggregation.retain-seconds is so that logs are going to be deleted, a cluster administrator should change the permissions on all user's logs to be 770.
hadoop fs -chmod -R 770 <log-dir>/*
Please be sure that <log-dir> itself is not modified.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437028#comment-13437028 ]
Thomas Graves commented on MAPREDUCE-4296:
------------------------------------------
+1 Thanks Bobby!
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435367#comment-13435367 ]
Jason Lowe commented on MAPREDUCE-4296:
---------------------------------------
+1 (non-binding), lgtm. Thanks Bobby!
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435122#comment-13435122 ]
Robert Joseph Evans commented on MAPREDUCE-4296:
------------------------------------------------
Thanks for the review Jason I am fixing my fat fingered mistakes :)
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans reassigned MAPREDUCE-4296:
----------------------------------------------
Assignee: Robert Joseph Evans
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434659#comment-13434659 ]
Hadoop QA commented on MAPREDUCE-4296:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12540952/MR-4296.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2730//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2730//console
This message is automatically generated.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435180#comment-13435180 ]
Hadoop QA commented on MAPREDUCE-4296:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12541061/MR-4296.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2735//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2735//console
This message is automatically generated.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436167#comment-13436167 ]
Thomas Graves commented on MAPREDUCE-4296:
------------------------------------------
Mostly looks good. couple minor things:
* we should add new configs to the release notes.
* YarnConfiguration - the comment is wrong (+ /** Whether to enable log aggregation */) - should be to enable log aggregation deletion service.
* I'm on the wall about the 30 minute check. In one way its nice so users don't shoot themselves but at the same time it makes it harder to test and is less flexible. I think in the minimum we should document that we have the 30 minute config, otherwise perhaps just remove the check and add a comment that you shouldn't set it to low.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436999#comment-13436999 ]
Robert Joseph Evans commented on MAPREDUCE-4296:
------------------------------------------------
Will do. I'll remove the 30 min check, I don't think we really need it, but I'll put in a comment with it warning people.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4296:
-------------------------------------------
Target Version/s: 0.23.3 (was: 2.1.0-alpha)
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4296:
-------------------------------------------
Release Note:
When upgrading if log aggregation is enabled and yarn.log-aggregation.retain-seconds is set so that logs are going to be deleted, a cluster administrator should change the permissions on all user's logs to be 770.
hadoop fs -chmod -R 770 <log-dir>/*
Please be sure that <log-dir> itself is not modified.
was:
When upgrading if log aggregation is enabled and yarn.log-aggregation.retain-seconds is so that logs are going to be deleted, a cluster administrator should change the permissions on all user's logs to be 770.
hadoop fs -chmod -R 770 <log-dir>/*
Please be sure that <log-dir> itself is not modified.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13437030#comment-13437030 ]
Hadoop QA commented on MAPREDUCE-4296:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12541413/MR-4296.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2748//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2748//console
This message is automatically generated.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435262#comment-13435262 ]
Jason Lowe commented on MAPREDUCE-4296:
---------------------------------------
Getting close, a few more points:
* We're checking the permission of each app log directory, but the permission of the user log directory above them could prevent us from deleting the app directory.
* Checking for group write doesn't imply we'd still be able to delete. I'm thinking in particular of the setup issues discussed in MAPREDUCE-4557.
Rather than trying to avoid the errant deletes and potentially screwing up those checks, we could just proceed with the delete attempt and not log a giant exception if the error is a permission problem. We could still end up logging a line per application in a misconfigured user log directory, but that's better than logging dozens of lines for a full exception trace that isn't helpful for permission issues anyway.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Jason Lowe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434587#comment-13434587 ]
Jason Lowe commented on MAPREDUCE-4296:
---------------------------------------
Patch looks good overall, some cleanup comments:
* Probably need a release note stating the necessary chmod operations on user directories under the top-level aggregated log directory
* Speaking of which, is it worth checking the permissions of the user directory before we try to delete all the application logs underneath it? I'm thinking of all the exceptions that will be logged periodically if one user's log directory doesn't have the proper permissions to allow deletion but there are hundreds of application logs underneath it.
* With all the nested try/catch blocks, it might be easier to read if one or more of the {{for}} loops were factored out into private methods
* Spelling nits (many are end-user visible and should be cleaned up):
** retain is misspelled in yarn.log-aggregation.retian-seconds
** being is misspelled as beign
** "desabled because retiontion" should be "disabled because retention"
** aggregated misspelled as aggregatied
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4296:
-------------------------------------------
Attachment: MR-4296.txt
This patch adds in deletion of logs that are too old. If you are upgrading an older cluster you will need to run a chmod on all of the existing logs or the user running History Server will probably not have write permissions to the files.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435326#comment-13435326 ]
Hadoop QA commented on MAPREDUCE-4296:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12541087/MR-4296.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2737//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2737//console
This message is automatically generated.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4296:
-------------------------------------------
Status: Patch Available (was: Open)
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435279#comment-13435279 ]
Robert Joseph Evans commented on MAPREDUCE-4296:
------------------------------------------------
Yes I agree, it was ugly when I put that check in, and after thinking about it I saw the check as well. I will update it. The issue with outputting only a small message is with HADOOP-8661, so I will have to do something a bit ugly to make the message small, at least until HADOOP-8661 goes in. If this one goes in first I will probably just undo the uglyness in the that patch.
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4296) remove old aggregated logs
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4296:
-------------------------------------------
Attachment: MR-4296.txt
Addressed comments with the patch
> remove old aggregated logs
> --------------------------
>
> Key: MAPREDUCE-4296
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4296
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.3
> Reporter: Thomas Graves
> Assignee: Robert Joseph Evans
> Attachments: MR-4296.txt, MR-4296.txt
>
>
> Currently the aggregated user logs under NM_REMOTE_APP_LOG_DIR are never removed. We should have mechanism to remove them after certain period.
> It might make sense for job history server to remove them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira