You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Patrick Hunt (Created) (JIRA)" <ji...@apache.org> on 2012/03/07 20:52:57 UTC
[jira] [Created] (HADOOP-8149) cap space usage of default log4j
rolling policy
cap space usage of default log4j rolling policy
------------------------------------------------
Key: HADOOP-8149
URL: https://issues.apache.org/jira/browse/HADOOP-8149
Project: Hadoop Common
Issue Type: Improvement
Components: conf
Reporter: Patrick Hunt
I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
I recommend we move to an appender which:
1) caps the max file size (configurable)
2) caps the max number of files to keep (configurable)
3) uses rolling file appender rather than DRFA, see the warning here:
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Attachment: HADOOP-8149.patch
The attached patch demonstrates the changes I'm suggesting.
I also noticed that the package/template log4j file was out of sync somewhat with the main file, I attempted to rectify this (lmk if that's not correct).
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Attachments: HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240803#comment-13240803 ]
Hadoop QA commented on HADOOP-8149:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12520335/HADOOP-8149.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/794//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/794//console
This message is automatically generated.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Kihwal Lee (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225663#comment-13225663 ]
Kihwal Lee commented on HADOOP-8149:
------------------------------------
This is a nice change. There is one thing I want to point out though. There can be external tools and processes that depend on the daily rolling of logs. It will be better if we let them continue until they are ready to transition and modify the tools.
There are couple of places in the scripts where values are hardcoded (e.g. hadoop-daemon.sh and hadoop-env.sh). If we can change these to be dynamic while making RFA to be the default, we can accommodate the people who want to retain the old behavior by simply letting them have custom configs.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Status: Open (was: Patch Available)
Yes, will do. Canceling the patch while I work on splitting it out.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Attachments: HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Philip Zeyliger (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231380#comment-13231380 ]
Philip Zeyliger commented on HADOOP-8149:
-----------------------------------------
5GB seems like a reasonable number to me. I'm excited that our defaults may no longer fill up our hard disks when stuff is awry.
Thanks!
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Release Note: Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively. (was: Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER environment variable to use DRFA.)
Status: Patch Available (was: Open)
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Rajiv Chittajallu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13233690#comment-13233690 ]
Rajiv Chittajallu commented on HADOOP-8149:
-------------------------------------------
bq. 5GB seems like a reasonable number to me. I'm excited that our defaults may no longer fill up our hard disks when stuff is awry.
from my experience in running hadoop, having sufficient logs helped us in debugging issues when there are issues. With the current logging and error reporting of hadoop daemons, I wouldn't recommend log4j purging logs, especially by space, to any one.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224658#comment-13224658 ]
Hadoop QA commented on HADOOP-8149:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517443/HADOOP-8149.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/683//console
This message is automatically generated.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Attachments: HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236101#comment-13236101 ]
Eli Collins commented on HADOOP-8149:
-------------------------------------
Pat and I sync up wrt Kihwal's point about HADOOP_NAMENODE_OPTS and his bullet #2 above.
- HADOOP_NAMENODE_OPTS, HADOOP_SECONDARYNAMENODE_OPTS, and HADOOP_DATANODE_OPTS all hard-code the logger values and instead use an environment variable. The NN and the 2NN are the same and can use HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER instead of needing new service specific variables (this is how yarn works).
- HADOOP_DATANODE_OPTS uses ERROR,DRFAS instead of INFO,DRFAS like the NN and 2NN. It was originally created this way, anyone know why? Seems like it should use the same setting as the NN and 2NN and be exported like them as well. If that the case then we can use HADOOP_ROOT/SECURITY_LOGGER here as well.
- Filed HADOOP-8200 with a patch to remove HADOOP_JOBTRACKER/TASKTRACKER_OPTS which are no longer relevant on trunk/23 so we can ignore these here
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Rajiv Chittajallu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228699#comment-13228699 ]
Rajiv Chittajallu commented on HADOOP-8149:
-------------------------------------------
This change should be an option not the default. This is going to break lot of current tools.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225657#comment-13225657 ]
Patrick Hunt commented on HADOOP-8149:
--------------------------------------
Notice that I did change the appender name from DRFA to RFA, this will likely require doc changes somewhere...
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-8149:
--------------------------------
Resolution: Fixed
Fix Version/s: 2.0.0
Target Version/s: (was: 0.23.3)
Release Note: Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings of hadoop.root.logger and hadoop.security.logger can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively. (was: Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively.)
Status: Resolved (was: Patch Available)
I've committed this and merged to branch-2.
Thanks Pat! Thank you Kihwal and Rajiv for the feedback!
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Fix For: 2.0.0
>
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225668#comment-13225668 ]
Hadoop QA commented on HADOOP-8149:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517633/HADOOP-8149.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.ipc.TestRPCCallBenchmark
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/690//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/690//console
This message is automatically generated.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Suresh Srinivas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229201#comment-13229201 ]
Suresh Srinivas commented on HADOOP-8149:
-----------------------------------------
bq. We don't ship with a default logger that fills up partitions, has known data loss issues
Have you noticed cases of data loss issues. Because I have not heard of these instances.
As regards to filling up partitions, many of the installations have tools to handle this and not let either logs or some other files to fill up the partitions.
As I said, I like the idea of providing this as a default option during fresh install. But during upgrade, changing the existing appender is problematic.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241782#comment-13241782 ]
Hudson commented on HADOOP-8149:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1962 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1962/])
HADOOP-8149. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Fix For: 2.0.0
>
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins reassigned HADOOP-8149:
-----------------------------------
Assignee: Patrick Hunt
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins reassigned HADOOP-8149:
-----------------------------------
Assignee: Eli Collins (was: Patrick Hunt)
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Eli Collins
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Status: Patch Available (was: Open)
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Attachment: HADOOP-8149.patch
I believe this updated patch addresses everyone's concerns:
1) upped the cap to 5gb (256M * 20)
2) addressed the security logger concerns for nn/2nn OPTS
3) I left DN as is for the time being per Kihwal's comment (indeed the SecurityLogger seems noisy).
4) HADOOP_ROOT_LOGGER is handled in hadoop-config.sh and not overridden by the nn/2nn OPTS, so I believe we're good with that as well.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241763#comment-13241763 ]
Hudson commented on HADOOP-8149:
--------------------------------
Integrated in Hadoop-Common-trunk-Commit #1949 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1949/])
HADOOP-8149. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Fix For: 2.0.0
>
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229845#comment-13229845 ]
Eli Collins commented on HADOOP-8149:
-------------------------------------
Per Kihwal above (correct me if I'm wrong!) they're OK with the log4.properties changes since they can easily set these with their own file, it's just the unconditional export of _LOGGER in hadoop-daemon.sh that we need to address.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Attachment: HADOOP-8149.patch
Updated per the review comments.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Kihwal Lee (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231298#comment-13231298 ]
Kihwal Lee commented on HADOOP-8149:
------------------------------------
Please make hadoop-env.sh honor user-specified log appender.
For example, when the name node is started through scripts, the last logger setting on the command line will be from $HADOOP_NAMENODE_OPTS, which is currently hardcoded in hadoop-env.sh. This will override whatever user specified. (last one wins)
As a side note, we need to fix the shell scripts so that they don't source the same thing again and again and duplicate or triplicate the same command line options. Users can easily be confused about when something is set, eval'ed or overridden. But this is beyond the scope of this jira.
As for the cap, we need to keep in mind that hadoop components are very noisy. I see hadoop service processes generating multiple gigabytes of log messages per day. So 1 GB feels too small, but, for the same reason, the default may need to be small enough to keep single-node desktop users from running into trouble. I will throw a number to get it going. How about 5GB? 5 GB * 5-8 processes <= 40 GB.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242314#comment-13242314 ]
Hudson commented on HADOOP-8149:
--------------------------------
Integrated in Hadoop-Hdfs-trunk #1000 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1000/])
HADOOP-8149. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106)
Result = FAILURE
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Fix For: 2.0.0
>
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229428#comment-13229428 ]
Eli Collins commented on HADOOP-8149:
-------------------------------------
bq. Have you noticed cases of data loss issues. Because I have not heard of these instances.
No, me neither. We do see the issue with partitions filling up frequently, people expect this to work out of the box.
bq. They should set the logger to RFA only when HADOOP_*_LOGGER is not already set
Sounds like a plan. Pat, how about updating the patch to (a) add back the DRFA log4j.properties section and (b) on set HADOOP_*_LOGGER if they're already unset so they can be overriden in hadoop-env.sh?
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13242344#comment-13242344 ]
Hudson commented on HADOOP-8149:
--------------------------------
Integrated in Hadoop-Mapreduce-trunk #1035 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1035/])
HADOOP-8149. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Fix For: 2.0.0
>
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Status: Patch Available (was: Open)
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Attachments: HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Attachment: HADOOP-8149.patch
Updated patch with common only in this patch. I re-verified by running from tarball with DEBUG turned on and the maxfilesize set to just 10k.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-8149:
--------------------------------
Release Note: Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively.
Hadoop Flags: Incompatible change,Reviewed
+1 to the latest patch. I've updated the release notes field with information on how to configure the old behavior for users who don't want to switch.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Kihwal Lee (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229198#comment-13229198 ]
Kihwal Lee commented on HADOOP-8149:
------------------------------------
I am fine with the default being RFA. But the latest patch still hardcodes the logger names in hadoop-env.sh and hadoop-daemons.sh, which will unconditionally override custom env var. They should set the logger to RFA only when HADOOP_*_LOGGER is not already set. I am not very concerned with the changes in log4j.properties, as it can be easily replaced by users, but leaving the DRFA settings won't hurt either and make some people's life easier.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241759#comment-13241759 ]
Hudson commented on HADOOP-8149:
--------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2024 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2024/])
HADOOP-8149. Cap space usage of default log4j rolling policy. Contributed by Patrick Hunt (Revision 1307106)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1307106
Files :
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemon.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/conf/log4j.properties
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/log4j.properties
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Fix For: 2.0.0
>
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Status: Open (was: Patch Available)
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13237980#comment-13237980 ]
Eli Collins commented on HADOOP-8149:
-------------------------------------
@Kihwal, does the above sound good to you?
@Pat, I committed HADOOP-8200, and think we can make HADOOP_DATANODE_OPTS use INFO,DRFAS, so you should just need to update the patch so HADOOP_ROOT/SECURITY_LOGGER are respected (ie HADOOP_*_OPTS should use the env instead of hard-coding them)
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Status: Open (was: Patch Available)
Sounds reasonable. Notice that in this patch DRFAS was renamed to RFAS and some of the appenders (DRFAS/JSA/MRAUDIT/RMSUMMARY) are changed from DailyRolling to just Rolling. These are fine or should some action be taken wrt backward compatibility?
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HADOOP-8149:
--------------------------------
Target Version/s: 0.23.3
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239010#comment-13239010 ]
Patrick Hunt commented on HADOOP-8149:
--------------------------------------
I've created HADOOP-8216 in order to address the DRFAAUDIT missing from the main log4j.properties.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins reassigned HADOOP-8149:
-----------------------------------
Assignee: Patrick Hunt (was: Eli Collins)
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13234846#comment-13234846 ]
Eli Collins commented on HADOOP-8149:
-------------------------------------
If you don't purge logs then eventually the partition fills up and your cluster fails. That's not an option. If you want to retain lots of logs then just bump the retention size in the config to whatever the partition your logs reside on can support and you're good.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224924#comment-13224924 ]
Eli Collins commented on HADOOP-8149:
-------------------------------------
+1 Great change, and the patch looks good. Pat mentioned to me offline that he tested via running a Hadoop tarball and verifying that it rolls when the log is at the configurable amount, and that it keeps the configurable of files, ie 100mb and 10, which seem like good defaults.
Jenkins wants the patches to be split, per project, so you'll need a MR and HDFS jira for those changes. If you create those I'll re-review the splitup patches.
Thanks,
Eli
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Attachments: HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Release Note: Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER environment variable to use DRFA. (was: Hadoop log files are now rolled by size instead of date (daily) by default. Tools that depend on the log file name format will need to be updated. Users who would like to maintain the previous settings can use their current log4j.properties files and update the HADOOP_ROOT_LOGGER and HADOOP_SECURITY_LOGGER environment variables to use DRFA and DRFAS respectively.)
Status: Patch Available (was: Open)
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241398#comment-13241398 ]
Patrick Hunt commented on HADOOP-8149:
--------------------------------------
In addition to running the unit tests I packaged everything up and ran that in the following scenarios:
a) with the default hadoop-env.sh and log4j.properties, everything looked fine here, info level logs, etc...
b) I added HADOOP_ROOT_LOGGER to the top of hadoop-env.sh with DEBUG logging turned on, I set the max file size to 10k and the max files to 3. started/stopped the daemons and verified that the files sizes were capped, and that rolling started once we reached 3 log files (backups)
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228910#comment-13228910 ]
Eli Collins commented on HADOOP-8149:
-------------------------------------
How about we add back the DRFA section in the log4j.properties as it currently is? This way:
# People can keep the current behavior by just updating the {{HADOOP_*_LOGGER}} env vars to "DRFA"
# We don't ship with a default logger that fills up partitions, has known data loss issues
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Hunt updated HADOOP-8149:
---------------------------------
Attachment: HADOOP-8149.patch
5) I realized I missed keeping DRFAS in addition to keeping DRFA. It's in this latest patch.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Philip Zeyliger (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224804#comment-13224804 ]
Philip Zeyliger commented on HADOOP-8149:
-----------------------------------------
+1 (from a non-committer).
I think this would be a great improvement. The previous defaults easily led to HDFS and MR eating up your disk if they're stuck in an error loop, and I've seen that happen quite often.
-- Philip
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Attachments: HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241313#comment-13241313 ]
Eli Collins commented on HADOOP-8149:
-------------------------------------
Also, mind chiming in wrt what testing you've done?
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224672#comment-13224672 ]
Patrick Hunt commented on HADOOP-8149:
--------------------------------------
Jenkins doesn't like me, but I'd still like to get some feedback if this approach is acceptable or not.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Attachments: HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Suresh Srinivas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226427#comment-13226427 ]
Suresh Srinivas commented on HADOOP-8149:
-----------------------------------------
Guys, can you please give me some time to look at this. I like the idea of capping the space used for logs. However, changing the appender and its impact on the file names etc. could break the existing tools/management around logs.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225706#comment-13225706 ]
Eli Collins commented on HADOOP-8149:
-------------------------------------
Kihwal,
Good point. Given that the 23.x releases are currently alpha/beta, introduce new daemons and logs, etc, isn't this a good time for people to update? People can update their log4j.properties and env variables if they'd like to stay on the current behavior, no code changes should be required.
Thanks,
Eli
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230718#comment-13230718 ]
Patrick Hunt commented on HADOOP-8149:
--------------------------------------
Hm. Well not all of them. Snuck in while I was testing. :-)
Suresh et al. suggestions on cap?
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Patrick Hunt (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231421#comment-13231421 ]
Patrick Hunt commented on HADOOP-8149:
--------------------------------------
Seems to be some consensus around 5gb. I'll update the patch to use max index of 20, with max file size of 256mb.
@kihwal - You are specifically referring to this line, correct?
bq. export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,RFAAUDIT $HADOOP_NAMENODE_OPTS"
where setting the security logger is overridden.
I was hesitant to change hadoop-env.sh for a couple reasons, appreciate your help on this.
1) my understanding is that hadoop-env.sh is customized by the end user. They would edit this file with specific requirements/customizations, but that the default should handle the common case.
2) notice the following four lines, they are a hodgepodge of inconsistent threshold/appender settings. Are you suggesting that we override all of these based on the HADOOP_SECURITY_LOGGER setting?
Esp wrt item 2) I was thinking along the lines of your other comment (re scope) - that addressing this would be good to break off into a separate jira focused exclusively on that issue. However perhaps I'm mistaken and there's a simple solution here that I'm missing.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Suresh Srinivas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230690#comment-13230690 ]
Suresh Srinivas commented on HADOOP-8149:
-----------------------------------------
Admins probably do not want on upgrade this change. At least in release notes we should cover this. Not sure your above comment addresses that, Eli. Also don't you guys think the cap size of 1GB is way too small.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8149) cap space usage of default log4j
rolling policy
Posted by "Kihwal Lee (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239550#comment-13239550 ]
Kihwal Lee commented on HADOOP-8149:
------------------------------------
Eli and Pat,
Thanks for fixing the script bugs. What you suggested sounds good. As for HADOOP_DATANODE_OPTS, it might have been set that way because INFO makes it too noisy and affects performance. The production clusters I've seen keep it at ERROR. But if you have seen INFO being used in production without causing insane log bloat or performance issues, then I agree we should use INFO. If unsure, we can leave it as ERROR and file a non-blocking jira to investigate. I tried running a DN at INFO and it seems the BlockPoolSliceScanner is the noisiest one, probably because the period is set to 0 by default.
> cap space usage of default log4j rolling policy
> ------------------------------------------------
>
> Key: HADOOP-8149
> URL: https://issues.apache.org/jira/browse/HADOOP-8149
> Project: Hadoop Common
> Issue Type: Improvement
> Components: conf
> Reporter: Patrick Hunt
> Assignee: Patrick Hunt
> Attachments: HADOOP-8149.patch, HADOOP-8149.patch, HADOOP-8149.patch
>
>
> I've seen several critical production issues because logs are not automatically removed after some time and accumulate. Changes to Hadoop's default log4j file appender would help with this.
> I recommend we move to an appender which:
> 1) caps the max file size (configurable)
> 2) caps the max number of files to keep (configurable)
> 3) uses rolling file appender rather than DRFA, see the warning here:
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html
> Specifically: "DailyRollingFileAppender has been observed to exhibit synchronization issues and data loss."
> We'd lose (based on the default log4j configuration) the daily rolling aspect, however increase reliability.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira