You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Jerome Boulon (JIRA)" <ji...@apache.org> on 2009/01/15 20:12:59 UTC
[jira] Created: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Add late initialization to the chukwa log4j appender
-----------------------------------------------------
Key: HADOOP-5058
URL: https://issues.apache.org/jira/browse/HADOOP-5058
Project: Hadoop Core
Issue Type: Improvement
Reporter: Jerome Boulon
With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerome Boulon updated HADOOP-5058:
----------------------------------
Release Note:
A junit is included but if you want to see the default behavior:
Edit: src/contrib/chukwa/src/test/org/apache/hadoop/chukwa/inputtools/log4j/late-log4j.properties
and change it according to:
log4j.appender.R=org.apache.log4j.RollingFileAppender
#log4j.appender.R=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
Status: Patch Available (was: In Progress)
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Attachments: HADOOP-5058.patch
>
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677895#action_12677895 ]
Hadoop QA commented on HADOOP-5058:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12401153/HADOOP-5058.patch
against trunk revision 748861.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 release audit. The applied patch generated 799 release audit warnings (more than the trunk's current 798 warnings).
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/29/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/29/artifact/trunk/current/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/29/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/29/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/29/console
This message is automatically generated.
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Attachments: HADOOP-5058.patch
>
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-5058:
------------------------------
Status: Open (was: Patch Available)
Resubmit for hudson test.
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Attachments: HADOOP-5058-2.patch, HADOOP-5058.patch
>
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678163#action_12678163 ]
Ari Rabkin commented on HADOOP-5058:
------------------------------------
I don't understand why we need the chmod -- what are the files created as, by default?
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Attachments: HADOOP-5058.patch
>
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678170#action_12678170 ]
Jerome Boulon commented on HADOOP-5058:
---------------------------------------
It depends on the user umask and since chukwaAgent is not running as a privilege user, most of the time you'll not have access to it.
BTW, this has been there for a while in Log4JMetricsContext.java
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Attachments: HADOOP-5058.patch
>
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerome Boulon updated HADOOP-5058:
----------------------------------
Component/s: contrib/chukwa
Assignee: Jerome Boulon
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerome Boulon updated HADOOP-5058:
----------------------------------
Attachment: HADOOP-5058.patch
- Move clientFinalizer definition to the top
- Delayed file initialization/creation until first log event
- do a chmod 640 on the file in order to give read access to Chukwa.
This should be revisit when we will start working on security.
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Attachments: HADOOP-5058.patch
>
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HADOOP-5058 started by Jerome Boulon.
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678505#action_12678505 ]
Ari Rabkin commented on HADOOP-5058:
------------------------------------
Let's give it a whirl! +1
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Attachments: HADOOP-5058.patch
>
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12678109#action_12678109 ]
Jerome Boulon commented on HADOOP-5058:
---------------------------------------
.TestChukwaDailyRollingFileAppender.testlateInit4ChukwaDailyRollingFileAppender failed because I forgot to commit the latest version of build.xml
org.apache.hadoop.chukwa.datacollection.adaptor.filetailer.TestStartAtOffset.testStartAtOffset is failing because of a bug not related to this patch, fileName is "0 /tmp/chukwaTest" instead of "/tmp/chukwaTest". My guess is that this bug has been introduced by HADOOP-5087
(The regex in FileTailingAdaptor is no longer valid with HADOOP-5087)
org.apache.hadoop.mapred.TestJobSysDirWithDFS.testWithDFS & org.apache.hadoop.mapred.TestJobHistory.testJobHistoryUserLogLocation are not related to that patch
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Attachments: HADOOP-5058.patch
>
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated HADOOP-5058:
------------------------------
Release Note:
What is new in HADOOP-5058:
Added late initialization for Chukwa Log4j appender for Hadoop metrics and audit log appender.
was:
A junit is included but if you want to see the default behavior:
Edit: src/contrib/chukwa/src/test/org/apache/hadoop/chukwa/inputtools/log4j/late-log4j.properties
and change it according to:
log4j.appender.R=org.apache.log4j.RollingFileAppender
#log4j.appender.R=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
Status: Patch Available (was: Open)
Comment from previous release note:
A junit is included but if you want to see the default behavior:
Edit: src/contrib/chukwa/src/test/org/apache/hadoop/chukwa/inputtools/log4j/late-log4j.properties
and change it according to:
log4j.appender.R=org.apache.log4j.RollingFileAppender
#log4j.appender.R=org.apache.hadoop.chukwa.inputtools.log4j.ChukwaDailyRollingFileAppender
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Attachments: HADOOP-5058-2.patch, HADOOP-5058.patch
>
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5058) Add late initialization to the
chukwa log4j appender
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerome Boulon updated HADOOP-5058:
----------------------------------
Attachment: HADOOP-5058-2.patch
copy late-log4j.properties to the right location
> Add late initialization to the chukwa log4j appender
> -----------------------------------------------------
>
> Key: HADOOP-5058
> URL: https://issues.apache.org/jira/browse/HADOOP-5058
> Project: Hadoop Core
> Issue Type: Improvement
> Components: contrib/chukwa
> Reporter: Jerome Boulon
> Assignee: Jerome Boulon
> Attachments: HADOOP-5058-2.patch, HADOOP-5058.patch
>
>
> With the current Chukwa log4j implementation, if you define a static log4j configuration the appender is initialized at creation time.
> This may cause some problems for example with Hadoop Audit log or Metrics context, if the log4j properties are statically defined then a userX running an hadoop command will trigger a permission denied exception since the appender will try to initialize itself and therefore try to create a file under userX ownership for each appender even if not data is going to be written to it.
> The goal is to delay this initialization until the first message is actually written to that log so this kind of issue could easily be avoided.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.