You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Chris Jansen (JIRA)" <ji...@apache.org> on 2011/08/15 14:59:27 UTC
[jira] [Created] (FLUME-743) Log4J Avro Appender Ignores Logging
Pattern
Log4J Avro Appender Ignores Logging Pattern
-------------------------------------------
Key: FLUME-743
URL: https://issues.apache.org/jira/browse/FLUME-743
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v0.9.4
Environment: Linux Ubuntu/Redhat
Reporter: Chris Jansen
Priority: Minor
The log4J Avro appender does not use the pattern layout to format event messages. It only takes the message body, meaning that important information such as thread name, MDC/NDC values and stacktrace are lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-743) Log4J Avro Appender Ignores Logging
Pattern
Posted by "Chris Jansen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085068#comment-13085068 ]
Chris Jansen commented on FLUME-743:
------------------------------------
I've been working on a fix for this as per the flume-user mailing list here: http://mail-archives.apache.org/mod_mbox/incubator-flume-user/201108.mbox/%3CCANpnisohwf6UqEgPyMxcFBSMutxXTvzbUVXPKe4A4=xxk+5XnQ@mail.gmail.com%3E
More than happy to be assigned this issue myself.
> Log4J Avro Appender Ignores Logging Pattern
> -------------------------------------------
>
> Key: FLUME-743
> URL: https://issues.apache.org/jira/browse/FLUME-743
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v0.9.4
> Environment: Linux Ubuntu/Redhat
> Reporter: Chris Jansen
> Priority: Minor
> Labels: log4j
>
> The log4J Avro appender does not use the pattern layout to format event messages. It only takes the message body, meaning that important information such as thread name, MDC/NDC values and stacktrace are lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-743) Log4J Avro Appender Ignores Logging
Pattern
Posted by "Chris Jansen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Jansen updated FLUME-743:
-------------------------------
Attachment: FLUME-743.diff
Code review: https://reviews.apache.org/r/1825/
> Log4J Avro Appender Ignores Logging Pattern
> -------------------------------------------
>
> Key: FLUME-743
> URL: https://issues.apache.org/jira/browse/FLUME-743
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v0.9.4
> Environment: Linux Ubuntu/Redhat
> Reporter: Chris Jansen
> Assignee: Chris Jansen
> Priority: Minor
> Labels: log4j
> Attachments: FLUME-743.diff
>
>
> The log4J Avro appender does not use the pattern layout to format event messages. It only takes the message body, meaning that important information such as thread name, MDC/NDC values and stacktrace are lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-743) Log4J Avro Appender Ignores Logging
Pattern
Posted by "E. Sammer (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
E. Sammer reassigned FLUME-743:
-------------------------------
Assignee: Chris Jansen
> Log4J Avro Appender Ignores Logging Pattern
> -------------------------------------------
>
> Key: FLUME-743
> URL: https://issues.apache.org/jira/browse/FLUME-743
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v0.9.4
> Environment: Linux Ubuntu/Redhat
> Reporter: Chris Jansen
> Assignee: Chris Jansen
> Priority: Minor
> Labels: log4j
>
> The log4J Avro appender does not use the pattern layout to format event messages. It only takes the message body, meaning that important information such as thread name, MDC/NDC values and stacktrace are lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-743) Log4J Avro Appender Ignores Logging
Pattern
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103535#comment-13103535 ]
jiraposter@reviews.apache.org commented on FLUME-743:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1825/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
Fix to allow the flume log4j avro appender to include the log4j pattern layout.
Updated the Log4JEventAdaptor constructor to require a layout object. As any class that uses the Log4JEventAdaptor extends the log4j AppenderSkeleton the layout object is always present. When a user wishes to configure a layout they use the setLayout method when setting up the log4j appender, if unset the layout will be defaulted to just displaying the event message body.
Updated the getBody() method of the Log4JEventAdaptor class to apply the layout to the log4j event and output stacktrace.
Updated FlumeLog4jAvroAppender to set up a default layout to display the message body if no user defined layout is specified.
Added a new test to TestFlumeLog4jAvroAppender to test a logging pattern.
Note that when using text sink only raw output will display stack traces on multiple lines, other formats display the newline escape characters.
This addresses bug FLUME-743.
https://issues.apache.org/jira/browse/FLUME-743
Diffs
-----
/trunk/flume-core/src/main/java/com/cloudera/flume/handlers/log4j/FlumeSeqfileAppender.java 1170098
/trunk/flume-core/src/main/java/com/cloudera/flume/handlers/log4j/Log4JAppenderEventSink.java 1170098
/trunk/flume-core/src/main/java/com/cloudera/flume/handlers/log4j/Log4JEventAdaptor.java 1170098
/trunk/flume-log4j-appender/src/main/java/com/cloudera/flume/log4j/appender/FlumeLog4jAvroAppender.java 1170098
/trunk/flume-log4j-appender/src/test/java/com/cloudera/flume/log4j/appender/TestFlumeLog4jAvroAppender.java 1170098
Diff: https://reviews.apache.org/r/1825/diff
Testing
-------
Set up two physical nodes, one as the source and one as the collector. Set up the source with an avro logical node listening on port 12345 with the sink being the collector. Set up a text sink on the collector to log to a file with the output format set to raw: text("/tmp/test.log","raw").
Use the test app code below to generate log events:
import com.cloudera.flume.log4j.appender.FlumeLog4jAvroAppender;
import org.apache.log4j.Logger;
import org.apache.log4j.MDC;
import org.apache.log4j.PatternLayout;
import java.util.UUID;
public class Main {
static Logger log = Logger.getLogger(Main.class);
public static void main(String[] args) {
try {
FlumeLog4jAvroAppender appender = new FlumeLog4jAvroAppender();
appender.setHostname("localhost");
appender.setPort(12345);
appender.setLayout(new PatternLayout("%d [%c] (%t) <%X{user} %X{field}> %m"));
appender.setReconnectAttempts(100);
appender.activateOptions();
log.addAppender(appender);
MDC.put("user", "chris");
while (true) {
MDC.put("field", UUID.randomUUID().toString());
log.info("Hello World");
try {
throw new Exception("Testing");
} catch (Exception e) {
log.error("Gone wrong ", e);
}
}
}
catch (Exception e) {
e.printStackTrace();
}
}
}
Thanks,
Chris
> Log4J Avro Appender Ignores Logging Pattern
> -------------------------------------------
>
> Key: FLUME-743
> URL: https://issues.apache.org/jira/browse/FLUME-743
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v0.9.4
> Environment: Linux Ubuntu/Redhat
> Reporter: Chris Jansen
> Assignee: Chris Jansen
> Priority: Minor
> Labels: log4j
>
> The log4J Avro appender does not use the pattern layout to format event messages. It only takes the message body, meaning that important information such as thread name, MDC/NDC values and stacktrace are lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-743) Log4J Avro Appender Ignores Logging
Pattern
Posted by "Chris Jansen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Jansen updated FLUME-743:
-------------------------------
Fix Version/s: v0.9.5
Release Note: Updated Log4J functionality to support layout formatting
Status: Patch Available (was: Open)
> Log4J Avro Appender Ignores Logging Pattern
> -------------------------------------------
>
> Key: FLUME-743
> URL: https://issues.apache.org/jira/browse/FLUME-743
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v0.9.4
> Environment: Linux Ubuntu/Redhat
> Reporter: Chris Jansen
> Assignee: Chris Jansen
> Priority: Minor
> Labels: log4j
> Fix For: v0.9.5
>
> Attachments: FLUME-743.diff
>
>
> The log4J Avro appender does not use the pattern layout to format event messages. It only takes the message body, meaning that important information such as thread name, MDC/NDC values and stacktrace are lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira