You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Chris Jansen <ch...@cognitomobile.com> on 2011/08/12 16:26:37 UTC
Log4j Appender Pattern Layout
Hello all,
I've spent a few hours today playing with the new Log4j avro appender,
thanks to Jonathan et al for getting that in the latest release. It's
seems to work very reliably, however it only seems to take the message
body of a log4j event and ignores the other elements. This is a bit of a
problem for me as our software tracing relies on displaying the class,
thread, MDC and stacktrace entries for debugging purposes. Essentially
flume is ignoring the log4j pattern.
I had a look in the source and found that the Log4JEventAdaptor class
and found that only the message of the event was being used, and the
layout ignored. So I created a new constructor to allow me to pass in a
Log4j layout to the event adaptor, I also modified the getBody method to
use the layout if the layout object is not null and I added some code to
print the stacktrace if it is present. See my hack below:
public Log4JEventAdaptor(LoggingEvent evt, Layout layout) {
super();
this.evt = evt;
this.layout = layout;
// This is needed to differentiate between events at the same
millisecond.
this.nanos = System.nanoTime();
}
public byte[] getBody() {
if (layout == null) {
return evt.getRenderedMessage().getBytes();
} else {
String body = layout.format(evt);
String[] s = evt.getThrowableStrRep();
if (s != null) {
int len = s.length;
for (int i = 0; i < len; i++) {
body = body + s[i];
body = body + Layout.LINE_SEP;
}
}
return body.getBytes();
}
}
So now I get formatted log4j entries in flume! My question is; was there
a reason for ignoring the log4j layout in the event adaptor and might
what I have done be useful to anyone else?
Also stack traces are appearing all on one line with the line separator
escape character instead of a new line. From what I have read the tail
source can cope with stacktraces and make them into a single event, is
it possible to do this with the log4j/avro source?
Thanks
Chris
NOTICE: Cognito Limited. Benham Valence, Newbury, Berkshire, RG20 8LU. UK. Company number 02723032. This e-mail message and any attachment is confidential. It may not be disclosed to or used by anyone other than the intended recipient. If you have received this e-mail in error please notify the sender immediately then delete it from your system. Whilst every effort has been made to check this mail is virus free we accept no responsibility for software viruses and you should check for viruses before opening any attachments. Opinions, conclusions and other information in this email and any attachments which do not relate to the official business of the company are neither given by the company nor endorsed by it.
This email message has been scanned for viruses by Mimecast
Re: Log4j Appender Pattern Layout
Posted by "ralph.goers @dslextreme.com" <ra...@dslextreme.com>.
I'm working on Log4J 2.0. I've written a Flume appender there that I would
like feedback on. The url is
https://svn.apache.org/repos/asf/logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers/log4j2-core/src/main/java/org/apache/logging/log4j/core/appender/flume/.
As the URL implies, this is still experimental code so I don't recommend it
for production, but if you are interested in helping out It would be most
welcome.
Ralph
On Mon, Aug 15, 2011 at 5:40 AM, Chris Jansen <
chris.jansen@cognitomobile.com> wrote:
> Thanks very much for the feedback Eric, I'm joining up to the developer
> mailing list and I've created a JIRA account so I can contribute my changes.
>
> I'll start enquiring on how the tail source copes with displaying multiline
> events so I can get that into the changes too.
>
> Rgds
>
> Chris
>
> -----Original Message-----
> From: Eric Sammer [mailto:esammer@cloudera.com]
> Sent: 12 August 2011 19:29
> To: flume-user@incubator.apache.org
> Subject: Re: Log4j Appender Pattern Layout
>
> Chris:
>
> On Fri, Aug 12, 2011 at 7:26 AM, Chris Jansen <
> chris.jansen@cognitomobile.com> wrote:
> > Hello all,
> >
> > I've spent a few hours today playing with the new Log4j avro appender,
> > thanks to Jonathan et al for getting that in the latest release. It's
> > seems to work very reliably, however it only seems to take the message
> > body of a log4j event and ignores the other elements. This is a bit of
> > a problem for me as our software tracing relies on displaying the
> > class, thread, MDC and stacktrace entries for debugging purposes.
> > Essentially flume is ignoring the log4j pattern.
> >
> > I had a look in the source and found that the Log4JEventAdaptor class
> > and found that only the message of the event was being used, and the
> > layout ignored. So I created a new constructor to allow me to pass in
> > a Log4j layout to the event adaptor, I also modified the getBody
> > method to use the layout if the layout object is not null and I added
> > some code to print the stacktrace if it is present. See my hack below:
> >
> > public Log4JEventAdaptor(LoggingEvent evt, Layout layout) {
> >
> > super();
> >
> > this.evt = evt;
> >
> > this.layout = layout;
> >
> > // This is needed to differentiate between events at the same
> > millisecond.
> >
> > this.nanos = System.nanoTime();
> >
> > }
> >
> >
> >
> > public byte[] getBody() {
> >
> > if (layout == null) {
> >
> > return evt.getRenderedMessage().getBytes();
> >
> > } else {
> >
> >
> >
> > String body = layout.format(evt);
> >
> >
> >
> > String[] s = evt.getThrowableStrRep();
> >
> > if (s != null) {
> >
> > int len = s.length;
> >
> > for (int i = 0; i < len; i++) {
> >
> > body = body + s[i];
> >
> > body = body + Layout.LINE_SEP;
> >
> > }
> >
> > }
> >
> >
> >
> > return body.getBytes();
> >
> > }
> >
> > }
> >
> > So now I get formatted log4j entries in flume! My question is; was
> > there a reason for ignoring the log4j layout in the event adaptor and
> > might what I have done be useful to anyone else?
>
> It wasn't a choice to ignore the layout, more a lack of knowledge of the
> finer points of writing a log4j appender. I think this would definitely be
> useful to others. A few other folks have mentions things like the MDC / NDC
> data (which I know less about) as something else Flume doesn't carry
> properly. Definitely check out
> https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute if
> you're interested in getting these changes back into Flume proper.
>
> (If you do decide to contribute, check out StringBuilder. :))
>
> > Also stack traces are appearing all on one line with the line
> > separator escape character instead of a new line. From what I have
> > read the tail source can cope with stacktraces and make them into a
> > single event, is it possible to do this with the log4j/avro source?
>
> Ideally, a single log4j event becomes a single Flume event. If the content
> of that event is multiple lines, that's fine. Flume should not mutate the
> content of the message.
>
> Thanks for the comments. The changes sound great.
>
> >
> >
> >
> > Thanks
> >
> >
> >
> > Chris
> >
> >
> >
> >
> > NOTICE: Cognito Limited. Benham Valence, Newbury, Berkshire, RG20 8LU.
> UK.
> > Company number 02723032. This e-mail message and any attachment is
> > confidential. It may not be disclosed to or used by anyone other than
> > the intended recipient. If you have received this e-mail in error
> > please notify the sender immediately then delete it from your system.
> > Whilst every effort has been made to check this mail is virus free we
> > accept no responsibility for software viruses and you should check for
> > viruses before opening any attachments. Opinions, conclusions and
> > other information in this email and any attachments which do not
> > relate to the official business of the company are neither given by the
> company nor endorsed by it.
> >
> > This email message has been scanned for viruses by Mimecast
>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>
>
>
RE: Log4j Appender Pattern Layout
Posted by Chris Jansen <ch...@cognitomobile.com>.
Thanks very much for the feedback Eric, I'm joining up to the developer mailing list and I've created a JIRA account so I can contribute my changes.
I'll start enquiring on how the tail source copes with displaying multiline events so I can get that into the changes too.
Rgds
Chris
-----Original Message-----
From: Eric Sammer [mailto:esammer@cloudera.com]
Sent: 12 August 2011 19:29
To: flume-user@incubator.apache.org
Subject: Re: Log4j Appender Pattern Layout
Chris:
On Fri, Aug 12, 2011 at 7:26 AM, Chris Jansen <ch...@cognitomobile.com> wrote:
> Hello all,
>
> I've spent a few hours today playing with the new Log4j avro appender,
> thanks to Jonathan et al for getting that in the latest release. It's
> seems to work very reliably, however it only seems to take the message
> body of a log4j event and ignores the other elements. This is a bit of
> a problem for me as our software tracing relies on displaying the
> class, thread, MDC and stacktrace entries for debugging purposes.
> Essentially flume is ignoring the log4j pattern.
>
> I had a look in the source and found that the Log4JEventAdaptor class
> and found that only the message of the event was being used, and the
> layout ignored. So I created a new constructor to allow me to pass in
> a Log4j layout to the event adaptor, I also modified the getBody
> method to use the layout if the layout object is not null and I added
> some code to print the stacktrace if it is present. See my hack below:
>
> public Log4JEventAdaptor(LoggingEvent evt, Layout layout) {
>
> super();
>
> this.evt = evt;
>
> this.layout = layout;
>
> // This is needed to differentiate between events at the same
> millisecond.
>
> this.nanos = System.nanoTime();
>
> }
>
>
>
> public byte[] getBody() {
>
> if (layout == null) {
>
> return evt.getRenderedMessage().getBytes();
>
> } else {
>
>
>
> String body = layout.format(evt);
>
>
>
> String[] s = evt.getThrowableStrRep();
>
> if (s != null) {
>
> int len = s.length;
>
> for (int i = 0; i < len; i++) {
>
> body = body + s[i];
>
> body = body + Layout.LINE_SEP;
>
> }
>
> }
>
>
>
> return body.getBytes();
>
> }
>
> }
>
> So now I get formatted log4j entries in flume! My question is; was
> there a reason for ignoring the log4j layout in the event adaptor and
> might what I have done be useful to anyone else?
It wasn't a choice to ignore the layout, more a lack of knowledge of the finer points of writing a log4j appender. I think this would definitely be useful to others. A few other folks have mentions things like the MDC / NDC data (which I know less about) as something else Flume doesn't carry properly. Definitely check out https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute if you're interested in getting these changes back into Flume proper.
(If you do decide to contribute, check out StringBuilder. :))
> Also stack traces are appearing all on one line with the line
> separator escape character instead of a new line. From what I have
> read the tail source can cope with stacktraces and make them into a
> single event, is it possible to do this with the log4j/avro source?
Ideally, a single log4j event becomes a single Flume event. If the content of that event is multiple lines, that's fine. Flume should not mutate the content of the message.
Thanks for the comments. The changes sound great.
>
>
>
> Thanks
>
>
>
> Chris
>
>
>
>
> NOTICE: Cognito Limited. Benham Valence, Newbury, Berkshire, RG20 8LU. UK.
> Company number 02723032. This e-mail message and any attachment is
> confidential. It may not be disclosed to or used by anyone other than
> the intended recipient. If you have received this e-mail in error
> please notify the sender immediately then delete it from your system.
> Whilst every effort has been made to check this mail is virus free we
> accept no responsibility for software viruses and you should check for
> viruses before opening any attachments. Opinions, conclusions and
> other information in this email and any attachments which do not
> relate to the official business of the company are neither given by the company nor endorsed by it.
>
> This email message has been scanned for viruses by Mimecast
--
Eric Sammer
twitter: esammer
data: www.cloudera.com
Re: Log4j Appender Pattern Layout
Posted by Eric Sammer <es...@cloudera.com>.
Chris:
On Fri, Aug 12, 2011 at 7:26 AM, Chris Jansen
<ch...@cognitomobile.com> wrote:
> Hello all,
>
> I’ve spent a few hours today playing with the new Log4j avro appender,
> thanks to Jonathan et al for getting that in the latest release. It’s seems
> to work very reliably, however it only seems to take the message body of a
> log4j event and ignores the other elements. This is a bit of a problem for
> me as our software tracing relies on displaying the class, thread, MDC and
> stacktrace entries for debugging purposes. Essentially flume is ignoring the
> log4j pattern.
>
> I had a look in the source and found that the Log4JEventAdaptor class and
> found that only the message of the event was being used, and the layout
> ignored. So I created a new constructor to allow me to pass in a Log4j
> layout to the event adaptor, I also modified the getBody method to use the
> layout if the layout object is not null and I added some code to print the
> stacktrace if it is present. See my hack below:
>
> public Log4JEventAdaptor(LoggingEvent evt, Layout layout) {
>
> super();
>
> this.evt = evt;
>
> this.layout = layout;
>
> // This is needed to differentiate between events at the same
> millisecond.
>
> this.nanos = System.nanoTime();
>
> }
>
>
>
> public byte[] getBody() {
>
> if (layout == null) {
>
> return evt.getRenderedMessage().getBytes();
>
> } else {
>
>
>
> String body = layout.format(evt);
>
>
>
> String[] s = evt.getThrowableStrRep();
>
> if (s != null) {
>
> int len = s.length;
>
> for (int i = 0; i < len; i++) {
>
> body = body + s[i];
>
> body = body + Layout.LINE_SEP;
>
> }
>
> }
>
>
>
> return body.getBytes();
>
> }
>
> }
>
> So now I get formatted log4j entries in flume! My question is; was there a
> reason for ignoring the log4j layout in the event adaptor and might what I
> have done be useful to anyone else?
It wasn't a choice to ignore the layout, more a lack of knowledge of
the finer points of writing a log4j appender. I think this would
definitely be useful to others. A few other folks have mentions things
like the MDC / NDC data (which I know less about) as something else
Flume doesn't carry properly. Definitely check out
https://cwiki.apache.org/confluence/display/FLUME/How+to+Contribute if
you're interested in getting these changes back into Flume proper.
(If you do decide to contribute, check out StringBuilder. :))
> Also stack traces are appearing all on one line with the line separator
> escape character instead of a new line. From what I have read the tail
> source can cope with stacktraces and make them into a single event, is it
> possible to do this with the log4j/avro source?
Ideally, a single log4j event becomes a single Flume event. If the
content of that event is multiple lines, that's fine. Flume should not
mutate the content of the message.
Thanks for the comments. The changes sound great.
>
>
>
> Thanks
>
>
>
> Chris
>
>
>
>
> NOTICE: Cognito Limited. Benham Valence, Newbury, Berkshire, RG20 8LU. UK.
> Company number 02723032. This e-mail message and any attachment is
> confidential. It may not be disclosed to or used by anyone other than the
> intended recipient. If you have received this e-mail in error please notify
> the sender immediately then delete it from your system. Whilst every effort
> has been made to check this mail is virus free we accept no responsibility
> for software viruses and you should check for viruses before opening any
> attachments. Opinions, conclusions and other information in this email and
> any attachments which do not relate to the official business of the company
> are neither given by the company nor endorsed by it.
>
> This email message has been scanned for viruses by Mimecast
--
Eric Sammer
twitter: esammer
data: www.cloudera.com