log4net duplicate filtering

[James Wilkinson <jw...@gmail.com>]

Hi,

I wanted to post the code I'd written for duplicate filtering a couple of
weeks ago, but got distracted with some other work. After noticing the
recent messages about duplicate filtering, I just figured that posting it in
its current state might be of use to people.

It's a modified forwarding appender that does duplicate event filtering with
a summary message inserted after the series of duplicate events has
completed. I started off trying to implement it as a filter, but hit the
problem of not being able to insert the summary message into the stream.

here's the source...
http://www.idiots.org.uk/log4net/DupeFilteringForewardingAppender_08-08-07.txt<http://www.idiots.org.uk/log4net/DupeFilteringForewardingAppender_08-07-07.txt>

Typical usage would be something like this...

  <!-- use this forwarding appender as a dupe filter which then passes on to
the console -->
  <appender name="DupeFilteringForwardingAppender" type="
log4net.Appender.DupeFilteringForwardingAppender" >
    <appender-ref ref="ConsoleAppender" />
    <FlushTimeout value="3" />
  </appender>

FlushTimeout (integer seconds) - see the comment in the source for the
public property for how this works.

There are a whole load of things I'm not really sure I've done very well.
The name of the thing for a start. The message fingerprinting might be more
efficiently done as a checksum and from what I know, I think the reliance on
the .net 2.0 generic-based list probably isn't in the spirit of the log4net
coding standards. The hard-coded way that event fingerprints are built up
feels like it needs thought. Anyway, it's maybe a start.

Also, thanks to Peter for clearing up my misconception about log4net
plugins.

James

Re: log4net duplicate filtering

[Jerry Shea <je...@gmail.com>]

That is a really useful bit of code - thanks for posting James.

I've done a bit of digging into using a filter to do this, as described in this
post<http://www.mail-archive.com/log4j-user@jakarta.apache.org/msg10314.html>.
I tried writing a DuplicateFilter which keeps track of the last message sent
through it and "Denies" if the current loggingEvent matches the previous.
Removing duplicates works fine except it is impossible to send the summary
log message at the end - like James I would like to raise a log message
saying "20 duplicate messages removed". I use this code inside the filter's
Decide method:

                    ILog sourceLogger = LogManager.GetLogger(
loggingEvent.LoggerName);
                    sourceLogger.Info("duplicate messages removed");

but this message never gets logged - some investigation shows that
AppenderSkeleton's m_recursiveGuard field is already set and so the appender
will not log a message raised in a filter.

This post<http://mail-archives.apache.org/mod_mbox/logging-log4j-user/200611.mbox/%253Cc83e39890611091345l30a43353pbe8fedc004c9b51e@mail.gmail.com%253E>seems
to indicate that log4j after
1.3 will allow this.


On 8/8/07, James Wilkinson <jw...@gmail.com> wrote:
>
> Hi,
>
> I wanted to post the code I'd written for duplicate filtering a couple of
> weeks ago, but got distracted with some other work. After noticing the
> recent messages about duplicate filtering, I just figured that posting it in
> its current state might be of use to people.
>
> It's a modified forwarding appender that does duplicate event filtering
> with a summary message inserted after the series of duplicate events has
> completed. I started off trying to implement it as a filter, but hit the
> problem of not being able to insert the summary message into the stream.
>
> here's the source...
> http://www.idiots.org.uk/log4net/DupeFilteringForewardingAppender_08-08-07.txt
>
> <http://www.idiots.org.uk/log4net/DupeFilteringForewardingAppender_08-07-07.txt>
>
> Typical usage would be something like this...
>
>   <!-- use this forwarding appender as a dupe filter which then passes on
> to the console -->
>   <appender name="DupeFilteringForwardingAppender" type="
> log4net.Appender.DupeFilteringForwardingAppender " >
>     <appender-ref ref="ConsoleAppender" />
>     <FlushTimeout value="3" />
>   </appender>
>
> FlushTimeout (integer seconds) - see the comment in the source for the
> public property for how this works.
>
> There are a whole load of things I'm not really sure I've done very well.
> The name of the thing for a start. The message fingerprinting might be more
> efficiently done as a checksum and from what I know, I think the reliance on
> the .net 2.0 generic-based list probably isn't in the spirit of the
> log4net coding standards. The hard-coded way that event fingerprints are
> built up feels like it needs thought. Anyway, it's maybe a start.
>
> Also, thanks to Peter for clearing up my misconception about log4net
> plugins.
>
> James
>