You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4net-user@logging.apache.org by Eric Marthinsen <em...@agilecommerce.com> on 2008/12/02 01:31:12 UTC
Best Way to Set Up Massive Number of Loggers
Hello-
I'm working on an app that coordinates data transfers among multiple trading
partners. I want to create a log file for each trading partner. I had a few
ideas of how to do this, but am not sure which would be considered the
optimal way. The expected number of trading partners is likely to be around
100, but could be higher. Here are my thoughts:
1. Add a bunch of named loggers to my log4net.config file
2. Do programmatic configuration of the loggers on application start
3. Write my own appender that is based on the RollingFileAppender
Do any of these scream out as being a good or bad solution? Is there a
better approach that I might be missing? Thanks in advance.
Regards,
Eric
Re: Best Way to Set Up Massive Number of Loggers
Posted by Ron Grabowski <ro...@yahoo.com>.
It would be nice if the File property on the FileAppender had more robust support for patterns so you could write to a dynamic file. NLog and log4j have had support for this for a while. I think they keep a pool of files open then close a file if it hasn't been used recently.
________________________________
From: Eric Marthinsen <em...@agilecommerce.com>
To: Log4NET User <lo...@logging.apache.org>
Sent: Monday, December 1, 2008 7:48:46 PM
Subject: Re: Best Way to Set Up Massive Number of Loggers
Hi Walden-
I like the grep idea. The motivation for the multiple log files is twofold. First, they get really big really quickly. I realize I can roll them on size, but it's nice to have a daily log file. Second, we have some utilities that can take a log file and replay previous transactions (we are logging a lot of data in custom formats). These are still programmer tools (and QA).
-Eric
On Mon, Dec 1, 2008 at 7:39 PM, Walden H. Leverich <Wa...@techsoftinc.com> wrote:
Since you're asking for thoughts... bad idea. To me log4net is
plumbing, that is it's programmer based stuff. Sounds to me like you're attempting
to use it to satisfy a user requirement. I think you'd be better off adding the
logging as part of the business processing. Now, having said that, there are
still times when it's nice to know which trading partner you're dealing with on
a log message. In that case I'd push the partner id into the NDC at the start
of the process, and pop it at the end. That way you'll have it in the log (if
you format it correctly). You'll have a single log file, but you can always
fgrep it. We do something similar with several levels of NDC and can quickly
come up with a issue-specific logfile using grep.
-Walden
--
Walden H Leverich III
Tech Software
(516) 627-3800 x3051
WaldenL@TechSoftInc.com
http://www.TechSoftInc.com
Quiquid latine dictum sit altum viditur.
(Whatever is said in Latin seems profound.)
From:Eric Marthinsen
[mailto:emarthinsen@agilecommerce.com]
Sent: Monday, December 01, 2008 7:31 PM
To: log4net-user@logging.apache.org
Subject: Best Way to Set Up Massive Number of Loggers
Hello-
I'm working on an app that coordinates data transfers among multiple trading
partners. I want to create a log file for each trading partner. I had a few
ideas of how to do this, but am not sure which would be considered the optimal
way. The expected number of trading partners is likely to be around 100, but
could be higher. Here are my thoughts:
1. Add a bunch of named loggers to my log4net.config file
2. Do programmatic configuration of the loggers on application start
3. Write my own appender that is based on the RollingFileAppender
Do any of these scream out as being a good or bad solution? Is there a better
approach that I might be missing? Thanks in advance.
Regards,
Eric
Re: Best Way to Set Up Massive Number of Loggers
Posted by Ron Grabowski <ro...@yahoo.com>.
If you don't want to type the same pattern over and over, extend PatternLayout and provide your own default pattern:
<layout type="Company.Logging.MyPatternLayout, Company.Logging" />
That can be overridden when necessary:
<layout type="Company.Logging.MyPatternLayout, Company.Logging">
<conversionPattern" value="%d %m%n" />
</layout>
You can also extend RollingFileAppender and set default values in there:
<appender name="Comp1" type="Company.Logging.CompRollingFileAppender, Company.Logging">
<comp value="1"/>
</appender>
<appender name="Comp2" type="Company.Logging.CompRollingFileAppender, Company.Logging">
<comp value="2"/>
</appender>
<appender name="Comp3" type="Company.Logging.CompRollingFileAppender, Company.Logging">
<comp value="3"/>
<layout type="Company.Logging.MyPatternLayout, Company.Logging">
<conversionPattern" value="%d %m%n" />
</layout>
</appender>
<appender name="Comp4" type="Company.Logging.CompRollingFileAppender, Company.Logging">
<comp value="4"/>
</appender>
----- Original Message ----
From: Loren Keagle <Lo...@aps-technology.com>
To: Log4NET User <lo...@logging.apache.org>
Sent: Tuesday, December 2, 2008 2:19:47 AM
Subject: Re: Best Way to Set Up Massive Number of Loggers
This is the only way I know how to do this in log4net. We do this to handle remote logging from embedded controllers. We use .NET remoting appenders in this case, so we get the HostName property of each remote host logging to the sink. But if you're local you could do the same thing with a string filter or other property filter. This gives you individual layout control over each appender, and hence each log source.
If only log4net would add in a layout-ref tag so we wouldn't have to repeat ourselves 75 times...
<log4net>
<appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
<layout type="log4net.Layout.PatternLayout">
<param name="ConversionPattern" value="%property{log4net:HostName} %d{MM/dd/yyyy HH:mm:ss.fff} [%p] %c{1}: %m%n" />
</layout>
</appender>
<appender name="Comp1" type="log4net.Appender.RollingFileAppender">
<file value="Comp1/Comp1.log" />
<staticLogFileName value="true" />
<filter type="log4net.Filter.PropertyFilter">
<Key value="log4net:HostName" />
<StringToMatch value="Comp1" />
</filter>
<filter type="log4net.Filter.DenyAllFilter" />
<appendToFile value="true" />
<rollingStyle value="Date" />
<datePattern value="yyyyMMdd’.log’" />
<layout type="log4net.Layout.PatternLayout">
<param name="ConversionPattern" value="%d{MM/dd/yyyy HH:mm:ss.fff} [%p] %c{1}: %m%n" />
</layout>
</appender>
<appender name="Comp2" type="log4net.Appender.RollingFileAppender">
<file value="Comp2/Comp2.log" />
<staticLogFileName value="true" />
<filter type="log4net.Filter.PropertyFilter">
<Key value="log4net:HostName" />
<StringToMatch value="Comp2" />
</filter>
<filter type="log4net.Filter.DenyAllFilter" />
<appendToFile value="true" />
<rollingStyle value="Date" />
<datePattern value="yyyyMMdd" />
<layout type="log4net.Layout.PatternLayout">
<param name="ConversionPattern" value="%d{MM/dd/yyyy HH:mm:ss.fff} [%p] %c{1}: %m%n" />
</layout>
</appender>
<appender name="Comp3" type="log4net.Appender.RollingFileAppender">
<file value="Comp3/Comp3.log" />
<staticLogFileName value="true" />
<filter type="log4net.Filter.PropertyFilter">
<Key value="log4net:HostName" />
<StringToMatch value="Comp3" />
</filter>
<filter type="log4net.Filter.DenyAllFilter" />
<appendToFile value="true" />
<rollingStyle value="Date" />
<datePattern value="yyyyMMdd" />
<layout type="log4net.Layout.PatternLayout">
<param name="ConversionPattern" value="%d{MM/dd/yyyy HH:mm:ss.fff} [%p] %c{1}: %m%n" />
</layout>
</appender>
...
<root>
<level value="ALL" />
<appender-ref ref="ConsoleAppender" />
<appender-ref ref="Comp1" />
<appender-ref ref="Comp2" />
<appender-ref ref="Comp3" />
...
</root>
</log4net>
Eric Marthinsen wrote:
> Hi Walden-
>
> I like the grep idea. The motivation for the multiple log files is twofold. First, they get really big really quickly. I realize I can roll them on size, but it's nice to have a daily log file. Second, we have some utilities that can take a log file and replay previous transactions (we are logging a lot of data in custom formats). These are still programmer tools (and QA).
>
> -Eric
>
>
> On Mon, Dec 1, 2008 at 7:39 PM, Walden H. Leverich <WaldenL@techsoftinc.com <ma...@techsoftinc.com>> wrote:
>
> Since you're asking for thoughts... bad idea. To me log4net is
> plumbing, that is it's programmer based stuff. Sounds to me like
> you're attempting to use it to satisfy a user requirement. I think
> you'd be better off adding the logging as part of the business
> processing. Now, having said that, there are still times when it's
> nice to know which trading partner you're dealing with on a log
> message. In that case I'd push the partner id into the NDC at the
> start of the process, and pop it at the end. That way you'll have
> it in the log (if you format it correctly). You'll have a single
> log file, but you can always fgrep it. We do something similar
> with several levels of NDC and can quickly come up with a
> issue-specific logfile using grep.
>
> -Walden
>
> --
> Walden H Leverich III
>
> Tech Software
>
> (516) 627-3800 x3051
>
> WaldenL@TechSoftInc.com <ma...@TechSoftInc.com>
>
> http://www.TechSoftInc.com <http://www.techsoftinc.com/>
>
> Quiquid latine dictum sit altum viditur.
>
> (Whatever is said in Latin seems profound.)
>
> *From:* Eric Marthinsen [mailto:emarthinsen@agilecommerce.com
> <ma...@agilecommerce.com>]
> *Sent:* Monday, December 01, 2008 7:31 PM
> *To:* log4net-user@logging.apache.org
> <ma...@logging.apache.org>
> *Subject:* Best Way to Set Up Massive Number of Loggers
>
> Hello-
>
> I'm working on an app that coordinates data transfers among
> multiple trading partners. I want to create a log file for each
> trading partner. I had a few ideas of how to do this, but am not
> sure which would be considered the optimal way. The expected
> number of trading partners is likely to be around 100, but could
> be higher. Here are my thoughts:
>
> 1. Add a bunch of named loggers to my log4net.config file
> 2. Do programmatic configuration of the loggers on application start
> 3. Write my own appender that is based on the RollingFileAppender
>
> Do any of these scream out as being a good or bad solution? Is
> there a better approach that I might be missing? Thanks in advance.
>
> Regards,
> Eric
>
>
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - http://www.avg.com Version: 8.0.176 / Virus Database: 270.9.12/1822 - Release Date: 12/1/2008 8:23 AM
>
>
Re: Best Way to Set Up Massive Number of Loggers
Posted by Loren Keagle <Lo...@aps-technology.com>.
This is the only way I know how to do this in log4net. We do this to
handle remote logging from embedded controllers. We use .NET remoting
appenders in this case, so we get the HostName property of each remote
host logging to the sink. But if you're local you could do the same
thing with a string filter or other property filter. This gives you
individual layout control over each appender, and hence each log source.
If only log4net would add in a layout-ref tag so we wouldn't have to
repeat ourselves 75 times...
<log4net>
<appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender">
<layout type="log4net.Layout.PatternLayout">
<param name="ConversionPattern" value="%property{log4net:HostName}
%d{MM/dd/yyyy HH:mm:ss.fff} [%p] %c{1}: %m%n" />
</layout>
</appender>
<appender name="Comp1" type="log4net.Appender.RollingFileAppender">
<file value="Comp1/Comp1.log" />
<staticLogFileName value="true" />
<filter type="log4net.Filter.PropertyFilter">
<Key value="log4net:HostName" />
<StringToMatch value="Comp1" />
</filter>
<filter type="log4net.Filter.DenyAllFilter" />
<appendToFile value="true" />
<rollingStyle value="Date" />
<datePattern value="yyyyMMdd’.log’" />
<layout type="log4net.Layout.PatternLayout">
<param name="ConversionPattern" value="%d{MM/dd/yyyy HH:mm:ss.fff} [%p]
%c{1}: %m%n" />
</layout>
</appender>
<appender name="Comp2" type="log4net.Appender.RollingFileAppender">
<file value="Comp2/Comp2.log" />
<staticLogFileName value="true" />
<filter type="log4net.Filter.PropertyFilter">
<Key value="log4net:HostName" />
<StringToMatch value="Comp2" />
</filter>
<filter type="log4net.Filter.DenyAllFilter" />
<appendToFile value="true" />
<rollingStyle value="Date" />
<datePattern value="yyyyMMdd" />
<layout type="log4net.Layout.PatternLayout">
<param name="ConversionPattern" value="%d{MM/dd/yyyy HH:mm:ss.fff} [%p]
%c{1}: %m%n" />
</layout>
</appender>
<appender name="Comp3" type="log4net.Appender.RollingFileAppender">
<file value="Comp3/Comp3.log" />
<staticLogFileName value="true" />
<filter type="log4net.Filter.PropertyFilter">
<Key value="log4net:HostName" />
<StringToMatch value="Comp3" />
</filter>
<filter type="log4net.Filter.DenyAllFilter" />
<appendToFile value="true" />
<rollingStyle value="Date" />
<datePattern value="yyyyMMdd" />
<layout type="log4net.Layout.PatternLayout">
<param name="ConversionPattern" value="%d{MM/dd/yyyy HH:mm:ss.fff} [%p]
%c{1}: %m%n" />
</layout>
</appender>
...
<root>
<level value="ALL" />
<appender-ref ref="ConsoleAppender" />
<appender-ref ref="Comp1" />
<appender-ref ref="Comp2" />
<appender-ref ref="Comp3" />
...
</root>
</log4net>
Eric Marthinsen wrote:
> Hi Walden-
>
> I like the grep idea. The motivation for the multiple log files is
> twofold. First, they get really big really quickly. I realize I can
> roll them on size, but it's nice to have a daily log file. Second, we
> have some utilities that can take a log file and replay previous
> transactions (we are logging a lot of data in custom formats). These
> are still programmer tools (and QA).
>
> -Eric
>
>
> On Mon, Dec 1, 2008 at 7:39 PM, Walden H. Leverich
> <WaldenL@techsoftinc.com <ma...@techsoftinc.com>> wrote:
>
> Since you're asking for thoughts... bad idea. To me log4net is
> plumbing, that is it's programmer based stuff. Sounds to me like
> you're attempting to use it to satisfy a user requirement. I think
> you'd be better off adding the logging as part of the business
> processing. Now, having said that, there are still times when it's
> nice to know which trading partner you're dealing with on a log
> message. In that case I'd push the partner id into the NDC at the
> start of the process, and pop it at the end. That way you'll have
> it in the log (if you format it correctly). You'll have a single
> log file, but you can always fgrep it. We do something similar
> with several levels of NDC and can quickly come up with a
> issue-specific logfile using grep.
>
> -Walden
>
> --
>
> Walden H Leverich III
>
> Tech Software
>
> (516) 627-3800 x3051
>
> WaldenL@TechSoftInc.com <ma...@TechSoftInc.com>
>
> http://www.TechSoftInc.com <http://www.techsoftinc.com/>
>
> Quiquid latine dictum sit altum viditur.
>
> (Whatever is said in Latin seems profound.)
>
> *From:* Eric Marthinsen [mailto:emarthinsen@agilecommerce.com
> <ma...@agilecommerce.com>]
> *Sent:* Monday, December 01, 2008 7:31 PM
> *To:* log4net-user@logging.apache.org
> <ma...@logging.apache.org>
> *Subject:* Best Way to Set Up Massive Number of Loggers
>
> Hello-
>
> I'm working on an app that coordinates data transfers among
> multiple trading partners. I want to create a log file for each
> trading partner. I had a few ideas of how to do this, but am not
> sure which would be considered the optimal way. The expected
> number of trading partners is likely to be around 100, but could
> be higher. Here are my thoughts:
>
> 1. Add a bunch of named loggers to my log4net.config file
> 2. Do programmatic configuration of the loggers on application start
> 3. Write my own appender that is based on the RollingFileAppender
>
> Do any of these scream out as being a good or bad solution? Is
> there a better approach that I might be missing? Thanks in advance.
>
> Regards,
> Eric
>
>
> ------------------------------------------------------------------------
>
>
> No virus found in this incoming message.
> Checked by AVG - http://www.avg.com
> Version: 8.0.176 / Virus Database: 270.9.12/1822 - Release Date: 12/1/2008 8:23 AM
>
>
Re: Best Way to Set Up Massive Number of Loggers
Posted by Eric Marthinsen <em...@agilecommerce.com>.
Hi Walden-
I like the grep idea. The motivation for the multiple log files is twofold.
First, they get really big really quickly. I realize I can roll them on
size, but it's nice to have a daily log file. Second, we have some utilities
that can take a log file and replay previous transactions (we are logging a
lot of data in custom formats). These are still programmer tools (and QA).
-Eric
On Mon, Dec 1, 2008 at 7:39 PM, Walden H. Leverich
<Wa...@techsoftinc.com>wrote:
> Since you're asking for thoughts... bad idea. To me log4net is plumbing,
> that is it's programmer based stuff. Sounds to me like you're attempting to
> use it to satisfy a user requirement. I think you'd be better off adding the
> logging as part of the business processing. Now, having said that, there are
> still times when it's nice to know which trading partner you're dealing with
> on a log message. In that case I'd push the partner id into the NDC at the
> start of the process, and pop it at the end. That way you'll have it in the
> log (if you format it correctly). You'll have a single log file, but you can
> always fgrep it. We do something similar with several levels of NDC and can
> quickly come up with a issue-specific logfile using grep.
>
>
>
> -Walden
>
>
>
> --
>
> Walden H Leverich III
>
> Tech Software
>
> (516) 627-3800 x3051
>
> WaldenL@TechSoftInc.com
>
> http://www.TechSoftInc.com <http://www.techsoftinc.com/>
>
>
>
> Quiquid latine dictum sit altum viditur.
>
> (Whatever is said in Latin seems profound.)
>
>
>
> *From:* Eric Marthinsen [mailto:emarthinsen@agilecommerce.com]
> *Sent:* Monday, December 01, 2008 7:31 PM
> *To:* log4net-user@logging.apache.org
> *Subject:* Best Way to Set Up Massive Number of Loggers
>
>
>
> Hello-
>
> I'm working on an app that coordinates data transfers among multiple
> trading partners. I want to create a log file for each trading partner. I
> had a few ideas of how to do this, but am not sure which would be considered
> the optimal way. The expected number of trading partners is likely to be
> around 100, but could be higher. Here are my thoughts:
>
> 1. Add a bunch of named loggers to my log4net.config file
> 2. Do programmatic configuration of the loggers on application start
> 3. Write my own appender that is based on the RollingFileAppender
>
> Do any of these scream out as being a good or bad solution? Is there a
> better approach that I might be missing? Thanks in advance.
>
> Regards,
> Eric
>
RE: Best Way to Set Up Massive Number of Loggers
Posted by "Walden H. Leverich" <Wa...@TechSoftInc.com>.
Since you're asking for thoughts... bad idea. To me log4net is plumbing,
that is it's programmer based stuff. Sounds to me like you're attempting
to use it to satisfy a user requirement. I think you'd be better off
adding the logging as part of the business processing. Now, having said
that, there are still times when it's nice to know which trading partner
you're dealing with on a log message. In that case I'd push the partner
id into the NDC at the start of the process, and pop it at the end. That
way you'll have it in the log (if you format it correctly). You'll have
a single log file, but you can always fgrep it. We do something similar
with several levels of NDC and can quickly come up with a issue-specific
logfile using grep.
-Walden
--
Walden H Leverich III
Tech Software
(516) 627-3800 x3051
WaldenL@TechSoftInc.com
http://www.TechSoftInc.com <http://www.techsoftinc.com/>
Quiquid latine dictum sit altum viditur.
(Whatever is said in Latin seems profound.)
From: Eric Marthinsen [mailto:emarthinsen@agilecommerce.com]
Sent: Monday, December 01, 2008 7:31 PM
To: log4net-user@logging.apache.org
Subject: Best Way to Set Up Massive Number of Loggers
Hello-
I'm working on an app that coordinates data transfers among multiple
trading partners. I want to create a log file for each trading partner.
I had a few ideas of how to do this, but am not sure which would be
considered the optimal way. The expected number of trading partners is
likely to be around 100, but could be higher. Here are my thoughts:
1. Add a bunch of named loggers to my log4net.config file
2. Do programmatic configuration of the loggers on application start
3. Write my own appender that is based on the RollingFileAppender
Do any of these scream out as being a good or bad solution? Is there a
better approach that I might be missing? Thanks in advance.
Regards,
Eric