You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4net-dev@logging.apache.org by Dominik Psenner <dp...@gmail.com> on 2012/06/08 08:39:41 UTC
Async logging
Hi,
In response to
https://issues.apache.org/jira/browse/LOG4NET-344?focusedCommentId=13291557
<https://issues.apache.org/jira/browse/LOG4NET-344?focusedCommentId=13291557
&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comme
nt-13291557>
&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comme
nt-13291557 I would like to start a discussion how log4net should support
asynchronous logging. Tom Tang implemented a general async forwarding
appender that glues together other appenders and thus adds general async
processing of log events. He implemented a good idea that misses only a few
small things to be an awesome improvement.
My objections are:
* the queue size should be configurable
* the ThreadPool should be used in favour over a designated worker
Thread
* if the queue is full, it should fallback to synchronous logging,
shouldn't it?
* queued log events should be processed when the appender gets
disposed
Cheers,
Dominik
Re: Async logging
Posted by Christian Grobmeier <gr...@gmail.com>.
Hopefully one of the log4net guys can apply that issue in near future :-)
If you have more such valuable contributions, they are much mor ethan
welcome - we are just a little short of manpower here
Cheers
On Thu, Jul 19, 2012 at 7:26 PM, George Chung <ge...@glympse.com> wrote:
> Hi,
>
> Thanks for taking a look at my “contribution”. J
>
>
>
> Just fyi, I am not a log4net developer…just a user who hacked up an
> AsyncSqlAppender. I am using it in a production environment with good
> results…feel free to use and comment.
>
>
>
> From what I can gather from the discussions on this email list, the
> development team is exploring a more generic Async framework to support
> Async logging.
>
>
>
> Cheers,
>
> George
>
>
>
> From: Roman Konovalov [mailto:roman@greenrain.biz]
> Sent: Tuesday, July 17, 2012 9:11 PM
> To: 'Log4NET Dev'
> Subject: RE: Async logging
>
>
>
> Hi,
>
>
>
> Is Async Logging already part of official release? If not, can you please
> advise when it might be? Many thanks in advance.
>
>
>
> Best regards,
>
> Roman
>
>
>
> From: George Chung [mailto:george@glympse.com]
> Sent: den 15 juni 2012 01:18
> To: Log4NET Dev
> Subject: Re: Async logging
>
>
>
> Based on the log4net-1.2.11 source tree...
>
>
>
> Just stick AsyncSqlAppender.cs into the Appender folder. See attached
> log4net.config file for example of how to use it. Please note, that I set
> the bufferSize = 1. No need to buffer if you're logging asynchronously. And
> I think I make an assumption that the LoggingEvents has a size of 1.
>
>
>
> Minor changes to existing files
>
> I modified AppenderSkeleton to make the append lock virtual.
> AsyncSqlAppender does not need to synchronize multiple writers Sql Server
> does a fine job of that. :)
>
>
>
> I fixed AdoNetAppender to swizzle "(null)" and "NOT AVAILABLE" to a real
> NULL in the db.
>
>
>
> I fixed AspNetRequestPatternConverter to handle situations where there is no
> request object.
>
> On Thu, Jun 14, 2012 at 12:36 PM, George Chung <ge...@glympse.com> wrote:
>
> Actually, I implemented an AsyncSqlAppender.
>
>
>
> On Thu, Jun 14, 2012 at 12:34 PM, George Chung <ge...@glympse.com> wrote:
>
> Yes, and when you submit async operations, those async requests are queued
> by the runtime and the completion routines are executed by threads from this
> thread pool. No need to manage threads at all. I did a pretty trivial hack
> to the AdoNetAppender to perform Async Sql inserts and it's been working
> flawlessly in our production environment.
>
>
>
> I'll attach the modified files shortly.
>
>
>
>
>
> On Fri, Jun 8, 2012 at 4:32 AM, Stefan Bodewig <bo...@apache.org> wrote:
>
> On 2012-06-08, Christian Grobmeier wrote:
>
>> Hi,
>
>> On Fri, Jun 8, 2012 at 8:39 AM, Dominik Psenner <dp...@gmail.com>
>> wrote:
>
>>> * the ThreadPool should be used in favour over a designated worker Thread
>
>
>> no idea becuase I have no clue on .NET, but it sounds like fun :-)
>
> .Net has a built-in thread pool it uses for all kinds of internal stuff,
> including async executions of user events or serving ASP.NET requests -
> and it provides access to it for user code. Rather than creating new
> ad-hoc Threads you re-use one of the pool.
>
> Stefan
>
>
>
>
>
>
--
http://www.grobmeier.de
https://www.timeandbill.de
RE: Async logging
Posted by George Chung <ge...@glympse.com>.
Hi,
Thanks for taking a look at my "contribution". J
Just fyi, I am not a log4net developer.just a user who hacked up an
AsyncSqlAppender. I am using it in a production environment with good
results.feel free to use and comment.
>From what I can gather from the discussions on this email list, the
development team is exploring a more generic Async framework to support
Async logging.
Cheers,
George
From: Roman Konovalov [mailto:roman@greenrain.biz]
Sent: Tuesday, July 17, 2012 9:11 PM
To: 'Log4NET Dev'
Subject: RE: Async logging
Hi,
Is Async Logging already part of official release? If not, can you please
advise when it might be? Many thanks in advance.
Best regards,
Roman
From: George Chung [mailto:george@glympse.com]
Sent: den 15 juni 2012 01:18
To: Log4NET Dev
Subject: Re: Async logging
Based on the log4net-1.2.11 source tree...
Just stick AsyncSqlAppender.cs into the Appender folder. See attached
log4net.config file for example of how to use it. Please note, that I set
the bufferSize = 1. No need to buffer if you're logging asynchronously. And
I think I make an assumption that the LoggingEvents has a size of 1.
Minor changes to existing files
I modified AppenderSkeleton to make the append lock virtual.
AsyncSqlAppender does not need to synchronize multiple writers Sql Server
does a fine job of that. :)
I fixed AdoNetAppender to swizzle "(null)" and "NOT AVAILABLE" to a real
NULL in the db.
I fixed AspNetRequestPatternConverter to handle situations where there is no
request object.
On Thu, Jun 14, 2012 at 12:36 PM, George Chung <ge...@glympse.com> wrote:
Actually, I implemented an AsyncSqlAppender.
On Thu, Jun 14, 2012 at 12:34 PM, George Chung <ge...@glympse.com> wrote:
Yes, and when you submit async operations, those async requests are queued
by the runtime and the completion routines are executed by threads from this
thread pool. No need to manage threads at all. I did a pretty trivial hack
to the AdoNetAppender to perform Async Sql inserts and it's been working
flawlessly in our production environment.
I'll attach the modified files shortly.
On Fri, Jun 8, 2012 at 4:32 AM, Stefan Bodewig <bo...@apache.org> wrote:
On 2012-06-08, Christian Grobmeier wrote:
> Hi,
> On Fri, Jun 8, 2012 at 8:39 AM, Dominik Psenner <dp...@gmail.com>
wrote:
>> * the ThreadPool should be used in favour over a designated worker Thread
> no idea becuase I have no clue on .NET, but it sounds like fun :-)
.Net has a built-in thread pool it uses for all kinds of internal stuff,
including async executions of user events or serving ASP.NET requests -
and it provides access to it for user code. Rather than creating new
ad-hoc Threads you re-use one of the pool.
Stefan
RE: Async logging
Posted by Roman Konovalov <ro...@greenrain.biz>.
Hi,
Is Async Logging already part of official release? If not, can you please
advise when it might be? Many thanks in advance.
Best regards,
Roman
From: George Chung [mailto:george@glympse.com]
Sent: den 15 juni 2012 01:18
To: Log4NET Dev
Subject: Re: Async logging
Based on the log4net-1.2.11 source tree...
Just stick AsyncSqlAppender.cs into the Appender folder. See attached
log4net.config file for example of how to use it. Please note, that I set
the bufferSize = 1. No need to buffer if you're logging asynchronously. And
I think I make an assumption that the LoggingEvents has a size of 1.
Minor changes to existing files
I modified AppenderSkeleton to make the append lock virtual.
AsyncSqlAppender does not need to synchronize multiple writers Sql Server
does a fine job of that. :)
I fixed AdoNetAppender to swizzle "(null)" and "NOT AVAILABLE" to a real
NULL in the db.
I fixed AspNetRequestPatternConverter to handle situations where there is no
request object.
On Thu, Jun 14, 2012 at 12:36 PM, George Chung <ge...@glympse.com> wrote:
Actually, I implemented an AsyncSqlAppender.
On Thu, Jun 14, 2012 at 12:34 PM, George Chung <ge...@glympse.com> wrote:
Yes, and when you submit async operations, those async requests are queued
by the runtime and the completion routines are executed by threads from this
thread pool. No need to manage threads at all. I did a pretty trivial hack
to the AdoNetAppender to perform Async Sql inserts and it's been working
flawlessly in our production environment.
I'll attach the modified files shortly.
On Fri, Jun 8, 2012 at 4:32 AM, Stefan Bodewig <bo...@apache.org> wrote:
On 2012-06-08, Christian Grobmeier wrote:
> Hi,
> On Fri, Jun 8, 2012 at 8:39 AM, Dominik Psenner <dp...@gmail.com>
wrote:
>> * the ThreadPool should be used in favour over a designated worker Thread
> no idea becuase I have no clue on .NET, but it sounds like fun :-)
.Net has a built-in thread pool it uses for all kinds of internal stuff,
including async executions of user events or serving ASP.NET requests -
and it provides access to it for user code. Rather than creating new
ad-hoc Threads you re-use one of the pool.
Stefan
Re: Async logging
Posted by George Chung <ge...@glympse.com>.
Based on the log4net-1.2.11 source tree...
Just stick *AsyncSqlAppender*.cs into the Appender folder. See attached *
log4net.config* file for example of how to use it. Please note, that I set
the bufferSize = 1. No need to buffer if you're logging asynchronously. And
I think I make an assumption that the LoggingEvents has a size of 1.
Minor changes to existing files
I modified AppenderSkeleton to make the append lock virtual.
AsyncSqlAppender does not need to synchronize multiple writers Sql Server
does a fine job of that. :)
I fixed AdoNetAppender to swizzle "(null)" and "NOT AVAILABLE" to a real
NULL in the db.
I fixed AspNetRequestPatternConverter to handle situations where there is
no request object.
On Thu, Jun 14, 2012 at 12:36 PM, George Chung <ge...@glympse.com> wrote:
> Actually, I implemented an AsyncSqlAppender.
>
>
> On Thu, Jun 14, 2012 at 12:34 PM, George Chung <ge...@glympse.com> wrote:
>
>> Yes, and when you submit async operations, those async requests are
>> queued by the runtime and the completion routines are executed by threads
>> from this thread pool. No need to manage threads at all. I did a pretty
>> trivial hack to the AdoNetAppender to perform Async Sql inserts and it's
>> been working flawlessly in our production environment.
>>
>> I'll attach the modified files shortly.
>>
>>
>> On Fri, Jun 8, 2012 at 4:32 AM, Stefan Bodewig <bo...@apache.org>wrote:
>>
>>> On 2012-06-08, Christian Grobmeier wrote:
>>>
>>> > Hi,
>>>
>>> > On Fri, Jun 8, 2012 at 8:39 AM, Dominik Psenner <dp...@gmail.com>
>>> wrote:
>>>
>>> >> * the ThreadPool should be used in favour over a designated worker
>>> Thread
>>>
>>> > no idea becuase I have no clue on .NET, but it sounds like fun :-)
>>>
>>> .Net has a built-in thread pool it uses for all kinds of internal stuff,
>>> including async executions of user events or serving ASP.NET requests -
>>> and it provides access to it for user code. Rather than creating new
>>> ad-hoc Threads you re-use one of the pool.
>>>
>>> Stefan
>>>
>>
>>
>
Re: Async logging
Posted by George Chung <ge...@glympse.com>.
Actually, I implemented an AsyncSqlAppender.
On Thu, Jun 14, 2012 at 12:34 PM, George Chung <ge...@glympse.com> wrote:
> Yes, and when you submit async operations, those async requests are queued
> by the runtime and the completion routines are executed by threads from
> this thread pool. No need to manage threads at all. I did a pretty trivial
> hack to the AdoNetAppender to perform Async Sql inserts and it's been
> working flawlessly in our production environment.
>
> I'll attach the modified files shortly.
>
>
> On Fri, Jun 8, 2012 at 4:32 AM, Stefan Bodewig <bo...@apache.org> wrote:
>
>> On 2012-06-08, Christian Grobmeier wrote:
>>
>> > Hi,
>>
>> > On Fri, Jun 8, 2012 at 8:39 AM, Dominik Psenner <dp...@gmail.com>
>> wrote:
>>
>> >> * the ThreadPool should be used in favour over a designated worker
>> Thread
>>
>> > no idea becuase I have no clue on .NET, but it sounds like fun :-)
>>
>> .Net has a built-in thread pool it uses for all kinds of internal stuff,
>> including async executions of user events or serving ASP.NET requests -
>> and it provides access to it for user code. Rather than creating new
>> ad-hoc Threads you re-use one of the pool.
>>
>> Stefan
>>
>
>
Re: Async logging
Posted by George Chung <ge...@glympse.com>.
Yes, and when you submit async operations, those async requests are queued
by the runtime and the completion routines are executed by threads from
this thread pool. No need to manage threads at all. I did a pretty trivial
hack to the AdoNetAppender to perform Async Sql inserts and it's been
working flawlessly in our production environment.
I'll attach the modified files shortly.
On Fri, Jun 8, 2012 at 4:32 AM, Stefan Bodewig <bo...@apache.org> wrote:
> On 2012-06-08, Christian Grobmeier wrote:
>
> > Hi,
>
> > On Fri, Jun 8, 2012 at 8:39 AM, Dominik Psenner <dp...@gmail.com>
> wrote:
>
> >> * the ThreadPool should be used in favour over a designated worker
> Thread
>
> > no idea becuase I have no clue on .NET, but it sounds like fun :-)
>
> .Net has a built-in thread pool it uses for all kinds of internal stuff,
> including async executions of user events or serving ASP.NET requests -
> and it provides access to it for user code. Rather than creating new
> ad-hoc Threads you re-use one of the pool.
>
> Stefan
>
Re: Async logging
Posted by Stefan Bodewig <bo...@apache.org>.
On 2012-06-08, Christian Grobmeier wrote:
> Hi,
> On Fri, Jun 8, 2012 at 8:39 AM, Dominik Psenner <dp...@gmail.com> wrote:
>> * the ThreadPool should be used in favour over a designated worker Thread
> no idea becuase I have no clue on .NET, but it sounds like fun :-)
.Net has a built-in thread pool it uses for all kinds of internal stuff,
including async executions of user events or serving ASP.NET requests -
and it provides access to it for user code. Rather than creating new
ad-hoc Threads you re-use one of the pool.
Stefan
RE: Async logging
Posted by Dominik Psenner <dp...@gmail.com>.
Hi Christian,
thanks for the feedback.
>> · the ThreadPool should be used in favour over a designated
>worker
>> Thread
>
>no idea becuase I have no clue on .NET, but it sounds like fun :-)
It just means that it is not written as:
new Thread(() => { ..; Thread.Sleep(x); }
but rather there is only a worker thread active when there are pending log
events. The algorithm is quite simple:
Append(logEvent) {
Queue.append(logEvent)
If(queue.size == 1) {
Threadpool.Invoke(ForwardMessages)
}
}
ForwardMessages() {
While(Queue.count > 0){
# forward
}
}
Of course it needs some locking and other dark magic in between to work as
expected.
>> · if the queue is full, it should fallback to synchronous
>logging,
>> shouldnt it?
>
>Not sure on this: this would probably impact the application in terms
>of performance right?
Yes. But it does in a "safer" manner. The philosophy is much like safer sex.
Either strategy means:
case Strategy.IGNORE:
RAM caching => OutOfMemoryException
case Strategy.DISMISS:
loss of events, I'm not a great fan of this
case Strategy.SYNC:
the calling application is slowed down approximately to how fast the log
events can be serialized to the underlying I/O, which means the async
implementation behaves more like a buffer
>Probably this should be configurable to:
>
>QUEUE_OVERFLOW_STRATEGY=DISMISS -> throws away the log events
>QUEUE_OVERFLOW_STRATEGY=SYNC -> goes back to synchronized logging
Good point. And for brave developers:
QUEUE_OVERFLOW_STRATEGY=IGNORE -> ignores queue overflow
>In either case, a warning message should be generated.
Of course. But we should make sure to not stress the CPU with too many
additional warnings since it probably would be already at its limits.
>Another interesting feature would to set a threshhold for the queue.
>Lets assume it is 100, and 100 messages are in the queue while 150 are
>allowed, one could get a warning log that the threshhold of 100 is
>passed.
Is it really required to make that configurable? If so, I would make it an
optional configuration with the fallback to something like
THRESHOLD=MAX_QUEUE_SIZE*0.9 that outputs warnings whenever the queue size
hits a multiple of TRESHOLD (i.e. queue.size % TRESHOLD == 0).
>
>> · queued log events should be processed when the appender gets
>> disposed
>
>+1
The problem here is that the CLR may take over (i.e. on crashes) and simply
terminate the appender together with the assembly that loaded it. Does
somebody know a way to preserve the last few log messages from destruction
in those corner cases?
Re: Async logging
Posted by Christian Grobmeier <gr...@gmail.com>.
Hi,
On Fri, Jun 8, 2012 at 8:39 AM, Dominik Psenner <dp...@gmail.com> wrote:
> · the queue size should be configurable
+1
>
> · the ThreadPool should be used in favour over a designated worker
> Thread
no idea becuase I have no clue on .NET, but it sounds like fun :-)
> · if the queue is full, it should fallback to synchronous logging,
> shouldn’t it?
Not sure on this: this would probably impact the application in terms
of performance right?
Probably this should be configurable to:
QUEUE_OVERFLOW_STRATEGY=DISMISS -> throws away the log events
QUEUE_OVERFLOW_STRATEGY=SYNC -> goes back to synchronized logging
In either case, a warning message should be generated.
Another interesting feature would to set a threshhold for the queue.
Lets assume it is 100, and 100 messages are in the queue while 150 are
allowed, one could get a warning log that the threshhold of 100 is
passed.
> · queued log events should be processed when the appender gets
> disposed
+1
Cheers
Christian
>
>
>
> Cheers,
>
> Dominik
--
http://www.grobmeier.de
https://www.timeandbill.de
Re: Async logging
Posted by Stefan Bodewig <bo...@apache.org>.
On 2012-06-08, Dominik Psenner wrote:
> In response to
> <https://issues.apache.org/jira/browse/LOG4NET-344?focusedCommentId=13291557>
> I would like to start a discussion how log4net should support
> asynchronous logging. Tom Tang implemented a general async forwarding
> appender that glues together other appenders and thus adds general
> async processing of log events. He implemented a good idea that misses
> only a few small things to be an awesome improvement.
Like I wrote over there, I need to give the code more than just a quick
glance, but it look really useful to me to.
> My objections are:
> * the queue size should be configurable
> * the ThreadPool should be used in favour over a designated worker Thread
+1 to both.
> * if the queue is full, it should fallback to synchronous logging,
> shouldn't it?
I agree with the idea proposed later during the thread of using a
strategy here. The IGNORE strategy would be the same as having some
"unlimited" value for the queue size, right?
> * queued log events should be processed when the appender gets
> disposed
I'd encourage you to verify how the RemotingAppender currently works, it
must be facing the same issue WRT being shut down. My guess is "do the
best we can do but don't guarantee anything".
Stefan