You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-dev@logging.apache.org by Curt Arnold <ca...@apache.org> on 2010/05/22 04:44:56 UTC
A new thread for log4j 2.0 discussion (Was Re: svn commit: r943816 [1/9] - in /logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers: ./ log4j12-api/ log4j12-api/src/ log4j12-api/src/main/ log4j12-api/src/main/java/ log4j12-api/src/main/java/org/ log4j12-api/src/main/java/org/apache/ log4j12-api/sr...)
Thought it might be best to leave the old thread behind, hope nobody minds.
On May 14, 2010, at 2:22 AM, Ralph Goers wrote:
> Thanks for trying it out. I hadn't actually built from the root - I've built the api and core separately - so I'm glad you were able to fix those problems.
>
> I didn't want to commit code until I had the core of something that actually functioned. I struggled for a couple of weeks over how to attack XMLConfiguration. I don't like the way Logback does it, I didn't want to bring in baggage from a third party framework, JAXB isn't a good fit either. In short, I wanted something that can be extended without needing to add "rules" to the XML processor. See below for what I came up with.
>
I understand the desire to have something fleshed out enough so your thoughts have concrete expression. Easier to point to code than to try to express in verbiage your thoughts on how everything is going to come together, but there are downsides and can be abused.
> First, while I looked at Log4j and somewhat at Logback, most of the core code is completely new. The exception to this is with the Converters for the PatternLayout. I took the EnhancedPatternLayout and modified it.
>
I had totally forgotten that I called that EnhancedPatternLayout, so I wasn't sure if you were talking about the extras version or the sandbox version.
> 1. I first created an API that had the features I was looking for. That is in log4j2-api. While it supports logging a String or an Object it really uses a Message interface which is valuable as it allows users to log self-describing objects in a convenient manner.
My thinking was the message interface would end up so minimal that might as well just use Object.
> 2. I don't like the way Logback binds to the implementation. I used a technique I had used in a previous logging framework and used a file to define the implementation class. In theory, the API could be modified to support multiple logging implementation simultaneously, although I have no plans to implement that.
Not something that I've thought about.
> 3. Logback suffers from a serious architectural problem that is rooted in Log4j. The configured loggers are mixed with the loggers returned from the Logger factory. This makes it impossible to reconfigure atomically. With Logback the reset method is called on the context which essentially causes the system to be in an undefined state until the new configuration is completed (log records that should be logged are lost). To solve this I used a design that I again borrowed from my previous framework. The configuration is separated and on a reconfiguration the new configuration will be created and then all the loggers will be updated to use it. While there will be a period where some loggers are using the old configuration and some the new there is never a point where loggers aren't configured at all.
I think that is along the line that I was thinking, basically there is an immutable configuration state object that is swapped out as an atomic operation with notification to allocated loggers.
> 4. Similar to Logback I added support for Markers and global filters. In addition, filters on Loggers are also supported. Unlike Logback, there is only a single Filter interface used for global filters, logger filters and appender filters.
I've never had a clear description of the use cases behind Markers. As far as I can gather, it is a specialized case of a user supplied context. At the core, I think it would fit into a general user-supplied context object. MDC and NDC would be part of the thread-supplied context and there'd be a global context and a call-site context. In the core, I'd expect the context parameters to just be Object and let the layout level cast to specific known interfaces as needed. During the synchronous extraction phase, an immutable package would be assembled for from log request parameters and the contexts based on the needs of the layout's formatting phase.
> 5. The XMLConfiguration is extremely simple. All it does is read the XML and convert the DOM structure to internal Node elements, which contain the node attributes, child node references and placeholders for the real object when it is constructed. It uses the XML element name to match to a Plugin, so instead of writing:
>
> <appender name="console" class="org.apache.log4j.ConsoleAppender">
> <param name="Target" value="System.out"/>
> <layout class="org.apache.log4j.PatternLayout">
> <param name="ConversionPattern" value="%-5p %c{1} - %m%n"/>
> </layout>
> </appender>
>
> you write:
>
> <appenders>
> <Console name="console" target="SYSTEM_OUT">
> <PatternLayout>%-5p %c{1} - %m%n</PatternLayout>
> </Console>
> </appenders>
>
> Note that it also would support
> <Console>
> <name>console</name>
> <target>SYSTEM_OUT</target>
> <PatternLayout>%-5p %c{1} - %m%n</PatternLayout>
> </Console>
>
> if you prefer using elements over attributes.
It is something necessary, but hopefully configuration ends up something outside the core using public methods to do its magic.
>
> 5. I implemented a "Plugin" concept. All core components use annotations to describe their name and type. This is used by the XML configuration to determine the element names that are used in the configuration file.
> a) Plugins are used for Logger, Appenders, Filters, etc. The BaseConfiguration processes the nodes created by XMLConfiguration and saves the resulting Objects as appropriate.
> b) PatternLayout uses plugins for the converters. To add a new converter an end user just needs to create the class with the correct annotation and add the package name to the XML configuration as an attribute.
> c) Configuration uses plugins to determine the Configuration implementation to use. The XMLConfiguration and DefaultConfiguration will always be present, but a weight can be specified to allow the new Configuration to take precedence. Again, simply placing the Configuration class (with the appropriate annotations) in the classpath will cause it to be used. However, if it is in a different package that package will have to be registered to be searched either by calling a static method on the PluginManager or via a system property (which hasn't been implemented yet).
Any thoughts on OSGi or other component frameworks? Not now, maybe, never?
> 6. java.util.concurrent is used where appropriate. Accordingly, minimal locking takes place. No lock is held while Appenders are called which is a serious bug in Log4j 1.x as it causes deadlocks.
> 7, All internal components use the logger API for status reporting. A specific StatusLogger provides the implementation.
> 8. Layouts return a byte array. In Logback Ceki recently came up with the idea of using encoders to allow binary data streams to be sent and received. While the requirement seems valid it seemed awkward to wrap a Layout in an encoder.
I think there will likely need to be separate byte and character Layout-like interfaces. There are sometimes you are writing to a character-oriented destination like a database API and other times to a byte-oriented destination like a file system or network socket. You'd like something like PatternLayout to be able to work in both instances (though it would need to be paired with a character encoder for a byte-oriented destination).
The XMLLayout in log4j 1.2 suffers from this since it really should be byte-oriented. At the present, invalid XML can be generated if you do not use UTF-8 or UTF-16.
The Java serialization in the socket appender is effectively a byte-oriented layout and there is no reason you should not be able to connect it to a file appender.
> 9. Obviously, everything leverages Java 5 (and might possibly require Java 6 since that is the JVM I've been using).
>
> The API is not compatible with log4j 1.x. My intention would be to create a compatible API (to the degree possible) in a log4j12-api package.
>
I believe that we should try to do design the core so it is the best core and then later see what, if any, needs to be tweaked.
> I've benchmarked this against Logback in both the SimplePerfTest and ThreadedPerfTest. Logback is slightly faster, probably due to the separation of the configuration from the logger, but the difference is about 1 or 2 hundredths of a second over a million log records that don't pass the log level (i.e. no log records are written).
>
> Although it is a decent amount of code this is meant to be just a starting point. My hope is that others, like yourself, will look at this and figure out how to improve on it. And, of course, there are a bunch of Appenders, Filters, and other components that are completely missing.
>
> Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: A new thread for log4j 2.0 discussion (Was Re: svn commit: r943816 [1/9] - in /logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers: ./ log4j12-api/ log4j12-api/src/ log4j12-api/src/main/ log4j12-api/src/main/java/ log4j12-api/src/main/java/org/ log4j12-api/src/main/java/org/apache/ log4j12-api/sr...)
Posted by Ralph Goers <ra...@dslextreme.com>.
On May 23, 2010, at 9:30 PM, Curt Arnold wrote:
>
>>
>>>
>>> The AsynchronousAppender in log4j requires trying to freeze LoggingEvent which is still subject to message objects changing underneath it. Also it generally perform unnecessary actions in freezing everything in the LoggingEvent even though much of it will not be used.
>>>
>>> The framework should be designed to separate the parts required to be synchronous (that is anything dealing with external objects) from the parts that can be deferred.
>>
>> I don't think I really agree with this, at least the way you are expressing it. Typically, the "parts required to be synchronous" are also the ones that are expensive to resolve. What you are suggesting is that somehow it is easier to deal with a LogEvent that has a reference to another object that contains the references to the "synchronous objects" than to just copy the whole LogEvent. But copying simple object references like Strings doesn't cost all that much and may not even be worth doing if the appender needs some sort of serialized version of the LogEvent instead (SyslogAppender using either the serialized logging event or the IETFSyslogLayout that I haven't added yet).
>
> The summary in the second sentence feels the antithesis of what I'm suggesting.
>
> Copying the "whole log event" so that the copy is not affected by any potential changes is complicated, expensive and error-prone as seen in log4j's AsyncAppender. The message parameter or any value in the MDC or NDC in the log4j API can be any arbitrary object. If it is immutable, then you are fine. If not and the object is serializable or clonable, you could clonable. Otherwise, you are subject to changes that could occur after the logging call.
>
> Restricting message, MDC and NDC parameters to strings is not desirable. log4j allows it and there are reasonable and worthwhile reasons to use them.
You are correct. I had forgotten MDC and NDC allow Objects. I've gotten used to SLF4J only allowing Strings on the MDC. But the MDC I checked in does support Objects. I will have to look at what is happening with that in the LogEvent again.
>
> AsyncAppender does a not totally satisfactory clone of a LoggingEvent and then runs the nested appender later on that clone, none of the nested appenders code runs during the synchronous phase. The alternative I'm suggesting is to allow appenders to see the LoggingEvent in the extraction phase, get all the information that it needs (or possibly do the whole action) and then leave a Runnable to complete the action. If fully synchronous, the runnable would be immediately executed, if not, then sent to a queue for later execution.
Using a thread for to perform a synchronous activity is not something I think I'd be in favor of. Creating a thread is not trivial. Creating thread pools causes problems that have to be dealt with in servlet containers. Ceki and I had a discussion about that with a bug in Logback and I tended to agree with his position that creating long lived threads is bad.
>
>
>
>>
>> In looking at Log4jLogEvent really the only fields that deal with external objects are the Message and the Throwable. I suppose a Throwable could be a problem if the caller has created their own exception type and added odd stuff to it (complex object references), although I can't say I've ever run across an Exception that does that. The Message can be solved by a) doing new SimpleMessage(message.getFormattedMessage()), b) serializing the message and deserializing it, or c) adding another method to the Message interface like "prepareForAsynch".
>
> "prepareForAsynch" has the same problem as serializing the logging event. There is no bulletproof way to make an invariant copy of every arbitrary objects and some may be extraordinary expensive. It seems to me to be much clearer to have a phase where you have access to the LoggingEvent interface in the scope of the logging call, but anything deferred must act on what was extracted from that call. At the simplest, it could be to run the layout in the extraction phase and hold the string for the completion phase.
You are going to have to show me some code as I really don't see the difference between "prepareForAsynch" and a "phase".
>
> If I have an "itch" for log4j 2.0, this is it. I'm not sure where the path would lead, but it is the path that I want to explore.
I'm certainly not going to get in the way. The whole point of putting something out there is to have a place to start from.
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: A new thread for log4j 2.0 discussion (Was Re: svn commit: r943816 [1/9] - in /logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers: ./ log4j12-api/ log4j12-api/src/ log4j12-api/src/main/ log4j12-api/src/main/java/ log4j12-api/src/main/java/org/ log4j12-api/src/main/java/org/apache/ log4j12-api/sr...)
Posted by Curt Arnold <ca...@apache.org>.
On May 23, 2010, at 2:27 AM, Ralph Goers wrote:
>
> On May 22, 2010, at 8:13 PM, Curt Arnold wrote:
>
>>
>> On May 22, 2010, at 10:50 AM, Ralph Goers wrote:
>>>
>>>>
>>>> Also, I have had an issue where the I wanted the toString() method on the object to be logged to be delayed as much as possible (somewhat helped with the {}-construct in slf4j). This might be doable with a specific log interface.
>>>
>>> That is exactly what Message does.
>>>
>>
>>
>> My thoughts, but not sure how much of this got into my earlier code, was to have a distinct extraction phase that occur synchronously with the logging call that would extract all info needed for a completion phase that could occur either immediately or be deferred. The extraction phase would need to extract any info from the call into immutable objects so state change, for example mutable message parameters, before the completion stage would not change the resulting log.
>
> There is no point in doing this in simple use cases such as using the FileAppender to write to a log file.
Usually not the simple cases that dictate the design.
>
>>
>> A example where this would be useful is logging floating point numbers. It can be surprisingly expensive to format floating point numbers, so it is highly beneficial to extract all the pieces needed for the formatting into an immutable package during the synchronous extraction phase and then do the formatting in the asynchronous completion phase.
>
> This assumes there is an asynchronous phase. In my audit logging use cases I do not want asynchronous logging because I want any exceptions that occur to be thrown back to the caller. An awful lot of logs just write Strings and simple values. While we should support use cases like what you are suggesting I think it would be unwise to optimize for them at the expense of typical usage.
If the framework allows for clear separation between required to be synchronous and allowed to be asynchronous, then doing everything synchronous is easy as immediately invoking run() on the returned Runnable. Might come with a performance cost when running purely synchronous, but without some experimentation can't quantity if it is significant.
>
>>
>> The AsynchronousAppender in log4j requires trying to freeze LoggingEvent which is still subject to message objects changing underneath it. Also it generally perform unnecessary actions in freezing everything in the LoggingEvent even though much of it will not be used.
>>
>> The framework should be designed to separate the parts required to be synchronous (that is anything dealing with external objects) from the parts that can be deferred.
>
> I don't think I really agree with this, at least the way you are expressing it. Typically, the "parts required to be synchronous" are also the ones that are expensive to resolve. What you are suggesting is that somehow it is easier to deal with a LogEvent that has a reference to another object that contains the references to the "synchronous objects" than to just copy the whole LogEvent. But copying simple object references like Strings doesn't cost all that much and may not even be worth doing if the appender needs some sort of serialized version of the LogEvent instead (SyslogAppender using either the serialized logging event or the IETFSyslogLayout that I haven't added yet).
The summary in the second sentence feels the antithesis of what I'm suggesting.
Copying the "whole log event" so that the copy is not affected by any potential changes is complicated, expensive and error-prone as seen in log4j's AsyncAppender. The message parameter or any value in the MDC or NDC in the log4j API can be any arbitrary object. If it is immutable, then you are fine. If not and the object is serializable or clonable, you could clonable. Otherwise, you are subject to changes that could occur after the logging call.
Restricting message, MDC and NDC parameters to strings is not desirable. log4j allows it and there are reasonable and worthwhile reasons to use them.
AsyncAppender does a not totally satisfactory clone of a LoggingEvent and then runs the nested appender later on that clone, none of the nested appenders code runs during the synchronous phase. The alternative I'm suggesting is to allow appenders to see the LoggingEvent in the extraction phase, get all the information that it needs (or possibly do the whole action) and then leave a Runnable to complete the action. If fully synchronous, the runnable would be immediately executed, if not, then sent to a queue for later execution.
>
> In looking at Log4jLogEvent really the only fields that deal with external objects are the Message and the Throwable. I suppose a Throwable could be a problem if the caller has created their own exception type and added odd stuff to it (complex object references), although I can't say I've ever run across an Exception that does that. The Message can be solved by a) doing new SimpleMessage(message.getFormattedMessage()), b) serializing the message and deserializing it, or c) adding another method to the Message interface like "prepareForAsynch".
"prepareForAsynch" has the same problem as serializing the logging event. There is no bulletproof way to make an invariant copy of every arbitrary objects and some may be extraordinary expensive. It seems to me to be much clearer to have a phase where you have access to the LoggingEvent interface in the scope of the logging call, but anything deferred must act on what was extracted from that call. At the simplest, it could be to run the layout in the extraction phase and hold the string for the completion phase.
If I have an "itch" for log4j 2.0, this is it. I'm not sure where the path would lead, but it is the path that I want to explore.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: A new thread for log4j 2.0 discussion (Was Re: svn commit: r943816 [1/9] - in /logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers: ./ log4j12-api/ log4j12-api/src/ log4j12-api/src/main/ log4j12-api/src/main/java/ log4j12-api/src/main/java/org/ log4j12-api/src/main/java/org/apache/ log4j12-api/sr...)
Posted by Ralph Goers <ra...@dslextreme.com>.
On May 22, 2010, at 8:13 PM, Curt Arnold wrote:
>
> On May 22, 2010, at 10:50 AM, Ralph Goers wrote:
>>
>>>
>>> Also, I have had an issue where the I wanted the toString() method on the object to be logged to be delayed as much as possible (somewhat helped with the {}-construct in slf4j). This might be doable with a specific log interface.
>>
>> That is exactly what Message does.
>>
>
>
> My thoughts, but not sure how much of this got into my earlier code, was to have a distinct extraction phase that occur synchronously with the logging call that would extract all info needed for a completion phase that could occur either immediately or be deferred. The extraction phase would need to extract any info from the call into immutable objects so state change, for example mutable message parameters, before the completion stage would not change the resulting log.
There is no point in doing this in simple use cases such as using the FileAppender to write to a log file.
>
> A example where this would be useful is logging floating point numbers. It can be surprisingly expensive to format floating point numbers, so it is highly beneficial to extract all the pieces needed for the formatting into an immutable package during the synchronous extraction phase and then do the formatting in the asynchronous completion phase.
This assumes there is an asynchronous phase. In my audit logging use cases I do not want asynchronous logging because I want any exceptions that occur to be thrown back to the caller. An awful lot of logs just write Strings and simple values. While we should support use cases like what you are suggesting I think it would be unwise to optimize for them at the expense of typical usage.
>
> The AsynchronousAppender in log4j requires trying to freeze LoggingEvent which is still subject to message objects changing underneath it. Also it generally perform unnecessary actions in freezing everything in the LoggingEvent even though much of it will not be used.
>
> The framework should be designed to separate the parts required to be synchronous (that is anything dealing with external objects) from the parts that can be deferred.
I don't think I really agree with this, at least the way you are expressing it. Typically, the "parts required to be synchronous" are also the ones that are expensive to resolve. What you are suggesting is that somehow it is easier to deal with a LogEvent that has a reference to another object that contains the references to the "synchronous objects" than to just copy the whole LogEvent. But copying simple object references like Strings doesn't cost all that much and may not even be worth doing if the appender needs some sort of serialized version of the LogEvent instead (SyslogAppender using either the serialized logging event or the IETFSyslogLayout that I haven't added yet).
In looking at Log4jLogEvent really the only fields that deal with external objects are the Message and the Throwable. I suppose a Throwable could be a problem if the caller has created their own exception type and added odd stuff to it (complex object references), although I can't say I've ever run across an Exception that does that. The Message can be solved by a) doing new SimpleMessage(message.getFormattedMessage()), b) serializing the message and deserializing it, or c) adding another method to the Message interface like "prepareForAsynch".
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: A new thread for log4j 2.0 discussion (Was Re: svn commit: r943816 [1/9] - in /logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers: ./ log4j12-api/ log4j12-api/src/ log4j12-api/src/main/ log4j12-api/src/main/java/ log4j12-api/src/main/java/org/ log4j12-api/src/main/java/org/apache/ log4j12-api/sr...)
Posted by Curt Arnold <ca...@apache.org>.
On May 22, 2010, at 10:50 AM, Ralph Goers wrote:
>
>>
>> Also, I have had an issue where the I wanted the toString() method on the object to be logged to be delayed as much as possible (somewhat helped with the {}-construct in slf4j). This might be doable with a specific log interface.
>
> That is exactly what Message does.
>
My thoughts, but not sure how much of this got into my earlier code, was to have a distinct extraction phase that occur synchronously with the logging call that would extract all info needed for a completion phase that could occur either immediately or be deferred. The extraction phase would need to extract any info from the call into immutable objects so state change, for example mutable message parameters, before the completion stage would not change the resulting log.
A example where this would be useful is logging floating point numbers. It can be surprisingly expensive to format floating point numbers, so it is highly beneficial to extract all the pieces needed for the formatting into an immutable package during the synchronous extraction phase and then do the formatting in the asynchronous completion phase.
The AsynchronousAppender in log4j requires trying to freeze LoggingEvent which is still subject to message objects changing underneath it. Also it generally perform unnecessary actions in freezing everything in the LoggingEvent even though much of it will not be used.
The framework should be designed to separate the parts required to be synchronous (that is anything dealing with external objects) from the parts that can be deferred.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: A new thread for log4j 2.0 discussion (Was Re: svn commit: r943816 [1/9] - in /logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers: ./ log4j12-api/ log4j12-api/src/ log4j12-api/src/main/ log4j12-api/src/main/java/ log4j12-api/src/main/java/org/ log4j12-api/src/main/java/org/apache/ log4j12-api/sr...)
Posted by Ralph Goers <ra...@dslextreme.com>.
On May 22, 2010, at 12:46 AM, Thorbjørn Ravn Andersen wrote:
> Den 22/05/10 04.44, Curt Arnold skrev:
>>
>>> 1. I first created an API that had the features I was looking for. That is in log4j2-api. While it supports logging a String or an Object it really uses a Message interface which is valuable as it allows users to log self-describing objects in a convenient manner.
>>>
>> My thinking was the message interface would end up so minimal that might as well just use Object.
>>
> Would it be feasible to use Object so any object can be logged, but support one or more interfaces that allow a given object to provide more information to the logger framework?
Take a look at the API. Specifically, Logger and AbstractLogger. debug(Object) is supported. Under the covers it turns into log(new ObjectMessage(object)). The advantage is if the Object implements the Message interface then the call is debug(Message) and the Object is thus self describing. Alternatively, the user could associate their object's with Message objects (i.e. debug(object.getMessage()). For example, if the Object contains data that shouldn't be logged (like a Social Security Number, PIN, etc.) this allows the Object the flexibility to control that.
>
> Also, I have had an issue where the I wanted the toString() method on the object to be logged to be delayed as much as possible (somewhat helped with the {}-construct in slf4j). This might be doable with a specific log interface.
That is exactly what Message does.
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: A new thread for log4j 2.0 discussion (Was Re: svn commit: r943816
[1/9] - in /logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers: ./ log4j12-api/
log4j12-api/src/ log4j12-api/src/main/ log4j12-api/src/main/java/ log4j12-api/src/main/java/org/
log4j12-api/src/main/java/org/apache/ log4j12-api/sr...)
Posted by Thorbjørn Ravn Andersen <no...@gmail.com>.
Den 22/05/10 04.44, Curt Arnold skrev:
>
>> 1. I first created an API that had the features I was looking for. That is in log4j2-api. While it supports logging a String or an Object it really uses a Message interface which is valuable as it allows users to log self-describing objects in a convenient manner.
>>
> My thinking was the message interface would end up so minimal that might as well just use Object.
>
Would it be feasible to use Object so any object can be logged, but
support one or more interfaces that allow a given object to provide more
information to the logger framework?
Also, I have had an issue where the I wanted the toString() method on
the object to be logged to be delayed as much as possible (somewhat
helped with the {}-construct in slf4j). This might be doable with a
specific log interface.
--
Thorbjørn Ravn Andersen "...plus... Tubular Bells!"
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: A new thread for log4j 2.0 discussion (Was Re: svn commit: r943816 [1/9] - in /logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers: ./ log4j12-api/ log4j12-api/src/ log4j12-api/src/main/ log4j12-api/src/main/java/ log4j12-api/src/main/java/org/ log4j12-api/src/main/java/org/apache/ log4j12-api/sr...)
Posted by Curt Arnold <ca...@apache.org>.
On May 21, 2010, at 11:15 PM, Ralph Goers wrote:
>>
>>> 1. I first created an API that had the features I was looking for. That is in log4j2-api. While it supports logging a String or an Object it really uses a Message interface which is valuable as it allows users to log self-describing objects in a convenient manner.
>>
>> My thinking was the message interface would end up so minimal that might as well just use Object.
>
> I don't think so. For example, take a look at StructuredDataMessage and LocalizedMessage. Those allow you to do some interesting things. The idea is that it is easy for users to extend without having to muck with the internals. Ceki handled this by creating logback-classis, logback-access and logback-audit. That seemed to me to be a very heavyweight approach to accomplish this.
Definitely will look at, just wanted to let you know my unchallenged inclination. I'm willing to be convinced.
A little off track, there was a long ago thread that wandered between log4j and commons logging about localization of logging that came to mind. Took me a while to find it, but the threads of interest are:
http://article.gmane.org/gmane.comp.jakarta.commons.devel/59103/match=arnold+enterprise
http://www.mail-archive.com/log4j-user@logging.apache.org/msg00479.html
>
>>
>>
>>> 2. I don't like the way Logback binds to the implementation. I used a technique I had used in a previous logging framework and used a file to define the implementation class. In theory, the API could be modified to support multiple logging implementation simultaneously, although I have no plans to implement that.
>>
>> Not something that I've thought about.
>
> Probably not. It just occurred to me that it is possible as I was implementing it but decided it wasn't an itch I needed to scratch at the moment.
>
>>
>>
>>> 3. Logback suffers from a serious architectural problem that is rooted in Log4j. The configured loggers are mixed with the loggers returned from the Logger factory. This makes it impossible to reconfigure atomically. With Logback the reset method is called on the context which essentially causes the system to be in an undefined state until the new configuration is completed (log records that should be logged are lost). To solve this I used a design that I again borrowed from my previous framework. The configuration is separated and on a reconfiguration the new configuration will be created and then all the loggers will be updated to use it. While there will be a period where some loggers are using the old configuration and some the new there is never a point where loggers aren't configured at all.
>>
>> I think that is along the line that I was thinking, basically there is an immutable configuration state object that is swapped out as an atomic operation with notification to allocated loggers.
>
> Exactly.
>
>>
>>
>>> 4. Similar to Logback I added support for Markers and global filters. In addition, filters on Loggers are also supported. Unlike Logback, there is only a single Filter interface used for global filters, logger filters and appender filters.
>>
>> I've never had a clear description of the use cases behind Markers. As far as I can gather, it is a specialized case of a user supplied context. At the core, I think it would fit into a general user-supplied context object. MDC and NDC would be part of the thread-supplied context and there'd be a global context and a call-site context. In the core, I'd expect the context parameters to just be Object and let the layout level cast to specific known interfaces as needed. During the synchronous extraction phase, an immutable package would be assembled for from log request parameters and the contexts based on the needs of the layout's formatting phase.
>
> I think I'd need to see code to get what you are driving at. In the code I checked in it does have the MDC & NDC and does have a global context, although I haven't added variables to it yet. It also has the configuration which can also have variables, but probably for a different purpose. I don't know what you mean by "call-site context".
I was meaning the context that is only valid for the duration of the call, essentially the info that you'd get from new Throwable().getStackTrace().
>
> The value of Markers is that they are fast and easy to filter on. I use them for Entry and Exit so that you can easily filter them out (admittedly this can also be done via the TRACE level) as well as catching and throwing. In addition, I also use a Marker to identify audit events. Anything with that Marker will always be logged.
>
Still seems like a specialized user context object to me. Don't know if there is a user-supplied context object outside of it. Will need to take a look.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org
Re: A new thread for log4j 2.0 discussion (Was Re: svn commit: r943816 [1/9] - in /logging/log4j/branches/BRANCH_2_0_EXPERIMENTAL/rgoers: ./ log4j12-api/ log4j12-api/src/ log4j12-api/src/main/ log4j12-api/src/main/java/ log4j12-api/src/main/java/org/ log4j12-api/src/main/java/org/apache/ log4j12-api/sr...)
Posted by Ralph Goers <ra...@dslextreme.com>.
On May 21, 2010, at 7:44 PM, Curt Arnold wrote:
> Thought it might be best to leave the old thread behind, hope nobody minds.
>
>
> On May 14, 2010, at 2:22 AM, Ralph Goers wrote:
>
>> Thanks for trying it out. I hadn't actually built from the root - I've built the api and core separately - so I'm glad you were able to fix those problems.
>>
>> I didn't want to commit code until I had the core of something that actually functioned. I struggled for a couple of weeks over how to attack XMLConfiguration. I don't like the way Logback does it, I didn't want to bring in baggage from a third party framework, JAXB isn't a good fit either. In short, I wanted something that can be extended without needing to add "rules" to the XML processor. See below for what I came up with.
>>
>
> I understand the desire to have something fleshed out enough so your thoughts have concrete expression. Easier to point to code than to try to express in verbiage your thoughts on how everything is going to come together, but there are downsides and can be abused.
Actually, I had no idea how it would come together. I just had some vague idea of what I wanted to do and watched what popped out.
>
>
>> First, while I looked at Log4j and somewhat at Logback, most of the core code is completely new. The exception to this is with the Converters for the PatternLayout. I took the EnhancedPatternLayout and modified it.
>>
>
>
> I had totally forgotten that I called that EnhancedPatternLayout, so I wasn't sure if you were talking about the extras version or the sandbox version.
>
>
>> 1. I first created an API that had the features I was looking for. That is in log4j2-api. While it supports logging a String or an Object it really uses a Message interface which is valuable as it allows users to log self-describing objects in a convenient manner.
>
> My thinking was the message interface would end up so minimal that might as well just use Object.
I don't think so. For example, take a look at StructuredDataMessage and LocalizedMessage. Those allow you to do some interesting things. The idea is that it is easy for users to extend without having to muck with the internals. Ceki handled this by creating logback-classis, logback-access and logback-audit. That seemed to me to be a very heavyweight approach to accomplish this.
>
>
>> 2. I don't like the way Logback binds to the implementation. I used a technique I had used in a previous logging framework and used a file to define the implementation class. In theory, the API could be modified to support multiple logging implementation simultaneously, although I have no plans to implement that.
>
> Not something that I've thought about.
Probably not. It just occurred to me that it is possible as I was implementing it but decided it wasn't an itch I needed to scratch at the moment.
>
>
>> 3. Logback suffers from a serious architectural problem that is rooted in Log4j. The configured loggers are mixed with the loggers returned from the Logger factory. This makes it impossible to reconfigure atomically. With Logback the reset method is called on the context which essentially causes the system to be in an undefined state until the new configuration is completed (log records that should be logged are lost). To solve this I used a design that I again borrowed from my previous framework. The configuration is separated and on a reconfiguration the new configuration will be created and then all the loggers will be updated to use it. While there will be a period where some loggers are using the old configuration and some the new there is never a point where loggers aren't configured at all.
>
> I think that is along the line that I was thinking, basically there is an immutable configuration state object that is swapped out as an atomic operation with notification to allocated loggers.
Exactly.
>
>
>> 4. Similar to Logback I added support for Markers and global filters. In addition, filters on Loggers are also supported. Unlike Logback, there is only a single Filter interface used for global filters, logger filters and appender filters.
>
> I've never had a clear description of the use cases behind Markers. As far as I can gather, it is a specialized case of a user supplied context. At the core, I think it would fit into a general user-supplied context object. MDC and NDC would be part of the thread-supplied context and there'd be a global context and a call-site context. In the core, I'd expect the context parameters to just be Object and let the layout level cast to specific known interfaces as needed. During the synchronous extraction phase, an immutable package would be assembled for from log request parameters and the contexts based on the needs of the layout's formatting phase.
I think I'd need to see code to get what you are driving at. In the code I checked in it does have the MDC & NDC and does have a global context, although I haven't added variables to it yet. It also has the configuration which can also have variables, but probably for a different purpose. I don't know what you mean by "call-site context".
The value of Markers is that they are fast and easy to filter on. I use them for Entry and Exit so that you can easily filter them out (admittedly this can also be done via the TRACE level) as well as catching and throwing. In addition, I also use a Marker to identify audit events. Anything with that Marker will always be logged.
>
>
>> 5. The XMLConfiguration is extremely simple. All it does is read the XML and convert the DOM structure to internal Node elements, which contain the node attributes, child node references and placeholders for the real object when it is constructed. It uses the XML element name to match to a Plugin, so instead of writing:
>>
>> <appender name="console" class="org.apache.log4j.ConsoleAppender">
>> <param name="Target" value="System.out"/>
>> <layout class="org.apache.log4j.PatternLayout">
>> <param name="ConversionPattern" value="%-5p %c{1} - %m%n"/>
>> </layout>
>> </appender>
>>
>> you write:
>>
>> <appenders>
>> <Console name="console" target="SYSTEM_OUT">
>> <PatternLayout>%-5p %c{1} - %m%n</PatternLayout>
>> </Console>
>> </appenders>
>>
>> Note that it also would support
>> <Console>
>> <name>console</name>
>> <target>SYSTEM_OUT</target>
>> <PatternLayout>%-5p %c{1} - %m%n</PatternLayout>
>> </Console>
>>
>> if you prefer using elements over attributes.
>
> It is something necessary, but hopefully configuration ends up something outside the core using public methods to do its magic.
The API doesn't have any visibility to the configuration. In this implementation BaseConfiguration really does all the work. Any number of implementations could extend it to support various ways of expressing the configuration.
>
>>
>> 5. I implemented a "Plugin" concept. All core components use annotations to describe their name and type. This is used by the XML configuration to determine the element names that are used in the configuration file.
>> a) Plugins are used for Logger, Appenders, Filters, etc. The BaseConfiguration processes the nodes created by XMLConfiguration and saves the resulting Objects as appropriate.
>> b) PatternLayout uses plugins for the converters. To add a new converter an end user just needs to create the class with the correct annotation and add the package name to the XML configuration as an attribute.
>> c) Configuration uses plugins to determine the Configuration implementation to use. The XMLConfiguration and DefaultConfiguration will always be present, but a weight can be specified to allow the new Configuration to take precedence. Again, simply placing the Configuration class (with the appropriate annotations) in the classpath will cause it to be used. However, if it is in a different package that package will have to be registered to be searched either by calling a static method on the PluginManager or via a system property (which hasn't been implemented yet).
>
> Any thoughts on OSGi or other component frameworks? Not now, maybe, never?
OSGi has to be supported. I would call myself an OSGi novice. Hopefully, this is an area where others with more experience can provide ideas and code.
>
>> 6. java.util.concurrent is used where appropriate. Accordingly, minimal locking takes place. No lock is held while Appenders are called which is a serious bug in Log4j 1.x as it causes deadlocks.
>> 7, All internal components use the logger API for status reporting. A specific StatusLogger provides the implementation.
>> 8. Layouts return a byte array. In Logback Ceki recently came up with the idea of using encoders to allow binary data streams to be sent and received. While the requirement seems valid it seemed awkward to wrap a Layout in an encoder.
>
> I think there will likely need to be separate byte and character Layout-like interfaces. There are sometimes you are writing to a character-oriented destination like a database API and other times to a byte-oriented destination like a file system or network socket. You'd like something like PatternLayout to be able to work in both instances (though it would need to be paired with a character encoder for a byte-oriented destination).
>
> The XMLLayout in log4j 1.2 suffers from this since it really should be byte-oriented. At the present, invalid XML can be generated if you do not use UTF-8 or UTF-16.
>
> The Java serialization in the socket appender is effectively a byte-oriented layout and there is no reason you should not be able to connect it to a file appender.
That is pretty much why I made the Layout return the byte array. Alls the Converters in the PatternLayout manipulate Strings. The resulting String is converted to a byte array at the last moment.
>
>
>> 9. Obviously, everything leverages Java 5 (and might possibly require Java 6 since that is the JVM I've been using).
>>
>> The API is not compatible with log4j 1.x. My intention would be to create a compatible API (to the degree possible) in a log4j12-api package.
>>
>
> I believe that we should try to do design the core so it is the best core and then later see what, if any, needs to be tweaked.
I checked in the log4j 1.x API the other day. It isn't completely compatible as the underlying foundation doesn't have a Hierarchy or Repository. And it may not be compatible operationally. But again, it is a place to start.
Ralph
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org