Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

Hi.

(Moving this discussion to logging dev mailing list.)

Have you tried to use:
<JsonLayout properties="true" eventEol="true" compact="true"/>

Then each log event will be terminated by end-of-line (\r\n).

I think it would be easy to implement 0-byte terminated log events in 
JsonLayout, and that would make sense since we have implemented support 
for that in GelfLayout. Created a JIRA issue for it: 
https://issues.apache.org/jira/browse/LOG4J2-1981

As for other tools, the only receivers for Log4j 2 SerializedLayout we 
know are Log4j's own SocketServer and Lilith. Chainsaw currently only 
support Log4j 1 SerializedLayout. Log4j's own SocketServer support 
JsonLayout and XmlLayout as well, and we are changing it to use 
JsonLayout by default.


On 2017-07-17 15:01, Joern Huxhorn wrote:
> Hi Mikael,
> 
> I've just taken a look at the JsonLayout and see no way to parse its
> output event by event.
> 
> It's semi-fine for files because then it will result in a well-formed
> array at the root level that can be read in one big swoop. It isn't
> really ideal for that use case either since it means that the whole
> array containing all events needs to be read into memory in one piece,
> causing OOM in case of huge files.
> 
> But in Lilith, I really need to process events one by one. This is
> absolutely crucial.
> 
> The only way to achieve this would essentially require me to
> re-implement a JSON parser that can cope with data like this.
> 
> I implemented a similar kludge to support reading of log4j xml files
> (see
> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
> ) but this was way easier since i could simply search for
> "</log4j:event>" to find the position at which I could split the stream
> of events into a valid XML document for every single event . This was
> still kind of nasty but doable, especially since processing an "offline"
> XML file doesn't have the same performance restriction as handling of
> live events.
> 
> I wouldn't have the luxury of such a unique split signal in JSON, though.
> 
> I would need to detect the "}" at the level of the root array which
> would not simply involve counting of opening and closing brackets but
> also ignoring of brackets inside of strings. This would boil down to
> implementing a custom JSON reader and I won't be doing this.
> 
> This isn't just me being lazy either. Such an implementation wouldn't be
> resource (CPU/memory) friendly anyway.
> 
> In my own JSON receiver using my own JSON format there are two ways to
> send those events:
> - write an int containing the amount of bytes representing the event,
> then the bytes of the event. This type of events also supports compression.
> - write the bytes of the event, followed by a 0-byte. This works fine
> and JavaScript/ActionScript is able to produce events like that while
> they are unable (or were unable 7 years ago) to count the bytes on their
> own. This type of events only supports plain text like JSON or XML and
> no compression since compressed events could contain the 0-byte while
> XML and JSON both won't allow it.
> 
> Both are essentially "binary" formats because of either the raw ints or
> the 0-bytes.
> 
> I'm not sure why you deprecate SerializedLayout since the security
> issues arise only while deserializing the events, not while serializing
> them. Is this just meant to educate the user about the issue?
> 
> I fixed the remote code execution exploit by implementing
> https://github.com/huxi/lilith/blob/master/lilith-engine/src/main/java/de/huxhorn/lilith/engine/impl/eventproducer/WhitelistObjectInputStream.java
> 
> This still doesn't fix DoS scenarios (endless loops or OOM in case of
> malicious data) but is close enough for me, mainly because fixing the
> DoS issues would really have to be handled by Java, not by user code.
> Manually re-implementing deserialization would likely be fragile and
> error prone, possibly even introducing other security issues on its own.
> 
> I'd be interested how other tools are tackling this issue. Is Chainsaw
> officially dead or are they implementing receivers for event streams
> like this?
> 
> I'm totally fine with implementing a new receiver as a replacement for
> log4j2 SerializedLayout (even though I won't be able to get rid of the
> Serializable receiver anyway) but the current JsonLayout just isn't up
> to the job at the moment.
> 
> Feel free to share this mail with your fellow developers. I'm open for
> suggestions.
> 
> I hope this doesn't come across in a harsh way. It isn't meant to be.
> 
> Cheers,
> Jörn.

Re: Lilith and Log4j 2

[Ralph Goers <ra...@dslextreme.com>]

From comments from Scott Deboy I believe Chainsaw also supports Log4j 2 but not in an official release.

Ralph

> On Jul 17, 2017, at 12:13 PM, Mikael Ståldal <mi...@apache.org> wrote:
> 
> Hi.
> 
> (Moving this discussion to logging dev mailing list.)
> 
> Have you tried to use:
> <JsonLayout properties="true" eventEol="true" compact="true"/>
> 
> Then each log event will be terminated by end-of-line (\r\n).
> 
> I think it would be easy to implement 0-byte terminated log events in JsonLayout, and that would make sense since we have implemented support for that in GelfLayout. Created a JIRA issue for it: https://issues.apache.org/jira/browse/LOG4J2-1981
> 
> As for other tools, the only receivers for Log4j 2 SerializedLayout we know are Log4j's own SocketServer and Lilith. Chainsaw currently only support Log4j 1 SerializedLayout. Log4j's own SocketServer support JsonLayout and XmlLayout as well, and we are changing it to use JsonLayout by default.
> 
> 
> On 2017-07-17 15:01, Joern Huxhorn wrote:
>> Hi Mikael,
>> I've just taken a look at the JsonLayout and see no way to parse its
>> output event by event.
>> It's semi-fine for files because then it will result in a well-formed
>> array at the root level that can be read in one big swoop. It isn't
>> really ideal for that use case either since it means that the whole
>> array containing all events needs to be read into memory in one piece,
>> causing OOM in case of huge files.
>> But in Lilith, I really need to process events one by one. This is
>> absolutely crucial.
>> The only way to achieve this would essentially require me to
>> re-implement a JSON parser that can cope with data like this.
>> I implemented a similar kludge to support reading of log4j xml files
>> (see
>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
>> ) but this was way easier since i could simply search for
>> "</log4j:event>" to find the position at which I could split the stream
>> of events into a valid XML document for every single event . This was
>> still kind of nasty but doable, especially since processing an "offline"
>> XML file doesn't have the same performance restriction as handling of
>> live events.
>> I wouldn't have the luxury of such a unique split signal in JSON, though.
>> I would need to detect the "}" at the level of the root array which
>> would not simply involve counting of opening and closing brackets but
>> also ignoring of brackets inside of strings. This would boil down to
>> implementing a custom JSON reader and I won't be doing this.
>> This isn't just me being lazy either. Such an implementation wouldn't be
>> resource (CPU/memory) friendly anyway.
>> In my own JSON receiver using my own JSON format there are two ways to
>> send those events:
>> - write an int containing the amount of bytes representing the event,
>> then the bytes of the event. This type of events also supports compression.
>> - write the bytes of the event, followed by a 0-byte. This works fine
>> and JavaScript/ActionScript is able to produce events like that while
>> they are unable (or were unable 7 years ago) to count the bytes on their
>> own. This type of events only supports plain text like JSON or XML and
>> no compression since compressed events could contain the 0-byte while
>> XML and JSON both won't allow it.
>> Both are essentially "binary" formats because of either the raw ints or
>> the 0-bytes.
>> I'm not sure why you deprecate SerializedLayout since the security
>> issues arise only while deserializing the events, not while serializing
>> them. Is this just meant to educate the user about the issue?
>> I fixed the remote code execution exploit by implementing
>> https://github.com/huxi/lilith/blob/master/lilith-engine/src/main/java/de/huxhorn/lilith/engine/impl/eventproducer/WhitelistObjectInputStream.java
>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
>> malicious data) but is close enough for me, mainly because fixing the
>> DoS issues would really have to be handled by Java, not by user code.
>> Manually re-implementing deserialization would likely be fragile and
>> error prone, possibly even introducing other security issues on its own.
>> I'd be interested how other tools are tackling this issue. Is Chainsaw
>> officially dead or are they implementing receivers for event streams
>> like this?
>> I'm totally fine with implementing a new receiver as a replacement for
>> log4j2 SerializedLayout (even though I won't be able to get rid of the
>> Serializable receiver anyway) but the current JsonLayout just isn't up
>> to the job at the moment.
>> Feel free to share this mail with your fellow developers. I'm open for
>> suggestions.
>> I hope this doesn't come across in a harsh way. It isn't meant to be.
>> Cheers,
>> Jörn.
> 
> 

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

Gary, what do you say? I think you once added that comment.


On 2017-07-22 16:27, Jörn Huxhorn wrote:
> Well… it says
> 
> /**
>   * A Jackson JSON {@link ObjectMapper} initialized for Log4j.
>   * <p>
>   * <em>Consider this class private.</em>
>   * </p>
>   */
> 
> so I wanted to holler about how to continue.
> 
> I’ll give it a shot.
> 
> 
> On 22. July 2017 at 15:06:28, Mikael Ståldal (mikes@apache.org) wrote:
>> Would it work to use
>> org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
>> public?
>>   
>> See here how it is used:
>>   
>> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/JsonInputStreamLogEventBridge.java
>>   
>>   
>> On 2017-07-22 13:47, Jörn Huxhorn wrote:
>>> It seems like I don’t get access to the log4j2 ObjectMapper or Jackson Module used for
>> JSON serialization, i.e. I can’t do something like this:
>>> new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace, false)
>>>
>>> It would make sense to reuse your ObjectMapper/Module in my code instead of reimplementing
>> them to prevent future compatibility issues down the road.
>>>
>>>
>>> On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxhorn@googlemail.com) wrote:
>>>> I’ll give this a shot over the next view days.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>> On 19. July 2017 at 21:33:49, Mikael Ståldal (mikes@apache.org) wrote:
>>>>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
>>>>> termination of log events. Will be part of the upcoming 2.9 release.
>>>>>
>>>>>
>>>>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
>>>>>> A general event header and footer would probably be overkill.
>>>>>>
>>>>>> I’m using the following interface in Lilith:
>>>>>>
>>>>>> public interface WriteByteStrategy {
>>>>>> void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
>>>>>> throws IOException;
>>>>>> }
>>>>>>
>>>>>> One implementation writes an int with the length of bytes before “bytes", the other
>>>>> implementation writes a 0-byte after “bytes".
>>>>>>
>>>>>> The problem with that approach is that it requires the intermediate “bytes” array
>>>> and
>>>>> is thus probably incompatible with your zero garbage approach.
>>>>>>
>>>>>> I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this with
>>>> you.
>>>>> I would also have to implement a new receiver type whereas I could just reuse an existing
>>>>> one in case of 0-byte termination.
>>>>>>
>>>>>> I’d prefer using 0-byte termination since it would be less fragile than end-of-line.
>>>>> The end-of-line approach would break if compact=“false” was set erroneously while
>>>>> you’d still have the option of using either compact=“true” or compact=“false” in
>> case
>>>>> of 0-byte termination. 0-byte termination is also slightly more efficient while
>> reading
>>>>> since the splitting can operate on bytes instead of chars or Strings - another reason
>>>>> why I didn’t just implement it and instead wanted to discuss this with you first.
>>>>>>
>>>>>> Joern
>>>>>>
>>>>>>
>>>>>> On 18. July 2017 at 03:26:32, Ralph Goers (ralph.goers@dslextreme.com) wrote:
>>>>>>> Right now that is all handled by the specific layouts. For example, by default the
>>>> RFC5424Layout
>>>>>>> doesn’t append newlines so when writing to a file they will all be on the same “line”,
>>>>> but
>>>>>>> it has an option to append one if needed. Doing the same would be another option for
>>>> the
>>>>>>> JSONLayout.
>>>>>>>
>>>>>>> I’d be ok with event header and footers but ONLY if they have zero overhead when not
>>>> present.
>>>>>>> IOW, when the Layout is initialized it would have to wire in the appropriate encode
>>>>> method.
>>>>>>>
>>>>>>> Ralph
>>>>>>>
>>>>>>>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
>>>>>>>>
>>>>>>>> Do we want a general event header and footer then?
>>>>>>>>
>>>>>>>> Gary
>>>>>>>>
>>>>>>>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
>>>>>>>>
>>>>>>>>> No. A Footer is only used at end of file. He needs to know how long each
>>>>>>>>> event is or when it is the start of a new event.
>>>>>>>>>
>>>>>>>>> Ralph
>>>>>>>>>
>>>>>>>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Can't you use a footer for any terminator you wish?
>>>>>>>>>>
>>>>>>>>>> Gary
>>>>>>>>>>
>>>>>>>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi.
>>>>>>>>>>>
>>>>>>>>>>> (Moving this discussion to logging dev mailing list.)
>>>>>>>>>>>
>>>>>>>>>>> Have you tried to use:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Then each log event will be terminated by end-of-line (\r\n).
>>>>>>>>>>>
>>>>>>>>>>> I think it would be easy to implement 0-byte terminated log events in
>>>>>>>>>>> JsonLayout, and that would make sense since we have implemented support
>>>>>>>>> for
>>>>>>>>>>> that in GelfLayout. Created a JIRA issue for it:
>>>>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
>>>>>>>>>>>
>>>>>>>>>>> As for other tools, the only receivers for Log4j 2 SerializedLayout we
>>>>>>>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
>>>>>>>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
>>>>>>>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
>>>>>>>>> JsonLayout
>>>>>>>>>>> by default.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Mikael,
>>>>>>>>>>>>
>>>>>>>>>>>> I've just taken a look at the JsonLayout and see no way to parse its
>>>>>>>>>>>> output event by event.
>>>>>>>>>>>>
>>>>>>>>>>>> It's semi-fine for files because then it will result in a well-formed
>>>>>>>>>>>> array at the root level that can be read in one big swoop. It isn't
>>>>>>>>>>>> really ideal for that use case either since it means that the whole
>>>>>>>>>>>> array containing all events needs to be read into memory in one piece,
>>>>>>>>>>>> causing OOM in case of huge files.
>>>>>>>>>>>>
>>>>>>>>>>>> But in Lilith, I really need to process events one by one. This is
>>>>>>>>>>>> absolutely crucial.
>>>>>>>>>>>>
>>>>>>>>>>>> The only way to achieve this would essentially require me to
>>>>>>>>>>>> re-implement a JSON parser that can cope with data like this.
>>>>>>>>>>>>
>>>>>>>>>>>> I implemented a similar kludge to support reading of log4j xml files
>>>>>>>>>>>> (see
>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
>>>>>>>>>>>> ) but this was way easier since i could simply search for
>>>>>>>>>>>> "" to find the position at which I could split the stream
>>>>>>>>>>>> of events into a valid XML document for every single event . This was
>>>>>>>>>>>> still kind of nasty but doable, especially since processing an
>>>>>>>>> "offline"
>>>>>>>>>>>> XML file doesn't have the same performance restriction as handling of
>>>>>>>>>>>> live events.
>>>>>>>>>>>>
>>>>>>>>>>>> I wouldn't have the luxury of such a unique split signal in JSON,
>>>>>>>>> though.
>>>>>>>>>>>>
>>>>>>>>>>>> I would need to detect the "}" at the level of the root array which
>>>>>>>>>>>> would not simply involve counting of opening and closing brackets but
>>>>>>>>>>>> also ignoring of brackets inside of strings. This would boil down to
>>>>>>>>>>>> implementing a custom JSON reader and I won't be doing this.
>>>>>>>>>>>>
>>>>>>>>>>>> This isn't just me being lazy either. Such an implementation wouldn't
>>>>>>>>> be
>>>>>>>>>>>> resource (CPU/memory) friendly anyway.
>>>>>>>>>>>>
>>>>>>>>>>>> In my own JSON receiver using my own JSON format there are two ways to
>>>>>>>>>>>> send those events:
>>>>>>>>>>>> - write an int containing the amount of bytes representing the event,
>>>>>>>>>>>> then the bytes of the event. This type of events also supports
>>>>>>>>>>>> compression.
>>>>>>>>>>>> - write the bytes of the event, followed by a 0-byte. This works fine
>>>>>>>>>>>> and JavaScript/ActionScript is able to produce events like that while
>>>>>>>>>>>> they are unable (or were unable 7 years ago) to count the bytes on
>>>>>>>>> their
>>>>>>>>>>>> own. This type of events only supports plain text like JSON or XML and
>>>>>>>>>>>> no compression since compressed events could contain the 0-byte while
>>>>>>>>>>>> XML and JSON both won't allow it.
>>>>>>>>>>>>
>>>>>>>>>>>> Both are essentially "binary" formats because of either the raw ints or
>>>>>>>>>>>> the 0-bytes.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm not sure why you deprecate SerializedLayout since the security
>>>>>>>>>>>> issues arise only while deserializing the events, not while serializing
>>>>>>>>>>>> them. Is this just meant to educate the user about the issue?
>>>>>>>>>>>>
>>>>>>>>>>>> I fixed the remote code execution exploit by implementing
>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
>>>>>>>>>>>> er/WhitelistObjectInputStream.java
>>>>>>>>>>>>
>>>>>>>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
>>>>>>>>>>>> malicious data) but is close enough for me, mainly because fixing the
>>>>>>>>>>>> DoS issues would really have to be handled by Java, not by user code.
>>>>>>>>>>>> Manually re-implementing deserialization would likely be fragile and
>>>>>>>>>>>> error prone, possibly even introducing other security issues on its
>>>>>>>>> own.
>>>>>>>>>>>>
>>>>>>>>>>>> I'd be interested how other tools are tackling this issue. Is Chainsaw
>>>>>>>>>>>> officially dead or are they implementing receivers for event streams
>>>>>>>>>>>> like this?
>>>>>>>>>>>>
>>>>>>>>>>>> I'm totally fine with implementing a new receiver as a replacement for
>>>>>>>>>>>> log4j2 SerializedLayout (even though I won't be able to get rid of the
>>>>>>>>>>>> Serializable receiver anyway) but the current JsonLayout just isn't up
>>>>>>>>>>>> to the job at the moment.
>>>>>>>>>>>>
>>>>>>>>>>>> Feel free to share this mail with your fellow developers. I'm open for
>>>>>>>>>>>> suggestions.
>>>>>>>>>>>>
>>>>>>>>>>>> I hope this doesn't come across in a harsh way. It isn't meant to be.
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Jörn.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>   
>>   
> 

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

That’s what I meant with too lenient. It was probably a bad idea.

Might make more sense to think about which Mixins, if any, should get a @JsonIgnoreProperties(ignoreUnknown=true) annotation… but that is unrelated to the general parsing API. I kind of derailed the topic because this really doesn’t need an immediate decision right now.


On 26. July 2017 at 13:32:28, Mikael Ståldal (mikes@apache.org) wrote:
> A slightly related question, what should the parser do about missing
> properties?
>  
> Currently it is so lenient that it will successfully parse an empty JSON
> object "{}" and just assign null (or other default values) to all fields.
>  
>  
> On 2017-07-25 23:24, Jörn Huxhorn wrote:
> > Hey Mikael,
> > sorry for the slow response time.
> >
> > GMail thought that - of all the dev mailing list mails - this specific mail looked like  
> spam. I just found it hidden in the spam folder...
> >
> > The API looks good to me, assuming that I’ll be fine with the default c’tor versions of  
> Log4jJsonObjectMapper, Log4jYamlObjectMapper and Log4jXmlObjectMapper. Otherwise  
> I'd need matching c’tors in the respective LogEventParser implementations.
> >
> > It may make sense to add this line to the ObjectMapper c'tors:
> > this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true);  
> >
> > This would make parsing less fragile. If new fields are added in future log4j versions  
> then previous versions could still read the event, just without filling the new field.  
> > Another example would be StackTraceElement getting new fields in Java 9, making it  
> possible to still parse Java 9 events in Java 8.
> >
> > Disclaimer: I didn’t test this. I usually annotate specific Mixins with "@JsonIgnoreProperties(ignoreUnknown=true)”  
> where it makes sense. This may be way *too* lenient.
> >
> > It isn’t too critical either and is just meant for your consideration. I’m fine with  
> the parser implementations regardless of such a change.
> >
> > Thanks a lot for providing an official parser API!
> >
> > Cheers,
> > Jörn.
> >
> >
> > On 25. July 2017 at 09:51:56, Mikael Ståldal (mikes@apache.org) wrote:
> >> Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you?
> >>
> >> On 2017-07-22 22:49, Mikael Ståldal wrote:
> >>> Work in progress in Git branch LOG4J2-1986.
> >>>
> >>> On 2017-07-22 22:31, Mikael Ståldal wrote:
> >>>> Yes, I am putting it in its own Java package.
> >>>>
> >>>>
> >>>> On 2017-07-22 22:13, Gary Gregory wrote:
> >>>>> Should this API sit in its own package?
> >>>>>
> >>>>> On Jul 22, 2017 11:41, "Mikael Ståldal" wrote:
> >>>>>
> >>>>>> I'll give it a shot:
> >>>>>>
> >>>>>> https://issues.apache.org/jira/browse/LOG4J2-1986
> >>>>>>
> >>>>>>
> >>>>>> On 2017-07-22 20:28, Ralph Goers wrote:
> >>>>>>
> >>>>>>> Yes, that is a good idea. Probably XML and YAML as well.
> >>>>>>>
> >>>>>>> Ralph
> >>>>>>>
> >>>>>>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote:
> >>>>>>>>
> >>>>>>>> But here we have a concrete use case: a third-party tool, Lilith,
> >>>>>>>> needs
> >>>>>>>> to parse log events from our JsonLayout. (Our SocketServer in
> >>>>>>>> log4j-server
> >>>>>>>> have the same need, and it uses this class.)
> >>>>>>>>
> >>>>>>>> Should we provide a public API in log4j-core for doing so, which
> >>>>>>>> Lilith,
> >>>>>>>> our SocketServer, and possibly others (Chainsaw maybe?) can use?
> >>>>>>>> Maybe such
> >>>>>>>> public API should be a facade around this which hides the fact
> >>>>>>>> that we use
> >>>>>>>> Jackson.
>  

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

A slightly related question, what should the parser do about missing 
properties?

Currently it is so lenient that it will successfully parse an empty JSON 
object "{}" and just assign null (or other default values) to all fields.


On 2017-07-25 23:24, Jörn Huxhorn wrote:
> Hey Mikael,
> sorry for the slow response time.
> 
> GMail thought that - of all the dev mailing list mails - this specific mail looked like spam. I just found it hidden in the spam folder...
> 
> The API looks good to me, assuming that I’ll be fine with the default c’tor versions of Log4jJsonObjectMapper, Log4jYamlObjectMapper and Log4jXmlObjectMapper. Otherwise I'd need matching c’tors in the respective LogEventParser implementations.
> 
> It may make sense to add this line to the ObjectMapper c'tors:
> this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true);
> 
> This would make parsing less fragile. If new fields are added in future log4j versions then previous versions could still read the event, just without filling the new field.
> Another example would be StackTraceElement getting new fields in Java 9, making it possible to still parse Java 9 events in Java 8.
> 
> Disclaimer: I didn’t test this. I usually annotate specific Mixins with "@JsonIgnoreProperties(ignoreUnknown=true)” where it makes sense. This may be way *too* lenient.
> 
> It isn’t too critical either and is just meant for your consideration. I’m fine with the parser implementations regardless of such a change.
> 
> Thanks a lot for providing an official parser API!
> 
> Cheers,
> Jörn.
> 
> 
> On 25. July 2017 at 09:51:56, Mikael Ståldal (mikes@apache.org) wrote:
>> Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you?
>>   
>> On 2017-07-22 22:49, Mikael Ståldal wrote:
>>> Work in progress in Git branch LOG4J2-1986.
>>>
>>> On 2017-07-22 22:31, Mikael Ståldal wrote:
>>>> Yes, I am putting it in its own Java package.
>>>>
>>>>
>>>> On 2017-07-22 22:13, Gary Gregory wrote:
>>>>> Should this API sit in its own package?
>>>>>
>>>>> On Jul 22, 2017 11:41, "Mikael Ståldal" wrote:
>>>>>
>>>>>> I'll give it a shot:
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1986
>>>>>>
>>>>>>
>>>>>> On 2017-07-22 20:28, Ralph Goers wrote:
>>>>>>
>>>>>>> Yes, that is a good idea. Probably XML and YAML as well.
>>>>>>>
>>>>>>> Ralph
>>>>>>>
>>>>>>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote:
>>>>>>>>
>>>>>>>> But here we have a concrete use case: a third-party tool, Lilith,
>>>>>>>> needs
>>>>>>>> to parse log events from our JsonLayout. (Our SocketServer in
>>>>>>>> log4j-server
>>>>>>>> have the same need, and it uses this class.)
>>>>>>>>
>>>>>>>> Should we provide a public API in log4j-core for doing so, which
>>>>>>>> Lilith,
>>>>>>>> our SocketServer, and possibly others (Chainsaw maybe?) can use?
>>>>>>>> Maybe such
>>>>>>>> public API should be a facade around this which hides the fact
>>>>>>>> that we use
>>>>>>>> Jackson.

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

That test was added as part of LOG4J2-1961 by Gary.


On 2017-08-03 16:00, Ralph Goers wrote:
> The error with the missing artifact happens when you skip tests. The test that failed for you never fails for me on OS X but fails randomly in Jenkins.  I’m not sure why no one has looked at it yet.
> 
> Ralph
> 
>> On Aug 3, 2017, at 6:52 AM, Jörn Huxhorn <jh...@googlemail.com> wrote:
>>
>> OK, everything works for me with 2.9-SNAPSHOT.
>> I can receive JSON, YAML and XML events just fine.
>>
>> I had the following error while building:
>> [ERROR] Failed to execute goal on project log4j-core-its: Could not resolve dependencies for project org.apache.logging.log4j:log4j-core-its:jar:2.9-SNAPSHOT: Could not find artifact org.apache.logging.log4j:log4j-server:jar:tests:2.9-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots), try downloading from https://logging.apache.org/log4j/2.x/download.html -> [Help 1]
>>
>> That wasn’t a serious problem for me since I only need API and core.
>> But I wanted to inform you about this issue since building log4j worked fine for me just a few days ago.
>>
>> I’m sure it worked at d3b722083b8f21ea105a4f01132f08b373cb401e and I think it also worked at b96f5220dbebf6e0a4fb9f7da07c3a9d7d443418
>>
>> Cheers,
>> Jörn.
>>
>>
>> On 3. August 2017 at 13:30:24, Mikael Ståldal (mikes@apache.org) wrote:
>>> Yes, that test is known to be unstablle.
>>>
>>> Just build with -DskipTests
>>>
>>>
>>> On 2017-08-03 11:24, Jörn Huxhorn wrote:
>>>> I get test failures building against HEAD (3ac727026f295941fe71a34d6c80afc242f38c0f).
>>>>
>>>> Environment:
>>>> $ mvn -version
>>>> Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T21:39:06+02:00)
>>>> Maven home: /Users/huxi/.sdkman/candidates/maven/current
>>>> Java version: 1.8.0_144, vendor: Oracle Corporation
>>>> Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre
>>>> Default locale: en_US, platform encoding: UTF-8
>>>> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: “mac"
>>>>
>>>> Test failure:
>>>> -------------------------------------------------------------------------------
>>>> Test set: org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest
>>>> -------------------------------------------------------------------------------
>>>> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.518 s <<< FAILURE!
>>> - in org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest
>>>> testRollingFileAppenderWithReconfigure(org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest)
>>> Time elapsed: 2.518 s <<< FAILURE!
>>>> java.lang.AssertionError:
>>>>
>>>> Expected: is <2>
>>>> but: was <3>
>>>> at org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest.testRollingFileAppenderWithReconfigure(RollingAppenderDirectWriteWithReconfigureTest.java:68)
>>>>
>>>>
>>>>
>>>>
>>>> On 2. August 2017 at 11:30:47, Mikael Ståldal (mikes@apache.org) wrote:
>>>>> In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout.
>>>>>
>>>>>
>>>>> On 2017-08-02 09:23, Jörn Huxhorn wrote:
>>>>>> I started working on the Lilith side of this and I think I’m pretty much done.
>>>>>>
>>>>>> To test the new functionality in a sandbox application, I’d now need the ability to
>>> switch
>>>>> JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with
>>> a 0-byte
>>>>> instead of being separated by “, ".
>>>>>>
>>>>>>
>>>>>> On 26. July 2017 at 21:14:41, Mikael Ståldal (mikes@apache.org) wrote:
>>>>>>> That would make sense.
>>>>>>>
>>>>>>> On 2017-07-26 19:51, Jörn Huxhorn wrote:
>>>>>>>> Isn’t this supposed to also use 0-byte terminated events in the future?
>>>>>>>> Because in that case it would make sense to change from String to byte[] processing.
>>>>>>>>
>>>>>>>> On 26. July 2017 at 17:49:07, Mikael Ståldal (mikes@apache.org) wrote:
>>>>>>>>> They just need the String version:
>>>>>>>>>
>>>>>>>>> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2017-07-26 17:40, Gary Gregory wrote:
>>>>>>>>>> What do our XML and JSON server need? Let's make sure we handle those use
>>>>>>>>>> cases.
>>>>>>>>>>
>>>>>>>>>> Gary
>>>>>>>>>>
>>>>>>>>>> On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote:
>>>>>>>>>>
>>>>>>>>>>> +1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
>>>>>>>>>>>> Maybe I should remove both the Reader and the InputStream versions,
>>>>>>>>>>>> since they cannot parse multiple log events from a stream, and are thus
>>>>>>>>>>>> not very useful.
>>>>>>>>>>>>
>>>>>>>>>>>> On 2017-07-26 10:34, Jörn Huxhorn wrote:
>>>>>>>>>>>>> It would maybe be a good idea to get rid of the Reader method
>>>>>>>>>>> altogether since it mainly
>>>>>>>>>>>> introduces an unnecessary point of failure if the Reader is using an
>>>>>>>>>>> encoding other than
>>>>>>>>>>>> UTF-8.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Using a FileReader would be an example for this since its limited
>>>>>>>>>>> c'tors are always using
>>>>>>>>>>>> Charset.defaultCharset().name().
>>>>>>>>>>>>>
>>>>>>>>>>>>> The correct way to create an UTF-8 file reader is the less than
>>>>>>>>>>> obvious "new InputStreamReader(new
>>>>>>>>>>>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common
>>>>>>>>>>> mistake.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
> 
> 

Re: Lilith and Log4j 2

[Ralph Goers <ra...@dslextreme.com>]

The error with the missing artifact happens when you skip tests. The test that failed for you never fails for me on OS X but fails randomly in Jenkins.  I’m not sure why no one has looked at it yet.

Ralph

> On Aug 3, 2017, at 6:52 AM, Jörn Huxhorn <jh...@googlemail.com> wrote:
> 
> OK, everything works for me with 2.9-SNAPSHOT.
> I can receive JSON, YAML and XML events just fine.
> 
> I had the following error while building:
> [ERROR] Failed to execute goal on project log4j-core-its: Could not resolve dependencies for project org.apache.logging.log4j:log4j-core-its:jar:2.9-SNAPSHOT: Could not find artifact org.apache.logging.log4j:log4j-server:jar:tests:2.9-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots), try downloading from https://logging.apache.org/log4j/2.x/download.html -> [Help 1]
> 
> That wasn’t a serious problem for me since I only need API and core.
> But I wanted to inform you about this issue since building log4j worked fine for me just a few days ago.
> 
> I’m sure it worked at d3b722083b8f21ea105a4f01132f08b373cb401e and I think it also worked at b96f5220dbebf6e0a4fb9f7da07c3a9d7d443418
> 
> Cheers,
> Jörn.
> 
> 
> On 3. August 2017 at 13:30:24, Mikael Ståldal (mikes@apache.org) wrote:
>> Yes, that test is known to be unstablle.
>> 
>> Just build with -DskipTests
>> 
>> 
>> On 2017-08-03 11:24, Jörn Huxhorn wrote:
>>> I get test failures building against HEAD (3ac727026f295941fe71a34d6c80afc242f38c0f).  
>>> 
>>> Environment:
>>> $ mvn -version
>>> Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T21:39:06+02:00)  
>>> Maven home: /Users/huxi/.sdkman/candidates/maven/current
>>> Java version: 1.8.0_144, vendor: Oracle Corporation
>>> Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre  
>>> Default locale: en_US, platform encoding: UTF-8
>>> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: “mac"
>>> 
>>> Test failure:
>>> -------------------------------------------------------------------------------  
>>> Test set: org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest  
>>> -------------------------------------------------------------------------------  
>>> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.518 s <<< FAILURE!  
>> - in org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest  
>>> testRollingFileAppenderWithReconfigure(org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest)  
>> Time elapsed: 2.518 s <<< FAILURE!
>>> java.lang.AssertionError:
>>> 
>>> Expected: is <2>
>>> but: was <3>
>>> at org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest.testRollingFileAppenderWithReconfigure(RollingAppenderDirectWriteWithReconfigureTest.java:68)  
>>> 
>>> 
>>> 
>>> 
>>> On 2. August 2017 at 11:30:47, Mikael Ståldal (mikes@apache.org) wrote:
>>>> In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout.
>>>> 
>>>> 
>>>> On 2017-08-02 09:23, Jörn Huxhorn wrote:
>>>>> I started working on the Lilith side of this and I think I’m pretty much done.
>>>>> 
>>>>> To test the new functionality in a sandbox application, I’d now need the ability to  
>> switch
>>>> JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with  
>> a 0-byte
>>>> instead of being separated by “, ".
>>>>> 
>>>>> 
>>>>> On 26. July 2017 at 21:14:41, Mikael Ståldal (mikes@apache.org) wrote:
>>>>>> That would make sense.
>>>>>> 
>>>>>> On 2017-07-26 19:51, Jörn Huxhorn wrote:
>>>>>>> Isn’t this supposed to also use 0-byte terminated events in the future?
>>>>>>> Because in that case it would make sense to change from String to byte[] processing.  
>>>>>>> 
>>>>>>> On 26. July 2017 at 17:49:07, Mikael Ståldal (mikes@apache.org) wrote:
>>>>>>>> They just need the String version:
>>>>>>>> 
>>>>>>>> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100  
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 2017-07-26 17:40, Gary Gregory wrote:
>>>>>>>>> What do our XML and JSON server need? Let's make sure we handle those use
>>>>>>>>> cases.
>>>>>>>>> 
>>>>>>>>> Gary
>>>>>>>>> 
>>>>>>>>> On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote:
>>>>>>>>> 
>>>>>>>>>> +1
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
>>>>>>>>>>> Maybe I should remove both the Reader and the InputStream versions,
>>>>>>>>>>> since they cannot parse multiple log events from a stream, and are thus
>>>>>>>>>>> not very useful.
>>>>>>>>>>> 
>>>>>>>>>>> On 2017-07-26 10:34, Jörn Huxhorn wrote:
>>>>>>>>>>>> It would maybe be a good idea to get rid of the Reader method
>>>>>>>>>> altogether since it mainly
>>>>>>>>>>> introduces an unnecessary point of failure if the Reader is using an
>>>>>>>>>> encoding other than
>>>>>>>>>>> UTF-8.
>>>>>>>>>>>> 
>>>>>>>>>>>> Using a FileReader would be an example for this since its limited
>>>>>>>>>> c'tors are always using
>>>>>>>>>>> Charset.defaultCharset().name().
>>>>>>>>>>>> 
>>>>>>>>>>>> The correct way to create an UTF-8 file reader is the less than
>>>>>>>>>> obvious "new InputStreamReader(new
>>>>>>>>>>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common  
>>>>>>>>>> mistake.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
> 
> 

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

OK, everything works for me with 2.9-SNAPSHOT.
I can receive JSON, YAML and XML events just fine.

I had the following error while building:
[ERROR] Failed to execute goal on project log4j-core-its: Could not resolve dependencies for project org.apache.logging.log4j:log4j-core-its:jar:2.9-SNAPSHOT: Could not find artifact org.apache.logging.log4j:log4j-server:jar:tests:2.9-SNAPSHOT in apache.snapshots (https://repository.apache.org/snapshots), try downloading from https://logging.apache.org/log4j/2.x/download.html -> [Help 1]

That wasn’t a serious problem for me since I only need API and core.
But I wanted to inform you about this issue since building log4j worked fine for me just a few days ago.

I’m sure it worked at d3b722083b8f21ea105a4f01132f08b373cb401e and I think it also worked at b96f5220dbebf6e0a4fb9f7da07c3a9d7d443418

Cheers,
Jörn.


On 3. August 2017 at 13:30:24, Mikael Ståldal (mikes@apache.org) wrote:
> Yes, that test is known to be unstablle.
>  
> Just build with -DskipTests
>  
>  
> On 2017-08-03 11:24, Jörn Huxhorn wrote:
> > I get test failures building against HEAD (3ac727026f295941fe71a34d6c80afc242f38c0f).  
> >
> > Environment:
> > $ mvn -version
> > Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T21:39:06+02:00)  
> > Maven home: /Users/huxi/.sdkman/candidates/maven/current
> > Java version: 1.8.0_144, vendor: Oracle Corporation
> > Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre  
> > Default locale: en_US, platform encoding: UTF-8
> > OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: “mac"
> >
> > Test failure:
> > -------------------------------------------------------------------------------  
> > Test set: org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest  
> > -------------------------------------------------------------------------------  
> > Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.518 s <<< FAILURE!  
> - in org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest  
> > testRollingFileAppenderWithReconfigure(org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest)  
> Time elapsed: 2.518 s <<< FAILURE!
> > java.lang.AssertionError:
> >
> > Expected: is <2>
> > but: was <3>
> > at org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest.testRollingFileAppenderWithReconfigure(RollingAppenderDirectWriteWithReconfigureTest.java:68)  
> >
> >
> >
> >
> > On 2. August 2017 at 11:30:47, Mikael Ståldal (mikes@apache.org) wrote:
> >> In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout.
> >>
> >>
> >> On 2017-08-02 09:23, Jörn Huxhorn wrote:
> >>> I started working on the Lilith side of this and I think I’m pretty much done.
> >>>
> >>> To test the new functionality in a sandbox application, I’d now need the ability to  
> switch
> >> JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with  
> a 0-byte
> >> instead of being separated by “, ".
> >>>
> >>>
> >>> On 26. July 2017 at 21:14:41, Mikael Ståldal (mikes@apache.org) wrote:
> >>>> That would make sense.
> >>>>
> >>>> On 2017-07-26 19:51, Jörn Huxhorn wrote:
> >>>>> Isn’t this supposed to also use 0-byte terminated events in the future?
> >>>>> Because in that case it would make sense to change from String to byte[] processing.  
> >>>>>
> >>>>> On 26. July 2017 at 17:49:07, Mikael Ståldal (mikes@apache.org) wrote:
> >>>>>> They just need the String version:
> >>>>>>
> >>>>>> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100  
> >>>>>>
> >>>>>>
> >>>>>> On 2017-07-26 17:40, Gary Gregory wrote:
> >>>>>>> What do our XML and JSON server need? Let's make sure we handle those use
> >>>>>>> cases.
> >>>>>>>
> >>>>>>> Gary
> >>>>>>>
> >>>>>>> On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote:
> >>>>>>>
> >>>>>>>> +1
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
> >>>>>>>>> Maybe I should remove both the Reader and the InputStream versions,
> >>>>>>>>> since they cannot parse multiple log events from a stream, and are thus
> >>>>>>>>> not very useful.
> >>>>>>>>>
> >>>>>>>>> On 2017-07-26 10:34, Jörn Huxhorn wrote:
> >>>>>>>>>> It would maybe be a good idea to get rid of the Reader method
> >>>>>>>> altogether since it mainly
> >>>>>>>>> introduces an unnecessary point of failure if the Reader is using an
> >>>>>>>> encoding other than
> >>>>>>>>> UTF-8.
> >>>>>>>>>>
> >>>>>>>>>> Using a FileReader would be an example for this since its limited
> >>>>>>>> c'tors are always using
> >>>>>>>>> Charset.defaultCharset().name().
> >>>>>>>>>>
> >>>>>>>>>> The correct way to create an UTF-8 file reader is the less than
> >>>>>>>> obvious "new InputStreamReader(new
> >>>>>>>>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common  
> >>>>>>>> mistake.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>  
>  

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

Yes, that test is known to be unstablle.

Just build with -DskipTests


On 2017-08-03 11:24, Jörn Huxhorn wrote:
> I get test failures building against HEAD (3ac727026f295941fe71a34d6c80afc242f38c0f).
> 
> Environment:
> $ mvn -version
> Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T21:39:06+02:00)
> Maven home: /Users/huxi/.sdkman/candidates/maven/current
> Java version: 1.8.0_144, vendor: Oracle Corporation
> Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: “mac"
> 
> Test failure:
> -------------------------------------------------------------------------------
> Test set: org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.518 s <<< FAILURE! - in org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest
> testRollingFileAppenderWithReconfigure(org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest)  Time elapsed: 2.518 s  <<< FAILURE!
> java.lang.AssertionError:
> 
> Expected: is <2>
>       but: was <3>
> 	at org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest.testRollingFileAppenderWithReconfigure(RollingAppenderDirectWriteWithReconfigureTest.java:68)
> 
> 
> 
> 
> On 2. August 2017 at 11:30:47, Mikael Ståldal (mikes@apache.org) wrote:
>> In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout.
>>   
>>   
>> On 2017-08-02 09:23, Jörn Huxhorn wrote:
>>> I started working on the Lilith side of this and I think I’m pretty much done.
>>>
>>> To test the new functionality in a sandbox application, I’d now need the ability to switch
>> JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with a 0-byte
>> instead of being separated by “, ".
>>>
>>>
>>> On 26. July 2017 at 21:14:41, Mikael Ståldal (mikes@apache.org) wrote:
>>>> That would make sense.
>>>>
>>>> On 2017-07-26 19:51, Jörn Huxhorn wrote:
>>>>> Isn’t this supposed to also use 0-byte terminated events in the future?
>>>>> Because in that case it would make sense to change from String to byte[] processing.
>>>>>
>>>>> On 26. July 2017 at 17:49:07, Mikael Ståldal (mikes@apache.org) wrote:
>>>>>> They just need the String version:
>>>>>>
>>>>>> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100
>>>>>>
>>>>>>
>>>>>> On 2017-07-26 17:40, Gary Gregory wrote:
>>>>>>> What do our XML and JSON server need? Let's make sure we handle those use
>>>>>>> cases.
>>>>>>>
>>>>>>> Gary
>>>>>>>
>>>>>>> On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote:
>>>>>>>
>>>>>>>> +1
>>>>>>>>
>>>>>>>>
>>>>>>>> On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
>>>>>>>>> Maybe I should remove both the Reader and the InputStream versions,
>>>>>>>>> since they cannot parse multiple log events from a stream, and are thus
>>>>>>>>> not very useful.
>>>>>>>>>
>>>>>>>>> On 2017-07-26 10:34, Jörn Huxhorn wrote:
>>>>>>>>>> It would maybe be a good idea to get rid of the Reader method
>>>>>>>> altogether since it mainly
>>>>>>>>> introduces an unnecessary point of failure if the Reader is using an
>>>>>>>> encoding other than
>>>>>>>>> UTF-8.
>>>>>>>>>>
>>>>>>>>>> Using a FileReader would be an example for this since its limited
>>>>>>>> c'tors are always using
>>>>>>>>> Charset.defaultCharset().name().
>>>>>>>>>>
>>>>>>>>>> The correct way to create an UTF-8 file reader is the less than
>>>>>>>> obvious "new InputStreamReader(new
>>>>>>>>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common
>>>>>>>> mistake.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>   
>>   
> 

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

I get test failures building against HEAD (3ac727026f295941fe71a34d6c80afc242f38c0f).

Environment:
$ mvn -version
Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T21:39:06+02:00)
Maven home: /Users/huxi/.sdkman/candidates/maven/current
Java version: 1.8.0_144, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_144.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: “mac"

Test failure:
-------------------------------------------------------------------------------
Test set: org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.518 s <<< FAILURE! - in org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest
testRollingFileAppenderWithReconfigure(org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest)  Time elapsed: 2.518 s  <<< FAILURE!
java.lang.AssertionError: 

Expected: is <2>
     but: was <3>
	at org.apache.logging.log4j.core.appender.rolling.RollingAppenderDirectWriteWithReconfigureTest.testRollingFileAppenderWithReconfigure(RollingAppenderDirectWriteWithReconfigureTest.java:68)




On 2. August 2017 at 11:30:47, Mikael Ståldal (mikes@apache.org) wrote:
> In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout.
>  
>  
> On 2017-08-02 09:23, Jörn Huxhorn wrote:
> > I started working on the Lilith side of this and I think I’m pretty much done.
> >
> > To test the new functionality in a sandbox application, I’d now need the ability to switch  
> JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with a 0-byte  
> instead of being separated by “, ".
> >
> >
> > On 26. July 2017 at 21:14:41, Mikael Ståldal (mikes@apache.org) wrote:
> >> That would make sense.
> >>
> >> On 2017-07-26 19:51, Jörn Huxhorn wrote:
> >>> Isn’t this supposed to also use 0-byte terminated events in the future?
> >>> Because in that case it would make sense to change from String to byte[] processing.  
> >>>
> >>> On 26. July 2017 at 17:49:07, Mikael Ståldal (mikes@apache.org) wrote:
> >>>> They just need the String version:
> >>>>
> >>>> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100  
> >>>>
> >>>>
> >>>> On 2017-07-26 17:40, Gary Gregory wrote:
> >>>>> What do our XML and JSON server need? Let's make sure we handle those use
> >>>>> cases.
> >>>>>
> >>>>> Gary
> >>>>>
> >>>>> On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote:
> >>>>>
> >>>>>> +1
> >>>>>>
> >>>>>>
> >>>>>> On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
> >>>>>>> Maybe I should remove both the Reader and the InputStream versions,
> >>>>>>> since they cannot parse multiple log events from a stream, and are thus
> >>>>>>> not very useful.
> >>>>>>>
> >>>>>>> On 2017-07-26 10:34, Jörn Huxhorn wrote:
> >>>>>>>> It would maybe be a good idea to get rid of the Reader method
> >>>>>> altogether since it mainly
> >>>>>>> introduces an unnecessary point of failure if the Reader is using an
> >>>>>> encoding other than
> >>>>>>> UTF-8.
> >>>>>>>>
> >>>>>>>> Using a FileReader would be an example for this since its limited
> >>>>>> c'tors are always using
> >>>>>>> Charset.defaultCharset().name().
> >>>>>>>>
> >>>>>>>> The correct way to create an UTF-8 file reader is the less than
> >>>>>> obvious "new InputStreamReader(new
> >>>>>>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common  
> >>>>>> mistake.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>  
>  

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

In 2.9-SNAPSHOT, you can use includeNullDelimiter="true" on the layout.


On 2017-08-02 09:23, Jörn Huxhorn wrote:
> I started working on the Lilith side of this and I think I’m pretty much done.
> 
> To test the new functionality in a sandbox application, I’d now need the ability to switch JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with a 0-byte instead of being separated by “, ".
> 
> 
> On 26. July 2017 at 21:14:41, Mikael Ståldal (mikes@apache.org) wrote:
>> That would make sense.
>>   
>> On 2017-07-26 19:51, Jörn Huxhorn wrote:
>>> Isn’t this supposed to also use 0-byte terminated events in the future?
>>> Because in that case it would make sense to change from String to byte[] processing.
>>>
>>> On 26. July 2017 at 17:49:07, Mikael Ståldal (mikes@apache.org) wrote:
>>>> They just need the String version:
>>>>
>>>> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100
>>>>
>>>>
>>>> On 2017-07-26 17:40, Gary Gregory wrote:
>>>>> What do our XML and JSON server need? Let's make sure we handle those use
>>>>> cases.
>>>>>
>>>>> Gary
>>>>>
>>>>> On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote:
>>>>>
>>>>>> +1
>>>>>>
>>>>>>
>>>>>> On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
>>>>>>> Maybe I should remove both the Reader and the InputStream versions,
>>>>>>> since they cannot parse multiple log events from a stream, and are thus
>>>>>>> not very useful.
>>>>>>>
>>>>>>> On 2017-07-26 10:34, Jörn Huxhorn wrote:
>>>>>>>> It would maybe be a good idea to get rid of the Reader method
>>>>>> altogether since it mainly
>>>>>>> introduces an unnecessary point of failure if the Reader is using an
>>>>>> encoding other than
>>>>>>> UTF-8.
>>>>>>>>
>>>>>>>> Using a FileReader would be an example for this since its limited
>>>>>> c'tors are always using
>>>>>>> Charset.defaultCharset().name().
>>>>>>>>
>>>>>>>> The correct way to create an UTF-8 file reader is the less than
>>>>>> obvious "new InputStreamReader(new
>>>>>>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common
>>>>>> mistake.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>   
>>   
> 

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

I started working on the Lilith side of this and I think I’m pretty much done.

To test the new functionality in a sandbox application, I’d now need the ability to switch JsonLayout, YamlLayout and XmlLayout into a mode were events are terminated with a 0-byte instead of being separated by “, ".


On 26. July 2017 at 21:14:41, Mikael Ståldal (mikes@apache.org) wrote:
> That would make sense.
>  
> On 2017-07-26 19:51, Jörn Huxhorn wrote:
> > Isn’t this supposed to also use 0-byte terminated events in the future?
> > Because in that case it would make sense to change from String to byte[] processing.  
> >
> > On 26. July 2017 at 17:49:07, Mikael Ståldal (mikes@apache.org) wrote:
> >> They just need the String version:
> >>
> >> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100  
> >>
> >>
> >> On 2017-07-26 17:40, Gary Gregory wrote:
> >>> What do our XML and JSON server need? Let's make sure we handle those use
> >>> cases.
> >>>
> >>> Gary
> >>>
> >>> On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote:
> >>>
> >>>> +1
> >>>>
> >>>>
> >>>> On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
> >>>>> Maybe I should remove both the Reader and the InputStream versions,
> >>>>> since they cannot parse multiple log events from a stream, and are thus
> >>>>> not very useful.
> >>>>>
> >>>>> On 2017-07-26 10:34, Jörn Huxhorn wrote:
> >>>>>> It would maybe be a good idea to get rid of the Reader method
> >>>> altogether since it mainly
> >>>>> introduces an unnecessary point of failure if the Reader is using an
> >>>> encoding other than
> >>>>> UTF-8.
> >>>>>>
> >>>>>> Using a FileReader would be an example for this since its limited
> >>>> c'tors are always using
> >>>>> Charset.defaultCharset().name().
> >>>>>>
> >>>>>> The correct way to create an UTF-8 file reader is the less than
> >>>> obvious "new InputStreamReader(new
> >>>>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common
> >>>> mistake.
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>  
>  

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

That would make sense.

On 2017-07-26 19:51, Jörn Huxhorn wrote:
> Isn’t this supposed to also use 0-byte terminated events in the future?
> Because in that case it would make sense to change from String to byte[] processing.
> 
> On 26. July 2017 at 17:49:07, Mikael Ståldal (mikes@apache.org) wrote:
>> They just need the String version:
>>   
>> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100
>>   
>>   
>> On 2017-07-26 17:40, Gary Gregory wrote:
>>> What do our XML and JSON server need? Let's make sure we handle those use
>>> cases.
>>>
>>> Gary
>>>
>>> On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote:
>>>
>>>> +1
>>>>
>>>>
>>>> On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
>>>>> Maybe I should remove both the Reader and the InputStream versions,
>>>>> since they cannot parse multiple log events from a stream, and are thus
>>>>> not very useful.
>>>>>
>>>>> On 2017-07-26 10:34, Jörn Huxhorn wrote:
>>>>>> It would maybe be a good idea to get rid of the Reader method
>>>> altogether since it mainly
>>>>> introduces an unnecessary point of failure if the Reader is using an
>>>> encoding other than
>>>>> UTF-8.
>>>>>>
>>>>>> Using a FileReader would be an example for this since its limited
>>>> c'tors are always using
>>>>> Charset.defaultCharset().name().
>>>>>>
>>>>>> The correct way to create an UTF-8 file reader is the less than
>>>> obvious "new InputStreamReader(new
>>>>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common
>>>> mistake.
>>>>>
>>>>
>>>>
>>>
>>   
>>   
> 

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

Isn’t this supposed to also use 0-byte terminated events in the future?
Because in that case it would make sense to change from String to byte[] processing.

On 26. July 2017 at 17:49:07, Mikael Ståldal (mikes@apache.org) wrote:
> They just need the String version:
>  
> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100  
>  
>  
> On 2017-07-26 17:40, Gary Gregory wrote:
> > What do our XML and JSON server need? Let's make sure we handle those use
> > cases.
> >
> > Gary
> >
> > On Jul 26, 2017 05:30, "Jörn Huxhorn" wrote:
> >
> >> +1
> >>
> >>
> >> On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
> >>> Maybe I should remove both the Reader and the InputStream versions,
> >>> since they cannot parse multiple log events from a stream, and are thus
> >>> not very useful.
> >>>
> >>> On 2017-07-26 10:34, Jörn Huxhorn wrote:
> >>>> It would maybe be a good idea to get rid of the Reader method
> >> altogether since it mainly
> >>> introduces an unnecessary point of failure if the Reader is using an
> >> encoding other than
> >>> UTF-8.
> >>>>
> >>>> Using a FileReader would be an example for this since its limited
> >> c'tors are always using
> >>> Charset.defaultCharset().name().
> >>>>
> >>>> The correct way to create an UTF-8 file reader is the less than
> >> obvious "new InputStreamReader(new
> >>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common
> >> mistake.
> >>>
> >>
> >>
> >
>  
>  

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

They just need the String version:

https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/InputStreamLogEventBridge.java#L100


On 2017-07-26 17:40, Gary Gregory wrote:
> What do our XML and JSON server need? Let's make sure we handle those use
> cases.
> 
> Gary
> 
> On Jul 26, 2017 05:30, "Jörn Huxhorn" <jh...@googlemail.com> wrote:
> 
>> +1
>>
>>
>> On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
>>> Maybe I should remove both the Reader and the InputStream versions,
>>> since they cannot parse multiple log events from a stream, and are thus
>>> not very useful.
>>>
>>> On 2017-07-26 10:34, Jörn Huxhorn wrote:
>>>> It would maybe be a good idea to get rid of the Reader method
>> altogether since it mainly
>>> introduces an unnecessary point of failure if the Reader is using an
>> encoding other than
>>> UTF-8.
>>>>
>>>> Using a FileReader would be an example for this since its limited
>> c'tors are always using
>>> Charset.defaultCharset().name().
>>>>
>>>> The correct way to create an UTF-8 file reader is the less than
>> obvious "new InputStreamReader(new
>>> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common
>> mistake.
>>>
>>
>>
> 

Re: Lilith and Log4j 2

[Gary Gregory <ga...@gmail.com>]

What do our XML and JSON server need? Let's make sure we handle those use
cases.

Gary

On Jul 26, 2017 05:30, "Jörn Huxhorn" <jh...@googlemail.com> wrote:

> +1
>
>
> On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
> > Maybe I should remove both the Reader and the InputStream versions,
> > since they cannot parse multiple log events from a stream, and are thus
> > not very useful.
> >
> > On 2017-07-26 10:34, Jörn Huxhorn wrote:
> > > It would maybe be a good idea to get rid of the Reader method
> altogether since it mainly
> > introduces an unnecessary point of failure if the Reader is using an
> encoding other than
> > UTF-8.
> > >
> > > Using a FileReader would be an example for this since its limited
> c'tors are always using
> > Charset.defaultCharset().name().
> > >
> > > The correct way to create an UTF-8 file reader is the less than
> obvious "new InputStreamReader(new
> > FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common
> mistake.
> >
>
>

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

+1


On 26. July 2017 at 13:29:27, Mikael Ståldal (mikes@apache.org) wrote:
> Maybe I should remove both the Reader and the InputStream versions,
> since they cannot parse multiple log events from a stream, and are thus
> not very useful.
>  
> On 2017-07-26 10:34, Jörn Huxhorn wrote:
> > It would maybe be a good idea to get rid of the Reader method altogether since it mainly  
> introduces an unnecessary point of failure if the Reader is using an encoding other than  
> UTF-8.
> >
> > Using a FileReader would be an example for this since its limited c'tors are always using  
> Charset.defaultCharset().name().
> >
> > The correct way to create an UTF-8 file reader is the less than obvious "new InputStreamReader(new  
> FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common mistake.  
>  

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

Maybe I should remove both the Reader and the InputStream versions, 
since they cannot parse multiple log events from a stream, and are thus 
not very useful.

On 2017-07-26 10:34, Jörn Huxhorn wrote:
> It would maybe be a good idea to get rid of the Reader method altogether since it mainly introduces an unnecessary point of failure if the Reader is using an encoding other than UTF-8.
> 
> Using a FileReader would be an example for this since its limited c'tors are always using Charset.defaultCharset().name().
> 
> The correct way to create an UTF-8 file reader is the less than obvious "new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common mistake.

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

Whoops... yes, of course. false. m)

I'd use the byte[] method in Lilith.

It would maybe be a good idea to get rid of the Reader method altogether since it mainly introduces an unnecessary point of failure if the Reader is using an encoding other than UTF-8.

Using a FileReader would be an example for this since its limited c'tors are always using Charset.defaultCharset().name().

The correct way to create an UTF-8 file reader is the less than obvious "new InputStreamReader(new FileInputStream(file), StandardCharsets.UTF_8)”. That’s a very common mistake.


On 26. July 2017 at 09:50:21, Mikael Ståldal (mikes@apache.org) wrote:
> I guess you mean:
>  
> this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);  
>  
> Out of curiosity, which of the methods would you use? Reading
> InputStream, Reader, String or byte[] ?
>  
>  
> On 2017-07-25 23:24, Jörn Huxhorn wrote:
> > Hey Mikael,
> > sorry for the slow response time.
> >
> > GMail thought that - of all the dev mailing list mails - this specific mail looked like  
> spam. I just found it hidden in the spam folder...
> >
> > The API looks good to me, assuming that I’ll be fine with the default c’tor versions of  
> Log4jJsonObjectMapper, Log4jYamlObjectMapper and Log4jXmlObjectMapper. Otherwise  
> I'd need matching c’tors in the respective LogEventParser implementations.
> >
> > It may make sense to add this line to the ObjectMapper c'tors:
> > this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true);  
> >
> > This would make parsing less fragile. If new fields are added in future log4j versions  
> then previous versions could still read the event, just without filling the new field.  
> > Another example would be StackTraceElement getting new fields in Java 9, making it  
> possible to still parse Java 9 events in Java 8.
> >
> > Disclaimer: I didn’t test this. I usually annotate specific Mixins with "@JsonIgnoreProperties(ignoreUnknown=true)”  
> where it makes sense. This may be way *too* lenient.
> >
> > It isn’t too critical either and is just meant for your consideration. I’m fine with  
> the parser implementations regardless of such a change.
> >
> > Thanks a lot for providing an official parser API!
> >
> > Cheers,
> > Jörn.
> >
> >
> > On 25. July 2017 at 09:51:56, Mikael Ståldal (mikes@apache.org) wrote:
> >> Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you?
> >>
> >> On 2017-07-22 22:49, Mikael Ståldal wrote:
> >>> Work in progress in Git branch LOG4J2-1986.
> >>>
> >>> On 2017-07-22 22:31, Mikael Ståldal wrote:
> >>>> Yes, I am putting it in its own Java package.
> >>>>
> >>>>
> >>>> On 2017-07-22 22:13, Gary Gregory wrote:
> >>>>> Should this API sit in its own package?
> >>>>>
> >>>>> On Jul 22, 2017 11:41, "Mikael Ståldal" wrote:
> >>>>>
> >>>>>> I'll give it a shot:
> >>>>>>
> >>>>>> https://issues.apache.org/jira/browse/LOG4J2-1986
> >>>>>>
> >>>>>>
> >>>>>> On 2017-07-22 20:28, Ralph Goers wrote:
> >>>>>>
> >>>>>>> Yes, that is a good idea. Probably XML and YAML as well.
> >>>>>>>
> >>>>>>> Ralph
> >>>>>>>
> >>>>>>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote:
> >>>>>>>>
> >>>>>>>> But here we have a concrete use case: a third-party tool, Lilith,
> >>>>>>>> needs
> >>>>>>>> to parse log events from our JsonLayout. (Our SocketServer in
> >>>>>>>> log4j-server
> >>>>>>>> have the same need, and it uses this class.)
> >>>>>>>>
> >>>>>>>> Should we provide a public API in log4j-core for doing so, which
> >>>>>>>> Lilith,
> >>>>>>>> our SocketServer, and possibly others (Chainsaw maybe?) can use?
> >>>>>>>> Maybe such
> >>>>>>>> public API should be a facade around this which hides the fact
> >>>>>>>> that we use
> >>>>>>>> Jackson.
> >>>
> >>
> >>
> >
>  
>  

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

I guess you mean:

this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);

Out of curiosity, which of the methods would you use? Reading 
InputStream, Reader, String or byte[] ?


On 2017-07-25 23:24, Jörn Huxhorn wrote:
> Hey Mikael,
> sorry for the slow response time.
> 
> GMail thought that - of all the dev mailing list mails - this specific mail looked like spam. I just found it hidden in the spam folder...
> 
> The API looks good to me, assuming that I’ll be fine with the default c’tor versions of Log4jJsonObjectMapper, Log4jYamlObjectMapper and Log4jXmlObjectMapper. Otherwise I'd need matching c’tors in the respective LogEventParser implementations.
> 
> It may make sense to add this line to the ObjectMapper c'tors:
> this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true);
> 
> This would make parsing less fragile. If new fields are added in future log4j versions then previous versions could still read the event, just without filling the new field.
> Another example would be StackTraceElement getting new fields in Java 9, making it possible to still parse Java 9 events in Java 8.
> 
> Disclaimer: I didn’t test this. I usually annotate specific Mixins with "@JsonIgnoreProperties(ignoreUnknown=true)” where it makes sense. This may be way *too* lenient.
> 
> It isn’t too critical either and is just meant for your consideration. I’m fine with the parser implementations regardless of such a change.
> 
> Thanks a lot for providing an official parser API!
> 
> Cheers,
> Jörn.
> 
> 
> On 25. July 2017 at 09:51:56, Mikael Ståldal (mikes@apache.org) wrote:
>> Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you?
>>   
>> On 2017-07-22 22:49, Mikael Ståldal wrote:
>>> Work in progress in Git branch LOG4J2-1986.
>>>
>>> On 2017-07-22 22:31, Mikael Ståldal wrote:
>>>> Yes, I am putting it in its own Java package.
>>>>
>>>>
>>>> On 2017-07-22 22:13, Gary Gregory wrote:
>>>>> Should this API sit in its own package?
>>>>>
>>>>> On Jul 22, 2017 11:41, "Mikael Ståldal" wrote:
>>>>>
>>>>>> I'll give it a shot:
>>>>>>
>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1986
>>>>>>
>>>>>>
>>>>>> On 2017-07-22 20:28, Ralph Goers wrote:
>>>>>>
>>>>>>> Yes, that is a good idea. Probably XML and YAML as well.
>>>>>>>
>>>>>>> Ralph
>>>>>>>
>>>>>>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote:
>>>>>>>>
>>>>>>>> But here we have a concrete use case: a third-party tool, Lilith,
>>>>>>>> needs
>>>>>>>> to parse log events from our JsonLayout. (Our SocketServer in
>>>>>>>> log4j-server
>>>>>>>> have the same need, and it uses this class.)
>>>>>>>>
>>>>>>>> Should we provide a public API in log4j-core for doing so, which
>>>>>>>> Lilith,
>>>>>>>> our SocketServer, and possibly others (Chainsaw maybe?) can use?
>>>>>>>> Maybe such
>>>>>>>> public API should be a facade around this which hides the fact
>>>>>>>> that we use
>>>>>>>> Jackson.
>>>
>>   
>>   
> 

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

Hey Mikael,
sorry for the slow response time.

GMail thought that - of all the dev mailing list mails - this specific mail looked like spam. I just found it hidden in the spam folder...

The API looks good to me, assuming that I’ll be fine with the default c’tor versions of Log4jJsonObjectMapper, Log4jYamlObjectMapper and Log4jXmlObjectMapper. Otherwise I'd need matching c’tors in the respective LogEventParser implementations.

It may make sense to add this line to the ObjectMapper c'tors:
this.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true);

This would make parsing less fragile. If new fields are added in future log4j versions then previous versions could still read the event, just without filling the new field.
Another example would be StackTraceElement getting new fields in Java 9, making it possible to still parse Java 9 events in Java 8.

Disclaimer: I didn’t test this. I usually annotate specific Mixins with "@JsonIgnoreProperties(ignoreUnknown=true)” where it makes sense. This may be way *too* lenient.

It isn’t too critical either and is just meant for your consideration. I’m fine with the parser implementations regardless of such a change.

Thanks a lot for providing an official parser API!

Cheers,
Jörn.


On 25. July 2017 at 09:51:56, Mikael Ståldal (mikes@apache.org) wrote:
> Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you?
>  
> On 2017-07-22 22:49, Mikael Ståldal wrote:
> > Work in progress in Git branch LOG4J2-1986.
> >
> > On 2017-07-22 22:31, Mikael Ståldal wrote:
> >> Yes, I am putting it in its own Java package.
> >>
> >>
> >> On 2017-07-22 22:13, Gary Gregory wrote:
> >>> Should this API sit in its own package?
> >>>
> >>> On Jul 22, 2017 11:41, "Mikael Ståldal" wrote:
> >>>
> >>>> I'll give it a shot:
> >>>>
> >>>> https://issues.apache.org/jira/browse/LOG4J2-1986
> >>>>
> >>>>
> >>>> On 2017-07-22 20:28, Ralph Goers wrote:
> >>>>
> >>>>> Yes, that is a good idea. Probably XML and YAML as well.
> >>>>>
> >>>>> Ralph
> >>>>>
> >>>>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal wrote:
> >>>>>>
> >>>>>> But here we have a concrete use case: a third-party tool, Lilith,
> >>>>>> needs
> >>>>>> to parse log events from our JsonLayout. (Our SocketServer in
> >>>>>> log4j-server
> >>>>>> have the same need, and it uses this class.)
> >>>>>>
> >>>>>> Should we provide a public API in log4j-core for doing so, which
> >>>>>> Lilith,
> >>>>>> our SocketServer, and possibly others (Chainsaw maybe?) can use?
> >>>>>> Maybe such
> >>>>>> public API should be a facade around this which hides the fact
> >>>>>> that we use
> >>>>>> Jackson.
> >
>  
>  

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

Jörn Huxhorn, would the API in Git branch LOG4J2-1986 work for you?

On 2017-07-22 22:49, Mikael Ståldal wrote:
> Work in progress in Git branch LOG4J2-1986.
> 
> On 2017-07-22 22:31, Mikael Ståldal wrote:
>> Yes, I am putting it in its own Java package.
>>
>>
>> On 2017-07-22 22:13, Gary Gregory wrote:
>>> Should this API sit in its own package?
>>>
>>> On Jul 22, 2017 11:41, "Mikael Ståldal" <mi...@apache.org> wrote:
>>>
>>>> I'll give it a shot:
>>>>
>>>> https://issues.apache.org/jira/browse/LOG4J2-1986
>>>>
>>>>
>>>> On 2017-07-22 20:28, Ralph Goers wrote:
>>>>
>>>>> Yes, that is a good idea. Probably XML and YAML as well.
>>>>>
>>>>> Ralph
>>>>>
>>>>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal <mi...@apache.org> wrote:
>>>>>>
>>>>>> But here we have a concrete use case: a third-party tool, Lilith, 
>>>>>> needs
>>>>>> to parse log events from our JsonLayout. (Our SocketServer in 
>>>>>> log4j-server
>>>>>> have the same need, and it uses this class.)
>>>>>>
>>>>>> Should we provide a public API in log4j-core for doing so, which 
>>>>>> Lilith,
>>>>>> our SocketServer, and possibly others (Chainsaw maybe?) can use? 
>>>>>> Maybe such
>>>>>> public API should be a facade around this which hides the fact 
>>>>>> that we use
>>>>>> Jackson.
> 

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

Work in progress in Git branch LOG4J2-1986.

On 2017-07-22 22:31, Mikael Ståldal wrote:
> Yes, I am putting it in its own Java package.
> 
> 
> On 2017-07-22 22:13, Gary Gregory wrote:
>> Should this API sit in its own package?
>>
>> On Jul 22, 2017 11:41, "Mikael Ståldal" <mi...@apache.org> wrote:
>>
>>> I'll give it a shot:
>>>
>>> https://issues.apache.org/jira/browse/LOG4J2-1986
>>>
>>>
>>> On 2017-07-22 20:28, Ralph Goers wrote:
>>>
>>>> Yes, that is a good idea. Probably XML and YAML as well.
>>>>
>>>> Ralph
>>>>
>>>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal <mi...@apache.org> wrote:
>>>>>
>>>>> But here we have a concrete use case: a third-party tool, Lilith, 
>>>>> needs
>>>>> to parse log events from our JsonLayout. (Our SocketServer in 
>>>>> log4j-server
>>>>> have the same need, and it uses this class.)
>>>>>
>>>>> Should we provide a public API in log4j-core for doing so, which 
>>>>> Lilith,
>>>>> our SocketServer, and possibly others (Chainsaw maybe?) can use? 
>>>>> Maybe such
>>>>> public API should be a facade around this which hides the fact that 
>>>>> we use
>>>>> Jackson.

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

Yes, I am putting it in its own Java package.


On 2017-07-22 22:13, Gary Gregory wrote:
> Should this API sit in its own package?
> 
> On Jul 22, 2017 11:41, "Mikael Ståldal" <mi...@apache.org> wrote:
> 
>> I'll give it a shot:
>>
>> https://issues.apache.org/jira/browse/LOG4J2-1986
>>
>>
>> On 2017-07-22 20:28, Ralph Goers wrote:
>>
>>> Yes, that is a good idea. Probably XML and YAML as well.
>>>
>>> Ralph
>>>
>>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal <mi...@apache.org> wrote:
>>>>
>>>> But here we have a concrete use case: a third-party tool, Lilith, needs
>>>> to parse log events from our JsonLayout. (Our SocketServer in log4j-server
>>>> have the same need, and it uses this class.)
>>>>
>>>> Should we provide a public API in log4j-core for doing so, which Lilith,
>>>> our SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe such
>>>> public API should be a facade around this which hides the fact that we use
>>>> Jackson.
>>>>
>>>>
>>>> On 2017-07-22 19:04, Gary Gregory wrote:
>>>>
>>>>> Hi All,
>>>>> Just like anything in log4j-core, we try not break binary compatibility
>>>>> when we can but we do not bend over backwards like we do for log4j-api.
>>>>> This class is public probably because Jackson requires it to be,
>>>>> otherwise
>>>>> I would have made it package private. The fact that we use Jackson for
>>>>> JSON
>>>>> IO is an implementation detail IMO, not an extension point. If anyone
>>>>> wants
>>>>> to hard-code a reference to log4j-core's guts then I think its fair to
>>>>> day
>>>>> that this comes with caveats.
>>>>> Gary
>>>>> On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn <jh...@googlemail.com>
>>>>> wrote:
>>>>>
>>>>>> Well… it says
>>>>>>
>>>>>> /**
>>>>>>    * A Jackson JSON {@link ObjectMapper} initialized for Log4j.
>>>>>>    * <p>
>>>>>>    * <em>Consider this class private.</em>
>>>>>>    * </p>
>>>>>>    */
>>>>>>
>>>>>> so I wanted to holler about how to continue.
>>>>>>
>>>>>> I’ll give it a shot.
>>>>>>
>>>>>>
>>>>>> On 22. July 2017 at 15:06:28, Mikael Ståldal (mikes@apache.org) wrote:
>>>>>>
>>>>>>> Would it work to use
>>>>>>> org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
>>>>>>> public?
>>>>>>>
>>>>>>> See here how it is used:
>>>>>>>
>>>>>>> https://github.com/apache/logging-log4j-tools/blob/
>>>>>>>
>>>>>> master/log4j-server/src/main/java/org/apache/logging/log4j/server/
>>>>>> JsonInputStreamLogEventBridge.java
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 2017-07-22 13:47, Jörn Huxhorn wrote:
>>>>>>>
>>>>>>>> It seems like I don’t get access to the log4j2 ObjectMapper or
>>>>>>>> Jackson
>>>>>>>>
>>>>>>> Module used for
>>>>>>
>>>>>>> JSON serialization, i.e. I can’t do something like this:
>>>>>>>
>>>>>>>> new JacksonFactory.JSON(encodeThreadContextAsList,
>>>>>>>> includeStacktrace,
>>>>>>>>
>>>>>>> false)
>>>>>>
>>>>>>>
>>>>>>>> It would make sense to reuse your ObjectMapper/Module in my code
>>>>>>>>
>>>>>>> instead of reimplementing
>>>>>>
>>>>>>> them to prevent future compatibility issues down the road.
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxhorn@googlemail.com)
>>>>>>>>
>>>>>>> wrote:
>>>>>>
>>>>>>> I’ll give this a shot over the next view days.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 19. July 2017 at 21:33:49, Mikael Ståldal (mikes@apache.org)
>>>>>>>>>
>>>>>>>> wrote:
>>>>>>
>>>>>>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
>>>>>>>>>> termination of log events. Will be part of the upcoming 2.9
>>>>>>>>>> release.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
>>>>>>>>>>
>>>>>>>>>>> A general event header and footer would probably be overkill.
>>>>>>>>>>>
>>>>>>>>>>> I’m using the following interface in Lilith:
>>>>>>>>>>>
>>>>>>>>>>> public interface WriteByteStrategy {
>>>>>>>>>>> void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
>>>>>>>>>>> throws IOException;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> One implementation writes an int with the length of bytes before
>>>>>>>>>>>
>>>>>>>>>> “bytes", the other
>>>>>>
>>>>>>> implementation writes a 0-byte after “bytes".
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The problem with that approach is that it requires the
>>>>>>>>>>> intermediate
>>>>>>>>>>>
>>>>>>>>>> “bytes” array
>>>>>>
>>>>>>> and
>>>>>>>>>
>>>>>>>>>> is thus probably incompatible with your zero garbage approach.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I didn’t try end-of-line (\r\n) mainly because I first wanted to
>>>>>>>>>>>
>>>>>>>>>> discuss this with
>>>>>>
>>>>>>> you.
>>>>>>>>>
>>>>>>>>>> I would also have to implement a new receiver type whereas I could
>>>>>>>>>>
>>>>>>>>> just reuse an existing
>>>>>>
>>>>>>> one in case of 0-byte termination.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I’d prefer using 0-byte termination since it would be less fragile
>>>>>>>>>>>
>>>>>>>>>> than end-of-line.
>>>>>>
>>>>>>> The end-of-line approach would break if compact=“false” was set
>>>>>>>>>>
>>>>>>>>> erroneously while
>>>>>>
>>>>>>> you’d still have the option of using either compact=“true” or
>>>>>>>>>>
>>>>>>>>> compact=“false” in
>>>>>>
>>>>>>> case
>>>>>>>
>>>>>>>> of 0-byte termination. 0-byte termination is also slightly more
>>>>>>>>>>
>>>>>>>>> efficient while
>>>>>>
>>>>>>> reading
>>>>>>>
>>>>>>>> since the splitting can operate on bytes instead of chars or Strings
>>>>>>>>>>
>>>>>>>>> - another reason
>>>>>>
>>>>>>> why I didn’t just implement it and instead wanted to discuss this
>>>>>>>>>>
>>>>>>>>> with you first.
>>>>>>
>>>>>>>
>>>>>>>>>>> Joern
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 18. July 2017 at 03:26:32, Ralph Goers (
>>>>>>>>>>>
>>>>>>>>>> ralph.goers@dslextreme.com) wrote:
>>>>>>
>>>>>>> Right now that is all handled by the specific layouts. For
>>>>>>>>>>>>
>>>>>>>>>>> example, by default the
>>>>>>
>>>>>>> RFC5424Layout
>>>>>>>>>
>>>>>>>>>> doesn’t append newlines so when writing to a file they will all be
>>>>>>>>>>>>
>>>>>>>>>>> on the same “line”,
>>>>>>
>>>>>>> but
>>>>>>>>>>
>>>>>>>>>>> it has an option to append one if needed. Doing the same would be
>>>>>>>>>>>>
>>>>>>>>>>> another option for
>>>>>>
>>>>>>> the
>>>>>>>>>
>>>>>>>>>> JSONLayout.
>>>>>>>>>>>>
>>>>>>>>>>>> I’d be ok with event header and footers but ONLY if they have
>>>>>>>>>>>> zero
>>>>>>>>>>>>
>>>>>>>>>>> overhead when not
>>>>>>
>>>>>>> present.
>>>>>>>>>
>>>>>>>>>> IOW, when the Layout is initialized it would have to wire in the
>>>>>>>>>>>>
>>>>>>>>>>> appropriate encode
>>>>>>
>>>>>>> method.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Ralph
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Do we want a general event header and footer then?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Gary
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> No. A Footer is only used at end of file. He needs to know how
>>>>>>>>>>>>>>
>>>>>>>>>>>>> long each
>>>>>>
>>>>>>> event is or when it is the start of a new event.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ralph
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can't you use a footer for any terminator you wish?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Gary
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (Moving this discussion to logging dev mailing list.)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Have you tried to use:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Then each log event will be terminated by end-of-line (\r\n).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think it would be easy to implement 0-byte terminated log
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> events in
>>>>>>
>>>>>>> JsonLayout, and that would make sense since we have
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> implemented support
>>>>>>
>>>>>>> for
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> that in GelfLayout. Created a JIRA issue for it:
>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> As for other tools, the only receivers for Log4j 2
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> SerializedLayout we
>>>>>>
>>>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> currently only
>>>>>>
>>>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> support
>>>>>>
>>>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> JsonLayout
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> by default.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Mikael,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've just taken a look at the JsonLayout and see no way to
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> parse its
>>>>>>
>>>>>>> output event by event.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It's semi-fine for files because then it will result in a
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> well-formed
>>>>>>
>>>>>>> array at the root level that can be read in one big swoop. It
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> isn't
>>>>>>
>>>>>>> really ideal for that use case either since it means that the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> whole
>>>>>>
>>>>>>> array containing all events needs to be read into memory in
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> one piece,
>>>>>>
>>>>>>> causing OOM in case of huge files.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But in Lilith, I really need to process events one by one.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is
>>>>>>
>>>>>>> absolutely crucial.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The only way to achieve this would essentially require me to
>>>>>>>>>>>>>>>>> re-implement a JSON parser that can cope with data like
>>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I implemented a similar kludge to support reading of log4j
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> xml files
>>>>>>
>>>>>>> (see
>>>>>>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
>>>>>>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Log4jImportCallable.java
>>>>>>
>>>>>>> ) but this was way easier since i could simply search for
>>>>>>>>>>>>>>>>> "" to find the position at which I could split the stream
>>>>>>>>>>>>>>>>> of events into a valid XML document for every single event .
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This was
>>>>>>
>>>>>>> still kind of nasty but doable, especially since processing an
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> "offline"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> XML file doesn't have the same performance restriction as
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> handling of
>>>>>>
>>>>>>> live events.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I wouldn't have the luxury of such a unique split signal in
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> JSON,
>>>>>>
>>>>>>> though.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I would need to detect the "}" at the level of the root
>>>>>>>>>>>>>>>>> array
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> which
>>>>>>
>>>>>>> would not simply involve counting of opening and closing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> brackets but
>>>>>>
>>>>>>> also ignoring of brackets inside of strings. This would boil
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> down to
>>>>>>
>>>>>>> implementing a custom JSON reader and I won't be doing this.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This isn't just me being lazy either. Such an implementation
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wouldn't
>>>>>>
>>>>>>> be
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> resource (CPU/memory) friendly anyway.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In my own JSON receiver using my own JSON format there are
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> two ways to
>>>>>>
>>>>>>> send those events:
>>>>>>>>>>>>>>>>> - write an int containing the amount of bytes representing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the event,
>>>>>>
>>>>>>> then the bytes of the event. This type of events also supports
>>>>>>>>>>>>>>>>> compression.
>>>>>>>>>>>>>>>>> - write the bytes of the event, followed by a 0-byte. This
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> works fine
>>>>>>
>>>>>>> and JavaScript/ActionScript is able to produce events like
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> that while
>>>>>>
>>>>>>> they are unable (or were unable 7 years ago) to count the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> bytes on
>>>>>>
>>>>>>> their
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> own. This type of events only supports plain text like JSON
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> or XML and
>>>>>>
>>>>>>> no compression since compressed events could contain the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 0-byte while
>>>>>>
>>>>>>> XML and JSON both won't allow it.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Both are essentially "binary" formats because of either the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> raw ints or
>>>>>>
>>>>>>> the 0-bytes.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm not sure why you deprecate SerializedLayout since the
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> security
>>>>>>
>>>>>>> issues arise only while deserializing the events, not while
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> serializing
>>>>>>
>>>>>>> them. Is this just meant to educate the user about the issue?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I fixed the remote code execution exploit by implementing
>>>>>>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
>>>>>>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
>>>>>>>>>>>>>>>>> er/WhitelistObjectInputStream.java
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM
>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> case of
>>>>>>
>>>>>>> malicious data) but is close enough for me, mainly because
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> fixing the
>>>>>>
>>>>>>> DoS issues would really have to be handled by Java, not by
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> user code.
>>>>>>
>>>>>>> Manually re-implementing deserialization would likely be
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> fragile and
>>>>>>
>>>>>>> error prone, possibly even introducing other security issues
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> on its
>>>>>>
>>>>>>> own.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'd be interested how other tools are tackling this issue.
>>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Chainsaw
>>>>>>
>>>>>>> officially dead or are they implementing receivers for event
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> streams
>>>>>>
>>>>>>> like this?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm totally fine with implementing a new receiver as a
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> replacement for
>>>>>>
>>>>>>> log4j2 SerializedLayout (even though I won't be able to get
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> rid of the
>>>>>>
>>>>>>> Serializable receiver anyway) but the current JsonLayout just
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> isn't up
>>>>>>
>>>>>>> to the job at the moment.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Feel free to share this mail with your fellow developers.
>>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> open for
>>>>>>
>>>>>>> suggestions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I hope this doesn't come across in a harsh way. It isn't
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> meant to be.
>>>>>>
>>>>>>>
>>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>>> Jörn.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>>
>>
> 

Re: Lilith and Log4j 2

[Gary Gregory <ga...@gmail.com>]

Should this API sit in its own package?

On Jul 22, 2017 11:41, "Mikael Ståldal" <mi...@apache.org> wrote:

> I'll give it a shot:
>
> https://issues.apache.org/jira/browse/LOG4J2-1986
>
>
> On 2017-07-22 20:28, Ralph Goers wrote:
>
>> Yes, that is a good idea. Probably XML and YAML as well.
>>
>> Ralph
>>
>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal <mi...@apache.org> wrote:
>>>
>>> But here we have a concrete use case: a third-party tool, Lilith, needs
>>> to parse log events from our JsonLayout. (Our SocketServer in log4j-server
>>> have the same need, and it uses this class.)
>>>
>>> Should we provide a public API in log4j-core for doing so, which Lilith,
>>> our SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe such
>>> public API should be a facade around this which hides the fact that we use
>>> Jackson.
>>>
>>>
>>> On 2017-07-22 19:04, Gary Gregory wrote:
>>>
>>>> Hi All,
>>>> Just like anything in log4j-core, we try not break binary compatibility
>>>> when we can but we do not bend over backwards like we do for log4j-api.
>>>> This class is public probably because Jackson requires it to be,
>>>> otherwise
>>>> I would have made it package private. The fact that we use Jackson for
>>>> JSON
>>>> IO is an implementation detail IMO, not an extension point. If anyone
>>>> wants
>>>> to hard-code a reference to log4j-core's guts then I think its fair to
>>>> day
>>>> that this comes with caveats.
>>>> Gary
>>>> On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn <jh...@googlemail.com>
>>>> wrote:
>>>>
>>>>> Well… it says
>>>>>
>>>>> /**
>>>>>   * A Jackson JSON {@link ObjectMapper} initialized for Log4j.
>>>>>   * <p>
>>>>>   * <em>Consider this class private.</em>
>>>>>   * </p>
>>>>>   */
>>>>>
>>>>> so I wanted to holler about how to continue.
>>>>>
>>>>> I’ll give it a shot.
>>>>>
>>>>>
>>>>> On 22. July 2017 at 15:06:28, Mikael Ståldal (mikes@apache.org) wrote:
>>>>>
>>>>>> Would it work to use
>>>>>> org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
>>>>>> public?
>>>>>>
>>>>>> See here how it is used:
>>>>>>
>>>>>> https://github.com/apache/logging-log4j-tools/blob/
>>>>>>
>>>>> master/log4j-server/src/main/java/org/apache/logging/log4j/server/
>>>>> JsonInputStreamLogEventBridge.java
>>>>>
>>>>>>
>>>>>>
>>>>>> On 2017-07-22 13:47, Jörn Huxhorn wrote:
>>>>>>
>>>>>>> It seems like I don’t get access to the log4j2 ObjectMapper or
>>>>>>> Jackson
>>>>>>>
>>>>>> Module used for
>>>>>
>>>>>> JSON serialization, i.e. I can’t do something like this:
>>>>>>
>>>>>>> new JacksonFactory.JSON(encodeThreadContextAsList,
>>>>>>> includeStacktrace,
>>>>>>>
>>>>>> false)
>>>>>
>>>>>>
>>>>>>> It would make sense to reuse your ObjectMapper/Module in my code
>>>>>>>
>>>>>> instead of reimplementing
>>>>>
>>>>>> them to prevent future compatibility issues down the road.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxhorn@googlemail.com)
>>>>>>>
>>>>>> wrote:
>>>>>
>>>>>> I’ll give this a shot over the next view days.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>>
>>>>>>>> On 19. July 2017 at 21:33:49, Mikael Ståldal (mikes@apache.org)
>>>>>>>>
>>>>>>> wrote:
>>>>>
>>>>>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
>>>>>>>>> termination of log events. Will be part of the upcoming 2.9
>>>>>>>>> release.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
>>>>>>>>>
>>>>>>>>>> A general event header and footer would probably be overkill.
>>>>>>>>>>
>>>>>>>>>> I’m using the following interface in Lilith:
>>>>>>>>>>
>>>>>>>>>> public interface WriteByteStrategy {
>>>>>>>>>> void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
>>>>>>>>>> throws IOException;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> One implementation writes an int with the length of bytes before
>>>>>>>>>>
>>>>>>>>> “bytes", the other
>>>>>
>>>>>> implementation writes a 0-byte after “bytes".
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The problem with that approach is that it requires the
>>>>>>>>>> intermediate
>>>>>>>>>>
>>>>>>>>> “bytes” array
>>>>>
>>>>>> and
>>>>>>>>
>>>>>>>>> is thus probably incompatible with your zero garbage approach.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I didn’t try end-of-line (\r\n) mainly because I first wanted to
>>>>>>>>>>
>>>>>>>>> discuss this with
>>>>>
>>>>>> you.
>>>>>>>>
>>>>>>>>> I would also have to implement a new receiver type whereas I could
>>>>>>>>>
>>>>>>>> just reuse an existing
>>>>>
>>>>>> one in case of 0-byte termination.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I’d prefer using 0-byte termination since it would be less fragile
>>>>>>>>>>
>>>>>>>>> than end-of-line.
>>>>>
>>>>>> The end-of-line approach would break if compact=“false” was set
>>>>>>>>>
>>>>>>>> erroneously while
>>>>>
>>>>>> you’d still have the option of using either compact=“true” or
>>>>>>>>>
>>>>>>>> compact=“false” in
>>>>>
>>>>>> case
>>>>>>
>>>>>>> of 0-byte termination. 0-byte termination is also slightly more
>>>>>>>>>
>>>>>>>> efficient while
>>>>>
>>>>>> reading
>>>>>>
>>>>>>> since the splitting can operate on bytes instead of chars or Strings
>>>>>>>>>
>>>>>>>> - another reason
>>>>>
>>>>>> why I didn’t just implement it and instead wanted to discuss this
>>>>>>>>>
>>>>>>>> with you first.
>>>>>
>>>>>>
>>>>>>>>>> Joern
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 18. July 2017 at 03:26:32, Ralph Goers (
>>>>>>>>>>
>>>>>>>>> ralph.goers@dslextreme.com) wrote:
>>>>>
>>>>>> Right now that is all handled by the specific layouts. For
>>>>>>>>>>>
>>>>>>>>>> example, by default the
>>>>>
>>>>>> RFC5424Layout
>>>>>>>>
>>>>>>>>> doesn’t append newlines so when writing to a file they will all be
>>>>>>>>>>>
>>>>>>>>>> on the same “line”,
>>>>>
>>>>>> but
>>>>>>>>>
>>>>>>>>>> it has an option to append one if needed. Doing the same would be
>>>>>>>>>>>
>>>>>>>>>> another option for
>>>>>
>>>>>> the
>>>>>>>>
>>>>>>>>> JSONLayout.
>>>>>>>>>>>
>>>>>>>>>>> I’d be ok with event header and footers but ONLY if they have
>>>>>>>>>>> zero
>>>>>>>>>>>
>>>>>>>>>> overhead when not
>>>>>
>>>>>> present.
>>>>>>>>
>>>>>>>>> IOW, when the Layout is initialized it would have to wire in the
>>>>>>>>>>>
>>>>>>>>>> appropriate encode
>>>>>
>>>>>> method.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Ralph
>>>>>>>>>>>
>>>>>>>>>>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Do we want a general event header and footer then?
>>>>>>>>>>>>
>>>>>>>>>>>> Gary
>>>>>>>>>>>>
>>>>>>>>>>>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> No. A Footer is only used at end of file. He needs to know how
>>>>>>>>>>>>>
>>>>>>>>>>>> long each
>>>>>
>>>>>> event is or when it is the start of a new event.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ralph
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
>>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can't you use a footer for any terminator you wish?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Gary
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
>>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (Moving this discussion to logging dev mailing list.)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Have you tried to use:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Then each log event will be terminated by end-of-line (\r\n).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think it would be easy to implement 0-byte terminated log
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> events in
>>>>>
>>>>>> JsonLayout, and that would make sense since we have
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> implemented support
>>>>>
>>>>>> for
>>>>>>>>>>>>>
>>>>>>>>>>>>>> that in GelfLayout. Created a JIRA issue for it:
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> As for other tools, the only receivers for Log4j 2
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> SerializedLayout we
>>>>>
>>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> currently only
>>>>>
>>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> support
>>>>>
>>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> JsonLayout
>>>>>>>>>>>>>
>>>>>>>>>>>>>> by default.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Mikael,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've just taken a look at the JsonLayout and see no way to
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> parse its
>>>>>
>>>>>> output event by event.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> It's semi-fine for files because then it will result in a
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> well-formed
>>>>>
>>>>>> array at the root level that can be read in one big swoop. It
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> isn't
>>>>>
>>>>>> really ideal for that use case either since it means that the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> whole
>>>>>
>>>>>> array containing all events needs to be read into memory in
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> one piece,
>>>>>
>>>>>> causing OOM in case of huge files.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But in Lilith, I really need to process events one by one.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is
>>>>>
>>>>>> absolutely crucial.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The only way to achieve this would essentially require me to
>>>>>>>>>>>>>>>> re-implement a JSON parser that can cope with data like
>>>>>>>>>>>>>>>> this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I implemented a similar kludge to support reading of log4j
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> xml files
>>>>>
>>>>>> (see
>>>>>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
>>>>>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Log4jImportCallable.java
>>>>>
>>>>>> ) but this was way easier since i could simply search for
>>>>>>>>>>>>>>>> "" to find the position at which I could split the stream
>>>>>>>>>>>>>>>> of events into a valid XML document for every single event .
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This was
>>>>>
>>>>>> still kind of nasty but doable, especially since processing an
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "offline"
>>>>>>>>>>>>>
>>>>>>>>>>>>>> XML file doesn't have the same performance restriction as
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> handling of
>>>>>
>>>>>> live events.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I wouldn't have the luxury of such a unique split signal in
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> JSON,
>>>>>
>>>>>> though.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I would need to detect the "}" at the level of the root
>>>>>>>>>>>>>>>> array
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> which
>>>>>
>>>>>> would not simply involve counting of opening and closing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> brackets but
>>>>>
>>>>>> also ignoring of brackets inside of strings. This would boil
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> down to
>>>>>
>>>>>> implementing a custom JSON reader and I won't be doing this.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This isn't just me being lazy either. Such an implementation
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wouldn't
>>>>>
>>>>>> be
>>>>>>>>>>>>>
>>>>>>>>>>>>>> resource (CPU/memory) friendly anyway.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In my own JSON receiver using my own JSON format there are
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> two ways to
>>>>>
>>>>>> send those events:
>>>>>>>>>>>>>>>> - write an int containing the amount of bytes representing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the event,
>>>>>
>>>>>> then the bytes of the event. This type of events also supports
>>>>>>>>>>>>>>>> compression.
>>>>>>>>>>>>>>>> - write the bytes of the event, followed by a 0-byte. This
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> works fine
>>>>>
>>>>>> and JavaScript/ActionScript is able to produce events like
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> that while
>>>>>
>>>>>> they are unable (or were unable 7 years ago) to count the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> bytes on
>>>>>
>>>>>> their
>>>>>>>>>>>>>
>>>>>>>>>>>>>> own. This type of events only supports plain text like JSON
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> or XML and
>>>>>
>>>>>> no compression since compressed events could contain the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 0-byte while
>>>>>
>>>>>> XML and JSON both won't allow it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Both are essentially "binary" formats because of either the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> raw ints or
>>>>>
>>>>>> the 0-bytes.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm not sure why you deprecate SerializedLayout since the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> security
>>>>>
>>>>>> issues arise only while deserializing the events, not while
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> serializing
>>>>>
>>>>>> them. Is this just meant to educate the user about the issue?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I fixed the remote code execution exploit by implementing
>>>>>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
>>>>>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
>>>>>>>>>>>>>>>> er/WhitelistObjectInputStream.java
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM
>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> case of
>>>>>
>>>>>> malicious data) but is close enough for me, mainly because
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> fixing the
>>>>>
>>>>>> DoS issues would really have to be handled by Java, not by
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> user code.
>>>>>
>>>>>> Manually re-implementing deserialization would likely be
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> fragile and
>>>>>
>>>>>> error prone, possibly even introducing other security issues
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> on its
>>>>>
>>>>>> own.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'd be interested how other tools are tackling this issue.
>>>>>>>>>>>>>>>> Is
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Chainsaw
>>>>>
>>>>>> officially dead or are they implementing receivers for event
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> streams
>>>>>
>>>>>> like this?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm totally fine with implementing a new receiver as a
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> replacement for
>>>>>
>>>>>> log4j2 SerializedLayout (even though I won't be able to get
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> rid of the
>>>>>
>>>>>> Serializable receiver anyway) but the current JsonLayout just
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> isn't up
>>>>>
>>>>>> to the job at the moment.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Feel free to share this mail with your fellow developers.
>>>>>>>>>>>>>>>> I'm
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> open for
>>>>>
>>>>>> suggestions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I hope this doesn't come across in a harsh way. It isn't
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> meant to be.
>>>>>
>>>>>>
>>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>>> Jörn.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>>
>

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

I'll give it a shot:

https://issues.apache.org/jira/browse/LOG4J2-1986


On 2017-07-22 20:28, Ralph Goers wrote:
> Yes, that is a good idea. Probably XML and YAML as well.
> 
> Ralph
> 
>> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal <mi...@apache.org> wrote:
>>
>> But here we have a concrete use case: a third-party tool, Lilith, needs to parse log events from our JsonLayout. (Our SocketServer in log4j-server have the same need, and it uses this class.)
>>
>> Should we provide a public API in log4j-core for doing so, which Lilith, our SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe such public API should be a facade around this which hides the fact that we use Jackson.
>>
>>
>> On 2017-07-22 19:04, Gary Gregory wrote:
>>> Hi All,
>>> Just like anything in log4j-core, we try not break binary compatibility
>>> when we can but we do not bend over backwards like we do for log4j-api.
>>> This class is public probably because Jackson requires it to be, otherwise
>>> I would have made it package private. The fact that we use Jackson for JSON
>>> IO is an implementation detail IMO, not an extension point. If anyone wants
>>> to hard-code a reference to log4j-core's guts then I think its fair to day
>>> that this comes with caveats.
>>> Gary
>>> On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn <jh...@googlemail.com>
>>> wrote:
>>>> Well… it says
>>>>
>>>> /**
>>>>   * A Jackson JSON {@link ObjectMapper} initialized for Log4j.
>>>>   * <p>
>>>>   * <em>Consider this class private.</em>
>>>>   * </p>
>>>>   */
>>>>
>>>> so I wanted to holler about how to continue.
>>>>
>>>> I’ll give it a shot.
>>>>
>>>>
>>>> On 22. July 2017 at 15:06:28, Mikael Ståldal (mikes@apache.org) wrote:
>>>>> Would it work to use
>>>>> org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
>>>>> public?
>>>>>
>>>>> See here how it is used:
>>>>>
>>>>> https://github.com/apache/logging-log4j-tools/blob/
>>>> master/log4j-server/src/main/java/org/apache/logging/log4j/server/
>>>> JsonInputStreamLogEventBridge.java
>>>>>
>>>>>
>>>>> On 2017-07-22 13:47, Jörn Huxhorn wrote:
>>>>>> It seems like I don’t get access to the log4j2 ObjectMapper or Jackson
>>>> Module used for
>>>>> JSON serialization, i.e. I can’t do something like this:
>>>>>> new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace,
>>>> false)
>>>>>>
>>>>>> It would make sense to reuse your ObjectMapper/Module in my code
>>>> instead of reimplementing
>>>>> them to prevent future compatibility issues down the road.
>>>>>>
>>>>>>
>>>>>> On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxhorn@googlemail.com)
>>>> wrote:
>>>>>>> I’ll give this a shot over the next view days.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>> On 19. July 2017 at 21:33:49, Mikael Ståldal (mikes@apache.org)
>>>> wrote:
>>>>>>>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
>>>>>>>> termination of log events. Will be part of the upcoming 2.9 release.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
>>>>>>>>> A general event header and footer would probably be overkill.
>>>>>>>>>
>>>>>>>>> I’m using the following interface in Lilith:
>>>>>>>>>
>>>>>>>>> public interface WriteByteStrategy {
>>>>>>>>> void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
>>>>>>>>> throws IOException;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> One implementation writes an int with the length of bytes before
>>>> “bytes", the other
>>>>>>>> implementation writes a 0-byte after “bytes".
>>>>>>>>>
>>>>>>>>> The problem with that approach is that it requires the intermediate
>>>> “bytes” array
>>>>>>> and
>>>>>>>> is thus probably incompatible with your zero garbage approach.
>>>>>>>>>
>>>>>>>>> I didn’t try end-of-line (\r\n) mainly because I first wanted to
>>>> discuss this with
>>>>>>> you.
>>>>>>>> I would also have to implement a new receiver type whereas I could
>>>> just reuse an existing
>>>>>>>> one in case of 0-byte termination.
>>>>>>>>>
>>>>>>>>> I’d prefer using 0-byte termination since it would be less fragile
>>>> than end-of-line.
>>>>>>>> The end-of-line approach would break if compact=“false” was set
>>>> erroneously while
>>>>>>>> you’d still have the option of using either compact=“true” or
>>>> compact=“false” in
>>>>> case
>>>>>>>> of 0-byte termination. 0-byte termination is also slightly more
>>>> efficient while
>>>>> reading
>>>>>>>> since the splitting can operate on bytes instead of chars or Strings
>>>> - another reason
>>>>>>>> why I didn’t just implement it and instead wanted to discuss this
>>>> with you first.
>>>>>>>>>
>>>>>>>>> Joern
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 18. July 2017 at 03:26:32, Ralph Goers (
>>>> ralph.goers@dslextreme.com) wrote:
>>>>>>>>>> Right now that is all handled by the specific layouts. For
>>>> example, by default the
>>>>>>> RFC5424Layout
>>>>>>>>>> doesn’t append newlines so when writing to a file they will all be
>>>> on the same “line”,
>>>>>>>> but
>>>>>>>>>> it has an option to append one if needed. Doing the same would be
>>>> another option for
>>>>>>> the
>>>>>>>>>> JSONLayout.
>>>>>>>>>>
>>>>>>>>>> I’d be ok with event header and footers but ONLY if they have zero
>>>> overhead when not
>>>>>>> present.
>>>>>>>>>> IOW, when the Layout is initialized it would have to wire in the
>>>> appropriate encode
>>>>>>>> method.
>>>>>>>>>>
>>>>>>>>>> Ralph
>>>>>>>>>>
>>>>>>>>>>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
>>>>>>>>>>>
>>>>>>>>>>> Do we want a general event header and footer then?
>>>>>>>>>>>
>>>>>>>>>>> Gary
>>>>>>>>>>>
>>>>>>>>>>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
>>>>>>>>>>>
>>>>>>>>>>>> No. A Footer is only used at end of file. He needs to know how
>>>> long each
>>>>>>>>>>>> event is or when it is the start of a new event.
>>>>>>>>>>>>
>>>>>>>>>>>> Ralph
>>>>>>>>>>>>
>>>>>>>>>>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can't you use a footer for any terminator you wish?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Gary
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (Moving this discussion to logging dev mailing list.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have you tried to use:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Then each log event will be terminated by end-of-line (\r\n).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I think it would be easy to implement 0-byte terminated log
>>>> events in
>>>>>>>>>>>>>> JsonLayout, and that would make sense since we have
>>>> implemented support
>>>>>>>>>>>> for
>>>>>>>>>>>>>> that in GelfLayout. Created a JIRA issue for it:
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> As for other tools, the only receivers for Log4j 2
>>>> SerializedLayout we
>>>>>>>>>>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw
>>>> currently only
>>>>>>>>>>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer
>>>> support
>>>>>>>>>>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
>>>>>>>>>>>> JsonLayout
>>>>>>>>>>>>>> by default.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Mikael,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've just taken a look at the JsonLayout and see no way to
>>>> parse its
>>>>>>>>>>>>>>> output event by event.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It's semi-fine for files because then it will result in a
>>>> well-formed
>>>>>>>>>>>>>>> array at the root level that can be read in one big swoop. It
>>>> isn't
>>>>>>>>>>>>>>> really ideal for that use case either since it means that the
>>>> whole
>>>>>>>>>>>>>>> array containing all events needs to be read into memory in
>>>> one piece,
>>>>>>>>>>>>>>> causing OOM in case of huge files.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But in Lilith, I really need to process events one by one.
>>>> This is
>>>>>>>>>>>>>>> absolutely crucial.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The only way to achieve this would essentially require me to
>>>>>>>>>>>>>>> re-implement a JSON parser that can cope with data like this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I implemented a similar kludge to support reading of log4j
>>>> xml files
>>>>>>>>>>>>>>> (see
>>>>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
>>>>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/
>>>> Log4jImportCallable.java
>>>>>>>>>>>>>>> ) but this was way easier since i could simply search for
>>>>>>>>>>>>>>> "" to find the position at which I could split the stream
>>>>>>>>>>>>>>> of events into a valid XML document for every single event .
>>>> This was
>>>>>>>>>>>>>>> still kind of nasty but doable, especially since processing an
>>>>>>>>>>>> "offline"
>>>>>>>>>>>>>>> XML file doesn't have the same performance restriction as
>>>> handling of
>>>>>>>>>>>>>>> live events.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I wouldn't have the luxury of such a unique split signal in
>>>> JSON,
>>>>>>>>>>>> though.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I would need to detect the "}" at the level of the root array
>>>> which
>>>>>>>>>>>>>>> would not simply involve counting of opening and closing
>>>> brackets but
>>>>>>>>>>>>>>> also ignoring of brackets inside of strings. This would boil
>>>> down to
>>>>>>>>>>>>>>> implementing a custom JSON reader and I won't be doing this.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This isn't just me being lazy either. Such an implementation
>>>> wouldn't
>>>>>>>>>>>> be
>>>>>>>>>>>>>>> resource (CPU/memory) friendly anyway.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In my own JSON receiver using my own JSON format there are
>>>> two ways to
>>>>>>>>>>>>>>> send those events:
>>>>>>>>>>>>>>> - write an int containing the amount of bytes representing
>>>> the event,
>>>>>>>>>>>>>>> then the bytes of the event. This type of events also supports
>>>>>>>>>>>>>>> compression.
>>>>>>>>>>>>>>> - write the bytes of the event, followed by a 0-byte. This
>>>> works fine
>>>>>>>>>>>>>>> and JavaScript/ActionScript is able to produce events like
>>>> that while
>>>>>>>>>>>>>>> they are unable (or were unable 7 years ago) to count the
>>>> bytes on
>>>>>>>>>>>> their
>>>>>>>>>>>>>>> own. This type of events only supports plain text like JSON
>>>> or XML and
>>>>>>>>>>>>>>> no compression since compressed events could contain the
>>>> 0-byte while
>>>>>>>>>>>>>>> XML and JSON both won't allow it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Both are essentially "binary" formats because of either the
>>>> raw ints or
>>>>>>>>>>>>>>> the 0-bytes.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm not sure why you deprecate SerializedLayout since the
>>>> security
>>>>>>>>>>>>>>> issues arise only while deserializing the events, not while
>>>> serializing
>>>>>>>>>>>>>>> them. Is this just meant to educate the user about the issue?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I fixed the remote code execution exploit by implementing
>>>>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
>>>>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
>>>>>>>>>>>>>>> er/WhitelistObjectInputStream.java
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM in
>>>> case of
>>>>>>>>>>>>>>> malicious data) but is close enough for me, mainly because
>>>> fixing the
>>>>>>>>>>>>>>> DoS issues would really have to be handled by Java, not by
>>>> user code.
>>>>>>>>>>>>>>> Manually re-implementing deserialization would likely be
>>>> fragile and
>>>>>>>>>>>>>>> error prone, possibly even introducing other security issues
>>>> on its
>>>>>>>>>>>> own.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'd be interested how other tools are tackling this issue. Is
>>>> Chainsaw
>>>>>>>>>>>>>>> officially dead or are they implementing receivers for event
>>>> streams
>>>>>>>>>>>>>>> like this?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm totally fine with implementing a new receiver as a
>>>> replacement for
>>>>>>>>>>>>>>> log4j2 SerializedLayout (even though I won't be able to get
>>>> rid of the
>>>>>>>>>>>>>>> Serializable receiver anyway) but the current JsonLayout just
>>>> isn't up
>>>>>>>>>>>>>>> to the job at the moment.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Feel free to share this mail with your fellow developers. I'm
>>>> open for
>>>>>>>>>>>>>>> suggestions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I hope this doesn't come across in a harsh way. It isn't
>>>> meant to be.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Jörn.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>
>>
> 
> 

Re: Lilith and Log4j 2

[Ralph Goers <ra...@dslextreme.com>]

Yes, that is a good idea. Probably XML and YAML as well.

Ralph

> On Jul 22, 2017, at 11:21 AM, Mikael Ståldal <mi...@apache.org> wrote:
> 
> But here we have a concrete use case: a third-party tool, Lilith, needs to parse log events from our JsonLayout. (Our SocketServer in log4j-server have the same need, and it uses this class.)
> 
> Should we provide a public API in log4j-core for doing so, which Lilith, our SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe such public API should be a facade around this which hides the fact that we use Jackson.
> 
> 
> On 2017-07-22 19:04, Gary Gregory wrote:
>> Hi All,
>> Just like anything in log4j-core, we try not break binary compatibility
>> when we can but we do not bend over backwards like we do for log4j-api.
>> This class is public probably because Jackson requires it to be, otherwise
>> I would have made it package private. The fact that we use Jackson for JSON
>> IO is an implementation detail IMO, not an extension point. If anyone wants
>> to hard-code a reference to log4j-core's guts then I think its fair to day
>> that this comes with caveats.
>> Gary
>> On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn <jh...@googlemail.com>
>> wrote:
>>> Well… it says
>>> 
>>> /**
>>>  * A Jackson JSON {@link ObjectMapper} initialized for Log4j.
>>>  * <p>
>>>  * <em>Consider this class private.</em>
>>>  * </p>
>>>  */
>>> 
>>> so I wanted to holler about how to continue.
>>> 
>>> I’ll give it a shot.
>>> 
>>> 
>>> On 22. July 2017 at 15:06:28, Mikael Ståldal (mikes@apache.org) wrote:
>>>> Would it work to use
>>>> org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
>>>> public?
>>>> 
>>>> See here how it is used:
>>>> 
>>>> https://github.com/apache/logging-log4j-tools/blob/
>>> master/log4j-server/src/main/java/org/apache/logging/log4j/server/
>>> JsonInputStreamLogEventBridge.java
>>>> 
>>>> 
>>>> On 2017-07-22 13:47, Jörn Huxhorn wrote:
>>>>> It seems like I don’t get access to the log4j2 ObjectMapper or Jackson
>>> Module used for
>>>> JSON serialization, i.e. I can’t do something like this:
>>>>> new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace,
>>> false)
>>>>> 
>>>>> It would make sense to reuse your ObjectMapper/Module in my code
>>> instead of reimplementing
>>>> them to prevent future compatibility issues down the road.
>>>>> 
>>>>> 
>>>>> On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxhorn@googlemail.com)
>>> wrote:
>>>>>> I’ll give this a shot over the next view days.
>>>>>> 
>>>>>> Thanks!
>>>>>> 
>>>>>> 
>>>>>> On 19. July 2017 at 21:33:49, Mikael Ståldal (mikes@apache.org)
>>> wrote:
>>>>>>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
>>>>>>> termination of log events. Will be part of the upcoming 2.9 release.
>>>>>>> 
>>>>>>> 
>>>>>>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
>>>>>>>> A general event header and footer would probably be overkill.
>>>>>>>> 
>>>>>>>> I’m using the following interface in Lilith:
>>>>>>>> 
>>>>>>>> public interface WriteByteStrategy {
>>>>>>>> void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
>>>>>>>> throws IOException;
>>>>>>>> }
>>>>>>>> 
>>>>>>>> One implementation writes an int with the length of bytes before
>>> “bytes", the other
>>>>>>> implementation writes a 0-byte after “bytes".
>>>>>>>> 
>>>>>>>> The problem with that approach is that it requires the intermediate
>>> “bytes” array
>>>>>> and
>>>>>>> is thus probably incompatible with your zero garbage approach.
>>>>>>>> 
>>>>>>>> I didn’t try end-of-line (\r\n) mainly because I first wanted to
>>> discuss this with
>>>>>> you.
>>>>>>> I would also have to implement a new receiver type whereas I could
>>> just reuse an existing
>>>>>>> one in case of 0-byte termination.
>>>>>>>> 
>>>>>>>> I’d prefer using 0-byte termination since it would be less fragile
>>> than end-of-line.
>>>>>>> The end-of-line approach would break if compact=“false” was set
>>> erroneously while
>>>>>>> you’d still have the option of using either compact=“true” or
>>> compact=“false” in
>>>> case
>>>>>>> of 0-byte termination. 0-byte termination is also slightly more
>>> efficient while
>>>> reading
>>>>>>> since the splitting can operate on bytes instead of chars or Strings
>>> - another reason
>>>>>>> why I didn’t just implement it and instead wanted to discuss this
>>> with you first.
>>>>>>>> 
>>>>>>>> Joern
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 18. July 2017 at 03:26:32, Ralph Goers (
>>> ralph.goers@dslextreme.com) wrote:
>>>>>>>>> Right now that is all handled by the specific layouts. For
>>> example, by default the
>>>>>> RFC5424Layout
>>>>>>>>> doesn’t append newlines so when writing to a file they will all be
>>> on the same “line”,
>>>>>>> but
>>>>>>>>> it has an option to append one if needed. Doing the same would be
>>> another option for
>>>>>> the
>>>>>>>>> JSONLayout.
>>>>>>>>> 
>>>>>>>>> I’d be ok with event header and footers but ONLY if they have zero
>>> overhead when not
>>>>>> present.
>>>>>>>>> IOW, when the Layout is initialized it would have to wire in the
>>> appropriate encode
>>>>>>> method.
>>>>>>>>> 
>>>>>>>>> Ralph
>>>>>>>>> 
>>>>>>>>>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
>>>>>>>>>> 
>>>>>>>>>> Do we want a general event header and footer then?
>>>>>>>>>> 
>>>>>>>>>> Gary
>>>>>>>>>> 
>>>>>>>>>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
>>>>>>>>>> 
>>>>>>>>>>> No. A Footer is only used at end of file. He needs to know how
>>> long each
>>>>>>>>>>> event is or when it is the start of a new event.
>>>>>>>>>>> 
>>>>>>>>>>> Ralph
>>>>>>>>>>> 
>>>>>>>>>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Can't you use a footer for any terminator you wish?
>>>>>>>>>>>> 
>>>>>>>>>>>> Gary
>>>>>>>>>>>> 
>>>>>>>>>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
>>>>>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> Hi.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> (Moving this discussion to logging dev mailing list.)
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Have you tried to use:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Then each log event will be terminated by end-of-line (\r\n).
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I think it would be easy to implement 0-byte terminated log
>>> events in
>>>>>>>>>>>>> JsonLayout, and that would make sense since we have
>>> implemented support
>>>>>>>>>>> for
>>>>>>>>>>>>> that in GelfLayout. Created a JIRA issue for it:
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
>>>>>>>>>>>>> 
>>>>>>>>>>>>> As for other tools, the only receivers for Log4j 2
>>> SerializedLayout we
>>>>>>>>>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw
>>> currently only
>>>>>>>>>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer
>>> support
>>>>>>>>>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
>>>>>>>>>>> JsonLayout
>>>>>>>>>>>>> by default.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Hi Mikael,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I've just taken a look at the JsonLayout and see no way to
>>> parse its
>>>>>>>>>>>>>> output event by event.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> It's semi-fine for files because then it will result in a
>>> well-formed
>>>>>>>>>>>>>> array at the root level that can be read in one big swoop. It
>>> isn't
>>>>>>>>>>>>>> really ideal for that use case either since it means that the
>>> whole
>>>>>>>>>>>>>> array containing all events needs to be read into memory in
>>> one piece,
>>>>>>>>>>>>>> causing OOM in case of huge files.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> But in Lilith, I really need to process events one by one.
>>> This is
>>>>>>>>>>>>>> absolutely crucial.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> The only way to achieve this would essentially require me to
>>>>>>>>>>>>>> re-implement a JSON parser that can cope with data like this.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I implemented a similar kludge to support reading of log4j
>>> xml files
>>>>>>>>>>>>>> (see
>>>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
>>>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/
>>> Log4jImportCallable.java
>>>>>>>>>>>>>> ) but this was way easier since i could simply search for
>>>>>>>>>>>>>> "" to find the position at which I could split the stream
>>>>>>>>>>>>>> of events into a valid XML document for every single event .
>>> This was
>>>>>>>>>>>>>> still kind of nasty but doable, especially since processing an
>>>>>>>>>>> "offline"
>>>>>>>>>>>>>> XML file doesn't have the same performance restriction as
>>> handling of
>>>>>>>>>>>>>> live events.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I wouldn't have the luxury of such a unique split signal in
>>> JSON,
>>>>>>>>>>> though.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I would need to detect the "}" at the level of the root array
>>> which
>>>>>>>>>>>>>> would not simply involve counting of opening and closing
>>> brackets but
>>>>>>>>>>>>>> also ignoring of brackets inside of strings. This would boil
>>> down to
>>>>>>>>>>>>>> implementing a custom JSON reader and I won't be doing this.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This isn't just me being lazy either. Such an implementation
>>> wouldn't
>>>>>>>>>>> be
>>>>>>>>>>>>>> resource (CPU/memory) friendly anyway.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> In my own JSON receiver using my own JSON format there are
>>> two ways to
>>>>>>>>>>>>>> send those events:
>>>>>>>>>>>>>> - write an int containing the amount of bytes representing
>>> the event,
>>>>>>>>>>>>>> then the bytes of the event. This type of events also supports
>>>>>>>>>>>>>> compression.
>>>>>>>>>>>>>> - write the bytes of the event, followed by a 0-byte. This
>>> works fine
>>>>>>>>>>>>>> and JavaScript/ActionScript is able to produce events like
>>> that while
>>>>>>>>>>>>>> they are unable (or were unable 7 years ago) to count the
>>> bytes on
>>>>>>>>>>> their
>>>>>>>>>>>>>> own. This type of events only supports plain text like JSON
>>> or XML and
>>>>>>>>>>>>>> no compression since compressed events could contain the
>>> 0-byte while
>>>>>>>>>>>>>> XML and JSON both won't allow it.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Both are essentially "binary" formats because of either the
>>> raw ints or
>>>>>>>>>>>>>> the 0-bytes.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm not sure why you deprecate SerializedLayout since the
>>> security
>>>>>>>>>>>>>> issues arise only while deserializing the events, not while
>>> serializing
>>>>>>>>>>>>>> them. Is this just meant to educate the user about the issue?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I fixed the remote code execution exploit by implementing
>>>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
>>>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
>>>>>>>>>>>>>> er/WhitelistObjectInputStream.java
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM in
>>> case of
>>>>>>>>>>>>>> malicious data) but is close enough for me, mainly because
>>> fixing the
>>>>>>>>>>>>>> DoS issues would really have to be handled by Java, not by
>>> user code.
>>>>>>>>>>>>>> Manually re-implementing deserialization would likely be
>>> fragile and
>>>>>>>>>>>>>> error prone, possibly even introducing other security issues
>>> on its
>>>>>>>>>>> own.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'd be interested how other tools are tackling this issue. Is
>>> Chainsaw
>>>>>>>>>>>>>> officially dead or are they implementing receivers for event
>>> streams
>>>>>>>>>>>>>> like this?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm totally fine with implementing a new receiver as a
>>> replacement for
>>>>>>>>>>>>>> log4j2 SerializedLayout (even though I won't be able to get
>>> rid of the
>>>>>>>>>>>>>> Serializable receiver anyway) but the current JsonLayout just
>>> isn't up
>>>>>>>>>>>>>> to the job at the moment.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Feel free to share this mail with your fellow developers. I'm
>>> open for
>>>>>>>>>>>>>> suggestions.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I hope this doesn't come across in a harsh way. It isn't
>>> meant to be.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Jörn.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
> 
> 

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

But here we have a concrete use case: a third-party tool, Lilith, needs 
to parse log events from our JsonLayout. (Our SocketServer in 
log4j-server have the same need, and it uses this class.)

Should we provide a public API in log4j-core for doing so, which Lilith, 
our SocketServer, and possibly others (Chainsaw maybe?) can use? Maybe 
such public API should be a facade around this which hides the fact that 
we use Jackson.


On 2017-07-22 19:04, Gary Gregory wrote:
> Hi All,
> 
> Just like anything in log4j-core, we try not break binary compatibility
> when we can but we do not bend over backwards like we do for log4j-api.
> 
> This class is public probably because Jackson requires it to be, otherwise
> I would have made it package private. The fact that we use Jackson for JSON
> IO is an implementation detail IMO, not an extension point. If anyone wants
> to hard-code a reference to log4j-core's guts then I think its fair to day
> that this comes with caveats.
> 
> Gary
> 
> On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn <jh...@googlemail.com>
> wrote:
> 
>> Well… it says
>>
>> /**
>>   * A Jackson JSON {@link ObjectMapper} initialized for Log4j.
>>   * <p>
>>   * <em>Consider this class private.</em>
>>   * </p>
>>   */
>>
>> so I wanted to holler about how to continue.
>>
>> I’ll give it a shot.
>>
>>
>> On 22. July 2017 at 15:06:28, Mikael Ståldal (mikes@apache.org) wrote:
>>> Would it work to use
>>> org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
>>> public?
>>>
>>> See here how it is used:
>>>
>>> https://github.com/apache/logging-log4j-tools/blob/
>> master/log4j-server/src/main/java/org/apache/logging/log4j/server/
>> JsonInputStreamLogEventBridge.java
>>>
>>>
>>> On 2017-07-22 13:47, Jörn Huxhorn wrote:
>>>> It seems like I don’t get access to the log4j2 ObjectMapper or Jackson
>> Module used for
>>> JSON serialization, i.e. I can’t do something like this:
>>>> new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace,
>> false)
>>>>
>>>> It would make sense to reuse your ObjectMapper/Module in my code
>> instead of reimplementing
>>> them to prevent future compatibility issues down the road.
>>>>
>>>>
>>>> On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxhorn@googlemail.com)
>> wrote:
>>>>> I’ll give this a shot over the next view days.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>> On 19. July 2017 at 21:33:49, Mikael Ståldal (mikes@apache.org)
>> wrote:
>>>>>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
>>>>>> termination of log events. Will be part of the upcoming 2.9 release.
>>>>>>
>>>>>>
>>>>>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
>>>>>>> A general event header and footer would probably be overkill.
>>>>>>>
>>>>>>> I’m using the following interface in Lilith:
>>>>>>>
>>>>>>> public interface WriteByteStrategy {
>>>>>>> void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
>>>>>>> throws IOException;
>>>>>>> }
>>>>>>>
>>>>>>> One implementation writes an int with the length of bytes before
>> “bytes", the other
>>>>>> implementation writes a 0-byte after “bytes".
>>>>>>>
>>>>>>> The problem with that approach is that it requires the intermediate
>> “bytes” array
>>>>> and
>>>>>> is thus probably incompatible with your zero garbage approach.
>>>>>>>
>>>>>>> I didn’t try end-of-line (\r\n) mainly because I first wanted to
>> discuss this with
>>>>> you.
>>>>>> I would also have to implement a new receiver type whereas I could
>> just reuse an existing
>>>>>> one in case of 0-byte termination.
>>>>>>>
>>>>>>> I’d prefer using 0-byte termination since it would be less fragile
>> than end-of-line.
>>>>>> The end-of-line approach would break if compact=“false” was set
>> erroneously while
>>>>>> you’d still have the option of using either compact=“true” or
>> compact=“false” in
>>> case
>>>>>> of 0-byte termination. 0-byte termination is also slightly more
>> efficient while
>>> reading
>>>>>> since the splitting can operate on bytes instead of chars or Strings
>> - another reason
>>>>>> why I didn’t just implement it and instead wanted to discuss this
>> with you first.
>>>>>>>
>>>>>>> Joern
>>>>>>>
>>>>>>>
>>>>>>> On 18. July 2017 at 03:26:32, Ralph Goers (
>> ralph.goers@dslextreme.com) wrote:
>>>>>>>> Right now that is all handled by the specific layouts. For
>> example, by default the
>>>>> RFC5424Layout
>>>>>>>> doesn’t append newlines so when writing to a file they will all be
>> on the same “line”,
>>>>>> but
>>>>>>>> it has an option to append one if needed. Doing the same would be
>> another option for
>>>>> the
>>>>>>>> JSONLayout.
>>>>>>>>
>>>>>>>> I’d be ok with event header and footers but ONLY if they have zero
>> overhead when not
>>>>> present.
>>>>>>>> IOW, when the Layout is initialized it would have to wire in the
>> appropriate encode
>>>>>> method.
>>>>>>>>
>>>>>>>> Ralph
>>>>>>>>
>>>>>>>>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
>>>>>>>>>
>>>>>>>>> Do we want a general event header and footer then?
>>>>>>>>>
>>>>>>>>> Gary
>>>>>>>>>
>>>>>>>>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
>>>>>>>>>
>>>>>>>>>> No. A Footer is only used at end of file. He needs to know how
>> long each
>>>>>>>>>> event is or when it is the start of a new event.
>>>>>>>>>>
>>>>>>>>>> Ralph
>>>>>>>>>>
>>>>>>>>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Can't you use a footer for any terminator you wish?
>>>>>>>>>>>
>>>>>>>>>>> Gary
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi.
>>>>>>>>>>>>
>>>>>>>>>>>> (Moving this discussion to logging dev mailing list.)
>>>>>>>>>>>>
>>>>>>>>>>>> Have you tried to use:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Then each log event will be terminated by end-of-line (\r\n).
>>>>>>>>>>>>
>>>>>>>>>>>> I think it would be easy to implement 0-byte terminated log
>> events in
>>>>>>>>>>>> JsonLayout, and that would make sense since we have
>> implemented support
>>>>>>>>>> for
>>>>>>>>>>>> that in GelfLayout. Created a JIRA issue for it:
>>>>>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
>>>>>>>>>>>>
>>>>>>>>>>>> As for other tools, the only receivers for Log4j 2
>> SerializedLayout we
>>>>>>>>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw
>> currently only
>>>>>>>>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer
>> support
>>>>>>>>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
>>>>>>>>>> JsonLayout
>>>>>>>>>>>> by default.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Mikael,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've just taken a look at the JsonLayout and see no way to
>> parse its
>>>>>>>>>>>>> output event by event.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's semi-fine for files because then it will result in a
>> well-formed
>>>>>>>>>>>>> array at the root level that can be read in one big swoop. It
>> isn't
>>>>>>>>>>>>> really ideal for that use case either since it means that the
>> whole
>>>>>>>>>>>>> array containing all events needs to be read into memory in
>> one piece,
>>>>>>>>>>>>> causing OOM in case of huge files.
>>>>>>>>>>>>>
>>>>>>>>>>>>> But in Lilith, I really need to process events one by one.
>> This is
>>>>>>>>>>>>> absolutely crucial.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The only way to achieve this would essentially require me to
>>>>>>>>>>>>> re-implement a JSON parser that can cope with data like this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I implemented a similar kludge to support reading of log4j
>> xml files
>>>>>>>>>>>>> (see
>>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
>>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/
>> Log4jImportCallable.java
>>>>>>>>>>>>> ) but this was way easier since i could simply search for
>>>>>>>>>>>>> "" to find the position at which I could split the stream
>>>>>>>>>>>>> of events into a valid XML document for every single event .
>> This was
>>>>>>>>>>>>> still kind of nasty but doable, especially since processing an
>>>>>>>>>> "offline"
>>>>>>>>>>>>> XML file doesn't have the same performance restriction as
>> handling of
>>>>>>>>>>>>> live events.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I wouldn't have the luxury of such a unique split signal in
>> JSON,
>>>>>>>>>> though.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I would need to detect the "}" at the level of the root array
>> which
>>>>>>>>>>>>> would not simply involve counting of opening and closing
>> brackets but
>>>>>>>>>>>>> also ignoring of brackets inside of strings. This would boil
>> down to
>>>>>>>>>>>>> implementing a custom JSON reader and I won't be doing this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This isn't just me being lazy either. Such an implementation
>> wouldn't
>>>>>>>>>> be
>>>>>>>>>>>>> resource (CPU/memory) friendly anyway.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In my own JSON receiver using my own JSON format there are
>> two ways to
>>>>>>>>>>>>> send those events:
>>>>>>>>>>>>> - write an int containing the amount of bytes representing
>> the event,
>>>>>>>>>>>>> then the bytes of the event. This type of events also supports
>>>>>>>>>>>>> compression.
>>>>>>>>>>>>> - write the bytes of the event, followed by a 0-byte. This
>> works fine
>>>>>>>>>>>>> and JavaScript/ActionScript is able to produce events like
>> that while
>>>>>>>>>>>>> they are unable (or were unable 7 years ago) to count the
>> bytes on
>>>>>>>>>> their
>>>>>>>>>>>>> own. This type of events only supports plain text like JSON
>> or XML and
>>>>>>>>>>>>> no compression since compressed events could contain the
>> 0-byte while
>>>>>>>>>>>>> XML and JSON both won't allow it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Both are essentially "binary" formats because of either the
>> raw ints or
>>>>>>>>>>>>> the 0-bytes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm not sure why you deprecate SerializedLayout since the
>> security
>>>>>>>>>>>>> issues arise only while deserializing the events, not while
>> serializing
>>>>>>>>>>>>> them. Is this just meant to educate the user about the issue?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I fixed the remote code execution exploit by implementing
>>>>>>>>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
>>>>>>>>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
>>>>>>>>>>>>> er/WhitelistObjectInputStream.java
>>>>>>>>>>>>>
>>>>>>>>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM in
>> case of
>>>>>>>>>>>>> malicious data) but is close enough for me, mainly because
>> fixing the
>>>>>>>>>>>>> DoS issues would really have to be handled by Java, not by
>> user code.
>>>>>>>>>>>>> Manually re-implementing deserialization would likely be
>> fragile and
>>>>>>>>>>>>> error prone, possibly even introducing other security issues
>> on its
>>>>>>>>>> own.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'd be interested how other tools are tackling this issue. Is
>> Chainsaw
>>>>>>>>>>>>> officially dead or are they implementing receivers for event
>> streams
>>>>>>>>>>>>> like this?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm totally fine with implementing a new receiver as a
>> replacement for
>>>>>>>>>>>>> log4j2 SerializedLayout (even though I won't be able to get
>> rid of the
>>>>>>>>>>>>> Serializable receiver anyway) but the current JsonLayout just
>> isn't up
>>>>>>>>>>>>> to the job at the moment.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Feel free to share this mail with your fellow developers. I'm
>> open for
>>>>>>>>>>>>> suggestions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I hope this doesn't come across in a harsh way. It isn't
>> meant to be.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Jörn.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
> 

Re: Lilith and Log4j 2

[Gary Gregory <ga...@gmail.com>]

Hi All,

Just like anything in log4j-core, we try not break binary compatibility
when we can but we do not bend over backwards like we do for log4j-api.

This class is public probably because Jackson requires it to be, otherwise
I would have made it package private. The fact that we use Jackson for JSON
IO is an implementation detail IMO, not an extension point. If anyone wants
to hard-code a reference to log4j-core's guts then I think its fair to day
that this comes with caveats.

Gary

On Sat, Jul 22, 2017 at 7:27 AM, Jörn Huxhorn <jh...@googlemail.com>
wrote:

> Well… it says
>
> /**
>  * A Jackson JSON {@link ObjectMapper} initialized for Log4j.
>  * <p>
>  * <em>Consider this class private.</em>
>  * </p>
>  */
>
> so I wanted to holler about how to continue.
>
> I’ll give it a shot.
>
>
> On 22. July 2017 at 15:06:28, Mikael Ståldal (mikes@apache.org) wrote:
> > Would it work to use
> > org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
> > public?
> >
> > See here how it is used:
> >
> > https://github.com/apache/logging-log4j-tools/blob/
> master/log4j-server/src/main/java/org/apache/logging/log4j/server/
> JsonInputStreamLogEventBridge.java
> >
> >
> > On 2017-07-22 13:47, Jörn Huxhorn wrote:
> > > It seems like I don’t get access to the log4j2 ObjectMapper or Jackson
> Module used for
> > JSON serialization, i.e. I can’t do something like this:
> > > new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace,
> false)
> > >
> > > It would make sense to reuse your ObjectMapper/Module in my code
> instead of reimplementing
> > them to prevent future compatibility issues down the road.
> > >
> > >
> > > On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxhorn@googlemail.com)
> wrote:
> > >> I’ll give this a shot over the next view days.
> > >>
> > >> Thanks!
> > >>
> > >>
> > >> On 19. July 2017 at 21:33:49, Mikael Ståldal (mikes@apache.org)
> wrote:
> > >>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
> > >>> termination of log events. Will be part of the upcoming 2.9 release.
> > >>>
> > >>>
> > >>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
> > >>>> A general event header and footer would probably be overkill.
> > >>>>
> > >>>> I’m using the following interface in Lilith:
> > >>>>
> > >>>> public interface WriteByteStrategy {
> > >>>> void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
> > >>>> throws IOException;
> > >>>> }
> > >>>>
> > >>>> One implementation writes an int with the length of bytes before
> “bytes", the other
> > >>> implementation writes a 0-byte after “bytes".
> > >>>>
> > >>>> The problem with that approach is that it requires the intermediate
> “bytes” array
> > >> and
> > >>> is thus probably incompatible with your zero garbage approach.
> > >>>>
> > >>>> I didn’t try end-of-line (\r\n) mainly because I first wanted to
> discuss this with
> > >> you.
> > >>> I would also have to implement a new receiver type whereas I could
> just reuse an existing
> > >>> one in case of 0-byte termination.
> > >>>>
> > >>>> I’d prefer using 0-byte termination since it would be less fragile
> than end-of-line.
> > >>> The end-of-line approach would break if compact=“false” was set
> erroneously while
> > >>> you’d still have the option of using either compact=“true” or
> compact=“false” in
> > case
> > >>> of 0-byte termination. 0-byte termination is also slightly more
> efficient while
> > reading
> > >>> since the splitting can operate on bytes instead of chars or Strings
> - another reason
> > >>> why I didn’t just implement it and instead wanted to discuss this
> with you first.
> > >>>>
> > >>>> Joern
> > >>>>
> > >>>>
> > >>>> On 18. July 2017 at 03:26:32, Ralph Goers (
> ralph.goers@dslextreme.com) wrote:
> > >>>>> Right now that is all handled by the specific layouts. For
> example, by default the
> > >> RFC5424Layout
> > >>>>> doesn’t append newlines so when writing to a file they will all be
> on the same “line”,
> > >>> but
> > >>>>> it has an option to append one if needed. Doing the same would be
> another option for
> > >> the
> > >>>>> JSONLayout.
> > >>>>>
> > >>>>> I’d be ok with event header and footers but ONLY if they have zero
> overhead when not
> > >> present.
> > >>>>> IOW, when the Layout is initialized it would have to wire in the
> appropriate encode
> > >>> method.
> > >>>>>
> > >>>>> Ralph
> > >>>>>
> > >>>>>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
> > >>>>>>
> > >>>>>> Do we want a general event header and footer then?
> > >>>>>>
> > >>>>>> Gary
> > >>>>>>
> > >>>>>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
> > >>>>>>
> > >>>>>>> No. A Footer is only used at end of file. He needs to know how
> long each
> > >>>>>>> event is or when it is the start of a new event.
> > >>>>>>>
> > >>>>>>> Ralph
> > >>>>>>>
> > >>>>>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> Can't you use a footer for any terminator you wish?
> > >>>>>>>>
> > >>>>>>>> Gary
> > >>>>>>>>
> > >>>>>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> Hi.
> > >>>>>>>>>
> > >>>>>>>>> (Moving this discussion to logging dev mailing list.)
> > >>>>>>>>>
> > >>>>>>>>> Have you tried to use:
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Then each log event will be terminated by end-of-line (\r\n).
> > >>>>>>>>>
> > >>>>>>>>> I think it would be easy to implement 0-byte terminated log
> events in
> > >>>>>>>>> JsonLayout, and that would make sense since we have
> implemented support
> > >>>>>>> for
> > >>>>>>>>> that in GelfLayout. Created a JIRA issue for it:
> > >>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
> > >>>>>>>>>
> > >>>>>>>>> As for other tools, the only receivers for Log4j 2
> SerializedLayout we
> > >>>>>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw
> currently only
> > >>>>>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer
> support
> > >>>>>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
> > >>>>>>> JsonLayout
> > >>>>>>>>> by default.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
> > >>>>>>>>>
> > >>>>>>>>>> Hi Mikael,
> > >>>>>>>>>>
> > >>>>>>>>>> I've just taken a look at the JsonLayout and see no way to
> parse its
> > >>>>>>>>>> output event by event.
> > >>>>>>>>>>
> > >>>>>>>>>> It's semi-fine for files because then it will result in a
> well-formed
> > >>>>>>>>>> array at the root level that can be read in one big swoop. It
> isn't
> > >>>>>>>>>> really ideal for that use case either since it means that the
> whole
> > >>>>>>>>>> array containing all events needs to be read into memory in
> one piece,
> > >>>>>>>>>> causing OOM in case of huge files.
> > >>>>>>>>>>
> > >>>>>>>>>> But in Lilith, I really need to process events one by one.
> This is
> > >>>>>>>>>> absolutely crucial.
> > >>>>>>>>>>
> > >>>>>>>>>> The only way to achieve this would essentially require me to
> > >>>>>>>>>> re-implement a JSON parser that can cope with data like this.
> > >>>>>>>>>>
> > >>>>>>>>>> I implemented a similar kludge to support reading of log4j
> xml files
> > >>>>>>>>>> (see
> > >>>>>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
> > >>>>>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/
> Log4jImportCallable.java
> > >>>>>>>>>> ) but this was way easier since i could simply search for
> > >>>>>>>>>> "" to find the position at which I could split the stream
> > >>>>>>>>>> of events into a valid XML document for every single event .
> This was
> > >>>>>>>>>> still kind of nasty but doable, especially since processing an
> > >>>>>>> "offline"
> > >>>>>>>>>> XML file doesn't have the same performance restriction as
> handling of
> > >>>>>>>>>> live events.
> > >>>>>>>>>>
> > >>>>>>>>>> I wouldn't have the luxury of such a unique split signal in
> JSON,
> > >>>>>>> though.
> > >>>>>>>>>>
> > >>>>>>>>>> I would need to detect the "}" at the level of the root array
> which
> > >>>>>>>>>> would not simply involve counting of opening and closing
> brackets but
> > >>>>>>>>>> also ignoring of brackets inside of strings. This would boil
> down to
> > >>>>>>>>>> implementing a custom JSON reader and I won't be doing this.
> > >>>>>>>>>>
> > >>>>>>>>>> This isn't just me being lazy either. Such an implementation
> wouldn't
> > >>>>>>> be
> > >>>>>>>>>> resource (CPU/memory) friendly anyway.
> > >>>>>>>>>>
> > >>>>>>>>>> In my own JSON receiver using my own JSON format there are
> two ways to
> > >>>>>>>>>> send those events:
> > >>>>>>>>>> - write an int containing the amount of bytes representing
> the event,
> > >>>>>>>>>> then the bytes of the event. This type of events also supports
> > >>>>>>>>>> compression.
> > >>>>>>>>>> - write the bytes of the event, followed by a 0-byte. This
> works fine
> > >>>>>>>>>> and JavaScript/ActionScript is able to produce events like
> that while
> > >>>>>>>>>> they are unable (or were unable 7 years ago) to count the
> bytes on
> > >>>>>>> their
> > >>>>>>>>>> own. This type of events only supports plain text like JSON
> or XML and
> > >>>>>>>>>> no compression since compressed events could contain the
> 0-byte while
> > >>>>>>>>>> XML and JSON both won't allow it.
> > >>>>>>>>>>
> > >>>>>>>>>> Both are essentially "binary" formats because of either the
> raw ints or
> > >>>>>>>>>> the 0-bytes.
> > >>>>>>>>>>
> > >>>>>>>>>> I'm not sure why you deprecate SerializedLayout since the
> security
> > >>>>>>>>>> issues arise only while deserializing the events, not while
> serializing
> > >>>>>>>>>> them. Is this just meant to educate the user about the issue?
> > >>>>>>>>>>
> > >>>>>>>>>> I fixed the remote code execution exploit by implementing
> > >>>>>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
> > >>>>>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
> > >>>>>>>>>> er/WhitelistObjectInputStream.java
> > >>>>>>>>>>
> > >>>>>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM in
> case of
> > >>>>>>>>>> malicious data) but is close enough for me, mainly because
> fixing the
> > >>>>>>>>>> DoS issues would really have to be handled by Java, not by
> user code.
> > >>>>>>>>>> Manually re-implementing deserialization would likely be
> fragile and
> > >>>>>>>>>> error prone, possibly even introducing other security issues
> on its
> > >>>>>>> own.
> > >>>>>>>>>>
> > >>>>>>>>>> I'd be interested how other tools are tackling this issue. Is
> Chainsaw
> > >>>>>>>>>> officially dead or are they implementing receivers for event
> streams
> > >>>>>>>>>> like this?
> > >>>>>>>>>>
> > >>>>>>>>>> I'm totally fine with implementing a new receiver as a
> replacement for
> > >>>>>>>>>> log4j2 SerializedLayout (even though I won't be able to get
> rid of the
> > >>>>>>>>>> Serializable receiver anyway) but the current JsonLayout just
> isn't up
> > >>>>>>>>>> to the job at the moment.
> > >>>>>>>>>>
> > >>>>>>>>>> Feel free to share this mail with your fellow developers. I'm
> open for
> > >>>>>>>>>> suggestions.
> > >>>>>>>>>>
> > >>>>>>>>>> I hope this doesn't come across in a harsh way. It isn't
> meant to be.
> > >>>>>>>>>>
> > >>>>>>>>>> Cheers,
> > >>>>>>>>>> Jörn.
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>
> > >>
> > >
> >
> >
>
>

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

Well… it says

/**
 * A Jackson JSON {@link ObjectMapper} initialized for Log4j.
 * <p>
 * <em>Consider this class private.</em>
 * </p>
 */

so I wanted to holler about how to continue.

I’ll give it a shot.


On 22. July 2017 at 15:06:28, Mikael Ståldal (mikes@apache.org) wrote:
> Would it work to use
> org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is
> public?
>  
> See here how it is used:
>  
> https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/JsonInputStreamLogEventBridge.java  
>  
>  
> On 2017-07-22 13:47, Jörn Huxhorn wrote:
> > It seems like I don’t get access to the log4j2 ObjectMapper or Jackson Module used for  
> JSON serialization, i.e. I can’t do something like this:
> > new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace, false)  
> >
> > It would make sense to reuse your ObjectMapper/Module in my code instead of reimplementing  
> them to prevent future compatibility issues down the road.
> >
> >
> > On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxhorn@googlemail.com) wrote:
> >> I’ll give this a shot over the next view days.
> >>
> >> Thanks!
> >>
> >>
> >> On 19. July 2017 at 21:33:49, Mikael Ståldal (mikes@apache.org) wrote:
> >>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
> >>> termination of log events. Will be part of the upcoming 2.9 release.
> >>>
> >>>
> >>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
> >>>> A general event header and footer would probably be overkill.
> >>>>
> >>>> I’m using the following interface in Lilith:
> >>>>
> >>>> public interface WriteByteStrategy {
> >>>> void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
> >>>> throws IOException;
> >>>> }
> >>>>
> >>>> One implementation writes an int with the length of bytes before “bytes", the other  
> >>> implementation writes a 0-byte after “bytes".
> >>>>
> >>>> The problem with that approach is that it requires the intermediate “bytes” array  
> >> and
> >>> is thus probably incompatible with your zero garbage approach.
> >>>>
> >>>> I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this with  
> >> you.
> >>> I would also have to implement a new receiver type whereas I could just reuse an existing  
> >>> one in case of 0-byte termination.
> >>>>
> >>>> I’d prefer using 0-byte termination since it would be less fragile than end-of-line.  
> >>> The end-of-line approach would break if compact=“false” was set erroneously while  
> >>> you’d still have the option of using either compact=“true” or compact=“false” in  
> case
> >>> of 0-byte termination. 0-byte termination is also slightly more efficient while  
> reading
> >>> since the splitting can operate on bytes instead of chars or Strings - another reason  
> >>> why I didn’t just implement it and instead wanted to discuss this with you first.
> >>>>
> >>>> Joern
> >>>>
> >>>>
> >>>> On 18. July 2017 at 03:26:32, Ralph Goers (ralph.goers@dslextreme.com) wrote:  
> >>>>> Right now that is all handled by the specific layouts. For example, by default the  
> >> RFC5424Layout
> >>>>> doesn’t append newlines so when writing to a file they will all be on the same “line”,  
> >>> but
> >>>>> it has an option to append one if needed. Doing the same would be another option for  
> >> the
> >>>>> JSONLayout.
> >>>>>
> >>>>> I’d be ok with event header and footers but ONLY if they have zero overhead when not  
> >> present.
> >>>>> IOW, when the Layout is initialized it would have to wire in the appropriate encode  
> >>> method.
> >>>>>
> >>>>> Ralph
> >>>>>
> >>>>>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
> >>>>>>
> >>>>>> Do we want a general event header and footer then?
> >>>>>>
> >>>>>> Gary
> >>>>>>
> >>>>>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
> >>>>>>
> >>>>>>> No. A Footer is only used at end of file. He needs to know how long each
> >>>>>>> event is or when it is the start of a new event.
> >>>>>>>
> >>>>>>> Ralph
> >>>>>>>
> >>>>>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Can't you use a footer for any terminator you wish?
> >>>>>>>>
> >>>>>>>> Gary
> >>>>>>>>
> >>>>>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Hi.
> >>>>>>>>>
> >>>>>>>>> (Moving this discussion to logging dev mailing list.)
> >>>>>>>>>
> >>>>>>>>> Have you tried to use:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Then each log event will be terminated by end-of-line (\r\n).
> >>>>>>>>>
> >>>>>>>>> I think it would be easy to implement 0-byte terminated log events in
> >>>>>>>>> JsonLayout, and that would make sense since we have implemented support
> >>>>>>> for
> >>>>>>>>> that in GelfLayout. Created a JIRA issue for it:
> >>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
> >>>>>>>>>
> >>>>>>>>> As for other tools, the only receivers for Log4j 2 SerializedLayout we
> >>>>>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
> >>>>>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
> >>>>>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
> >>>>>>> JsonLayout
> >>>>>>>>> by default.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
> >>>>>>>>>
> >>>>>>>>>> Hi Mikael,
> >>>>>>>>>>
> >>>>>>>>>> I've just taken a look at the JsonLayout and see no way to parse its
> >>>>>>>>>> output event by event.
> >>>>>>>>>>
> >>>>>>>>>> It's semi-fine for files because then it will result in a well-formed
> >>>>>>>>>> array at the root level that can be read in one big swoop. It isn't
> >>>>>>>>>> really ideal for that use case either since it means that the whole
> >>>>>>>>>> array containing all events needs to be read into memory in one piece,
> >>>>>>>>>> causing OOM in case of huge files.
> >>>>>>>>>>
> >>>>>>>>>> But in Lilith, I really need to process events one by one. This is
> >>>>>>>>>> absolutely crucial.
> >>>>>>>>>>
> >>>>>>>>>> The only way to achieve this would essentially require me to
> >>>>>>>>>> re-implement a JSON parser that can cope with data like this.
> >>>>>>>>>>
> >>>>>>>>>> I implemented a similar kludge to support reading of log4j xml files
> >>>>>>>>>> (see
> >>>>>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
> >>>>>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java  
> >>>>>>>>>> ) but this was way easier since i could simply search for
> >>>>>>>>>> "" to find the position at which I could split the stream
> >>>>>>>>>> of events into a valid XML document for every single event . This was
> >>>>>>>>>> still kind of nasty but doable, especially since processing an
> >>>>>>> "offline"
> >>>>>>>>>> XML file doesn't have the same performance restriction as handling of
> >>>>>>>>>> live events.
> >>>>>>>>>>
> >>>>>>>>>> I wouldn't have the luxury of such a unique split signal in JSON,
> >>>>>>> though.
> >>>>>>>>>>
> >>>>>>>>>> I would need to detect the "}" at the level of the root array which
> >>>>>>>>>> would not simply involve counting of opening and closing brackets but
> >>>>>>>>>> also ignoring of brackets inside of strings. This would boil down to
> >>>>>>>>>> implementing a custom JSON reader and I won't be doing this.
> >>>>>>>>>>
> >>>>>>>>>> This isn't just me being lazy either. Such an implementation wouldn't
> >>>>>>> be
> >>>>>>>>>> resource (CPU/memory) friendly anyway.
> >>>>>>>>>>
> >>>>>>>>>> In my own JSON receiver using my own JSON format there are two ways to
> >>>>>>>>>> send those events:
> >>>>>>>>>> - write an int containing the amount of bytes representing the event,
> >>>>>>>>>> then the bytes of the event. This type of events also supports
> >>>>>>>>>> compression.
> >>>>>>>>>> - write the bytes of the event, followed by a 0-byte. This works fine
> >>>>>>>>>> and JavaScript/ActionScript is able to produce events like that while
> >>>>>>>>>> they are unable (or were unable 7 years ago) to count the bytes on
> >>>>>>> their
> >>>>>>>>>> own. This type of events only supports plain text like JSON or XML and
> >>>>>>>>>> no compression since compressed events could contain the 0-byte while
> >>>>>>>>>> XML and JSON both won't allow it.
> >>>>>>>>>>
> >>>>>>>>>> Both are essentially "binary" formats because of either the raw ints or
> >>>>>>>>>> the 0-bytes.
> >>>>>>>>>>
> >>>>>>>>>> I'm not sure why you deprecate SerializedLayout since the security
> >>>>>>>>>> issues arise only while deserializing the events, not while serializing  
> >>>>>>>>>> them. Is this just meant to educate the user about the issue?
> >>>>>>>>>>
> >>>>>>>>>> I fixed the remote code execution exploit by implementing
> >>>>>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
> >>>>>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
> >>>>>>>>>> er/WhitelistObjectInputStream.java
> >>>>>>>>>>
> >>>>>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
> >>>>>>>>>> malicious data) but is close enough for me, mainly because fixing the
> >>>>>>>>>> DoS issues would really have to be handled by Java, not by user code.
> >>>>>>>>>> Manually re-implementing deserialization would likely be fragile and  
> >>>>>>>>>> error prone, possibly even introducing other security issues on its
> >>>>>>> own.
> >>>>>>>>>>
> >>>>>>>>>> I'd be interested how other tools are tackling this issue. Is Chainsaw
> >>>>>>>>>> officially dead or are they implementing receivers for event streams
> >>>>>>>>>> like this?
> >>>>>>>>>>
> >>>>>>>>>> I'm totally fine with implementing a new receiver as a replacement for
> >>>>>>>>>> log4j2 SerializedLayout (even though I won't be able to get rid of the
> >>>>>>>>>> Serializable receiver anyway) but the current JsonLayout just isn't up  
> >>>>>>>>>> to the job at the moment.
> >>>>>>>>>>
> >>>>>>>>>> Feel free to share this mail with your fellow developers. I'm open for
> >>>>>>>>>> suggestions.
> >>>>>>>>>>
> >>>>>>>>>> I hope this doesn't come across in a harsh way. It isn't meant to be.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Jörn.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >
>  
>  

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

Would it work to use 
org.apache.logging.log4j.core.jackson.Log4jJsonObjectMapper, which is 
public?

See here how it is used:

https://github.com/apache/logging-log4j-tools/blob/master/log4j-server/src/main/java/org/apache/logging/log4j/server/JsonInputStreamLogEventBridge.java


On 2017-07-22 13:47, Jörn Huxhorn wrote:
> It seems like I don’t get access to the log4j2 ObjectMapper or Jackson Module used for JSON serialization, i.e. I can’t do something like this:
> new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace, false)
> 
> It would make sense to reuse your ObjectMapper/Module in my code instead of reimplementing them to prevent future compatibility issues down the road.
> 
> 
> On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxhorn@googlemail.com) wrote:
>> I’ll give this a shot over the next view days.
>>   
>> Thanks!
>>   
>>   
>> On 19. July 2017 at 21:33:49, Mikael Ståldal (mikes@apache.org) wrote:
>>> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
>>> termination of log events. Will be part of the upcoming 2.9 release.
>>>
>>>
>>> On 2017-07-18 11:48, Jörn Huxhorn wrote:
>>>> A general event header and footer would probably be overkill.
>>>>
>>>> I’m using the following interface in Lilith:
>>>>
>>>> public interface WriteByteStrategy {
>>>> void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
>>>> throws IOException;
>>>> }
>>>>
>>>> One implementation writes an int with the length of bytes before “bytes", the other
>>> implementation writes a 0-byte after “bytes".
>>>>
>>>> The problem with that approach is that it requires the intermediate “bytes” array
>> and
>>> is thus probably incompatible with your zero garbage approach.
>>>>
>>>> I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this with
>> you.
>>> I would also have to implement a new receiver type whereas I could just reuse an existing
>>> one in case of 0-byte termination.
>>>>
>>>> I’d prefer using 0-byte termination since it would be less fragile than end-of-line.
>>> The end-of-line approach would break if compact=“false” was set erroneously while
>>> you’d still have the option of using either compact=“true” or compact=“false” in case
>>> of 0-byte termination. 0-byte termination is also slightly more efficient while reading
>>> since the splitting can operate on bytes instead of chars or Strings - another reason
>>> why I didn’t just implement it and instead wanted to discuss this with you first.
>>>>
>>>> Joern
>>>>
>>>>
>>>> On 18. July 2017 at 03:26:32, Ralph Goers (ralph.goers@dslextreme.com) wrote:
>>>>> Right now that is all handled by the specific layouts. For example, by default the
>> RFC5424Layout
>>>>> doesn’t append newlines so when writing to a file they will all be on the same “line”,
>>> but
>>>>> it has an option to append one if needed. Doing the same would be another option for
>> the
>>>>> JSONLayout.
>>>>>
>>>>> I’d be ok with event header and footers but ONLY if they have zero overhead when not
>> present.
>>>>> IOW, when the Layout is initialized it would have to wire in the appropriate encode
>>> method.
>>>>>
>>>>> Ralph
>>>>>
>>>>>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
>>>>>>
>>>>>> Do we want a general event header and footer then?
>>>>>>
>>>>>> Gary
>>>>>>
>>>>>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
>>>>>>
>>>>>>> No. A Footer is only used at end of file. He needs to know how long each
>>>>>>> event is or when it is the start of a new event.
>>>>>>>
>>>>>>> Ralph
>>>>>>>
>>>>>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Can't you use a footer for any terminator you wish?
>>>>>>>>
>>>>>>>> Gary
>>>>>>>>
>>>>>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi.
>>>>>>>>>
>>>>>>>>> (Moving this discussion to logging dev mailing list.)
>>>>>>>>>
>>>>>>>>> Have you tried to use:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Then each log event will be terminated by end-of-line (\r\n).
>>>>>>>>>
>>>>>>>>> I think it would be easy to implement 0-byte terminated log events in
>>>>>>>>> JsonLayout, and that would make sense since we have implemented support
>>>>>>> for
>>>>>>>>> that in GelfLayout. Created a JIRA issue for it:
>>>>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
>>>>>>>>>
>>>>>>>>> As for other tools, the only receivers for Log4j 2 SerializedLayout we
>>>>>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
>>>>>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
>>>>>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
>>>>>>> JsonLayout
>>>>>>>>> by default.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
>>>>>>>>>
>>>>>>>>>> Hi Mikael,
>>>>>>>>>>
>>>>>>>>>> I've just taken a look at the JsonLayout and see no way to parse its
>>>>>>>>>> output event by event.
>>>>>>>>>>
>>>>>>>>>> It's semi-fine for files because then it will result in a well-formed
>>>>>>>>>> array at the root level that can be read in one big swoop. It isn't
>>>>>>>>>> really ideal for that use case either since it means that the whole
>>>>>>>>>> array containing all events needs to be read into memory in one piece,
>>>>>>>>>> causing OOM in case of huge files.
>>>>>>>>>>
>>>>>>>>>> But in Lilith, I really need to process events one by one. This is
>>>>>>>>>> absolutely crucial.
>>>>>>>>>>
>>>>>>>>>> The only way to achieve this would essentially require me to
>>>>>>>>>> re-implement a JSON parser that can cope with data like this.
>>>>>>>>>>
>>>>>>>>>> I implemented a similar kludge to support reading of log4j xml files
>>>>>>>>>> (see
>>>>>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
>>>>>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
>>>>>>>>>> ) but this was way easier since i could simply search for
>>>>>>>>>> "" to find the position at which I could split the stream
>>>>>>>>>> of events into a valid XML document for every single event . This was
>>>>>>>>>> still kind of nasty but doable, especially since processing an
>>>>>>> "offline"
>>>>>>>>>> XML file doesn't have the same performance restriction as handling of
>>>>>>>>>> live events.
>>>>>>>>>>
>>>>>>>>>> I wouldn't have the luxury of such a unique split signal in JSON,
>>>>>>> though.
>>>>>>>>>>
>>>>>>>>>> I would need to detect the "}" at the level of the root array which
>>>>>>>>>> would not simply involve counting of opening and closing brackets but
>>>>>>>>>> also ignoring of brackets inside of strings. This would boil down to
>>>>>>>>>> implementing a custom JSON reader and I won't be doing this.
>>>>>>>>>>
>>>>>>>>>> This isn't just me being lazy either. Such an implementation wouldn't
>>>>>>> be
>>>>>>>>>> resource (CPU/memory) friendly anyway.
>>>>>>>>>>
>>>>>>>>>> In my own JSON receiver using my own JSON format there are two ways to
>>>>>>>>>> send those events:
>>>>>>>>>> - write an int containing the amount of bytes representing the event,
>>>>>>>>>> then the bytes of the event. This type of events also supports
>>>>>>>>>> compression.
>>>>>>>>>> - write the bytes of the event, followed by a 0-byte. This works fine
>>>>>>>>>> and JavaScript/ActionScript is able to produce events like that while
>>>>>>>>>> they are unable (or were unable 7 years ago) to count the bytes on
>>>>>>> their
>>>>>>>>>> own. This type of events only supports plain text like JSON or XML and
>>>>>>>>>> no compression since compressed events could contain the 0-byte while
>>>>>>>>>> XML and JSON both won't allow it.
>>>>>>>>>>
>>>>>>>>>> Both are essentially "binary" formats because of either the raw ints or
>>>>>>>>>> the 0-bytes.
>>>>>>>>>>
>>>>>>>>>> I'm not sure why you deprecate SerializedLayout since the security
>>>>>>>>>> issues arise only while deserializing the events, not while serializing
>>>>>>>>>> them. Is this just meant to educate the user about the issue?
>>>>>>>>>>
>>>>>>>>>> I fixed the remote code execution exploit by implementing
>>>>>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
>>>>>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
>>>>>>>>>> er/WhitelistObjectInputStream.java
>>>>>>>>>>
>>>>>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
>>>>>>>>>> malicious data) but is close enough for me, mainly because fixing the
>>>>>>>>>> DoS issues would really have to be handled by Java, not by user code.
>>>>>>>>>> Manually re-implementing deserialization would likely be fragile and
>>>>>>>>>> error prone, possibly even introducing other security issues on its
>>>>>>> own.
>>>>>>>>>>
>>>>>>>>>> I'd be interested how other tools are tackling this issue. Is Chainsaw
>>>>>>>>>> officially dead or are they implementing receivers for event streams
>>>>>>>>>> like this?
>>>>>>>>>>
>>>>>>>>>> I'm totally fine with implementing a new receiver as a replacement for
>>>>>>>>>> log4j2 SerializedLayout (even though I won't be able to get rid of the
>>>>>>>>>> Serializable receiver anyway) but the current JsonLayout just isn't up
>>>>>>>>>> to the job at the moment.
>>>>>>>>>>
>>>>>>>>>> Feel free to share this mail with your fellow developers. I'm open for
>>>>>>>>>> suggestions.
>>>>>>>>>>
>>>>>>>>>> I hope this doesn't come across in a harsh way. It isn't meant to be.
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>> Jörn.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>   
>>   
> 

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

It seems like I don’t get access to the log4j2 ObjectMapper or Jackson Module used for JSON serialization, i.e. I can’t do something like this:
new JacksonFactory.JSON(encodeThreadContextAsList, includeStacktrace, false)

It would make sense to reuse your ObjectMapper/Module in my code instead of reimplementing them to prevent future compatibility issues down the road.


On 19. July 2017 at 23:04:26, Jörn Huxhorn (jhuxhorn@googlemail.com) wrote:
> I’ll give this a shot over the next view days.
>  
> Thanks!
>  
>  
> On 19. July 2017 at 21:33:49, Mikael Ståldal (mikes@apache.org) wrote:
> > JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
> > termination of log events. Will be part of the upcoming 2.9 release.
> >
> >
> > On 2017-07-18 11:48, Jörn Huxhorn wrote:
> > > A general event header and footer would probably be overkill.
> > >
> > > I’m using the following interface in Lilith:
> > >
> > > public interface WriteByteStrategy {
> > > void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
> > > throws IOException;
> > > }
> > >
> > > One implementation writes an int with the length of bytes before “bytes", the other  
> > implementation writes a 0-byte after “bytes".
> > >
> > > The problem with that approach is that it requires the intermediate “bytes” array  
> and
> > is thus probably incompatible with your zero garbage approach.
> > >
> > > I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this with  
> you.
> > I would also have to implement a new receiver type whereas I could just reuse an existing  
> > one in case of 0-byte termination.
> > >
> > > I’d prefer using 0-byte termination since it would be less fragile than end-of-line.  
> > The end-of-line approach would break if compact=“false” was set erroneously while  
> > you’d still have the option of using either compact=“true” or compact=“false” in case  
> > of 0-byte termination. 0-byte termination is also slightly more efficient while reading  
> > since the splitting can operate on bytes instead of chars or Strings - another reason  
> > why I didn’t just implement it and instead wanted to discuss this with you first.
> > >
> > > Joern
> > >
> > >
> > > On 18. July 2017 at 03:26:32, Ralph Goers (ralph.goers@dslextreme.com) wrote:
> > >> Right now that is all handled by the specific layouts. For example, by default the  
> RFC5424Layout
> > >> doesn’t append newlines so when writing to a file they will all be on the same “line”,  
> > but
> > >> it has an option to append one if needed. Doing the same would be another option for  
> the
> > >> JSONLayout.
> > >>
> > >> I’d be ok with event header and footers but ONLY if they have zero overhead when not  
> present.
> > >> IOW, when the Layout is initialized it would have to wire in the appropriate encode  
> > method.
> > >>
> > >> Ralph
> > >>
> > >>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
> > >>>
> > >>> Do we want a general event header and footer then?
> > >>>
> > >>> Gary
> > >>>
> > >>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
> > >>>
> > >>>> No. A Footer is only used at end of file. He needs to know how long each
> > >>>> event is or when it is the start of a new event.
> > >>>>
> > >>>> Ralph
> > >>>>
> > >>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
> > >>>> wrote:
> > >>>>>
> > >>>>> Can't you use a footer for any terminator you wish?
> > >>>>>
> > >>>>> Gary
> > >>>>>
> > >>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
> > >>>> wrote:
> > >>>>>
> > >>>>>> Hi.
> > >>>>>>
> > >>>>>> (Moving this discussion to logging dev mailing list.)
> > >>>>>>
> > >>>>>> Have you tried to use:
> > >>>>>>
> > >>>>>>
> > >>>>>> Then each log event will be terminated by end-of-line (\r\n).
> > >>>>>>
> > >>>>>> I think it would be easy to implement 0-byte terminated log events in
> > >>>>>> JsonLayout, and that would make sense since we have implemented support
> > >>>> for
> > >>>>>> that in GelfLayout. Created a JIRA issue for it:
> > >>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
> > >>>>>>
> > >>>>>> As for other tools, the only receivers for Log4j 2 SerializedLayout we
> > >>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
> > >>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
> > >>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
> > >>>> JsonLayout
> > >>>>>> by default.
> > >>>>>>
> > >>>>>>
> > >>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
> > >>>>>>
> > >>>>>>> Hi Mikael,
> > >>>>>>>
> > >>>>>>> I've just taken a look at the JsonLayout and see no way to parse its
> > >>>>>>> output event by event.
> > >>>>>>>
> > >>>>>>> It's semi-fine for files because then it will result in a well-formed
> > >>>>>>> array at the root level that can be read in one big swoop. It isn't
> > >>>>>>> really ideal for that use case either since it means that the whole
> > >>>>>>> array containing all events needs to be read into memory in one piece,
> > >>>>>>> causing OOM in case of huge files.
> > >>>>>>>
> > >>>>>>> But in Lilith, I really need to process events one by one. This is
> > >>>>>>> absolutely crucial.
> > >>>>>>>
> > >>>>>>> The only way to achieve this would essentially require me to
> > >>>>>>> re-implement a JSON parser that can cope with data like this.
> > >>>>>>>
> > >>>>>>> I implemented a similar kludge to support reading of log4j xml files
> > >>>>>>> (see
> > >>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
> > >>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java  
> > >>>>>>> ) but this was way easier since i could simply search for
> > >>>>>>> "" to find the position at which I could split the stream
> > >>>>>>> of events into a valid XML document for every single event . This was
> > >>>>>>> still kind of nasty but doable, especially since processing an
> > >>>> "offline"
> > >>>>>>> XML file doesn't have the same performance restriction as handling of
> > >>>>>>> live events.
> > >>>>>>>
> > >>>>>>> I wouldn't have the luxury of such a unique split signal in JSON,
> > >>>> though.
> > >>>>>>>
> > >>>>>>> I would need to detect the "}" at the level of the root array which
> > >>>>>>> would not simply involve counting of opening and closing brackets but
> > >>>>>>> also ignoring of brackets inside of strings. This would boil down to
> > >>>>>>> implementing a custom JSON reader and I won't be doing this.
> > >>>>>>>
> > >>>>>>> This isn't just me being lazy either. Such an implementation wouldn't
> > >>>> be
> > >>>>>>> resource (CPU/memory) friendly anyway.
> > >>>>>>>
> > >>>>>>> In my own JSON receiver using my own JSON format there are two ways to
> > >>>>>>> send those events:
> > >>>>>>> - write an int containing the amount of bytes representing the event,
> > >>>>>>> then the bytes of the event. This type of events also supports
> > >>>>>>> compression.
> > >>>>>>> - write the bytes of the event, followed by a 0-byte. This works fine
> > >>>>>>> and JavaScript/ActionScript is able to produce events like that while
> > >>>>>>> they are unable (or were unable 7 years ago) to count the bytes on
> > >>>> their
> > >>>>>>> own. This type of events only supports plain text like JSON or XML and
> > >>>>>>> no compression since compressed events could contain the 0-byte while
> > >>>>>>> XML and JSON both won't allow it.
> > >>>>>>>
> > >>>>>>> Both are essentially "binary" formats because of either the raw ints or
> > >>>>>>> the 0-bytes.
> > >>>>>>>
> > >>>>>>> I'm not sure why you deprecate SerializedLayout since the security
> > >>>>>>> issues arise only while deserializing the events, not while serializing
> > >>>>>>> them. Is this just meant to educate the user about the issue?
> > >>>>>>>
> > >>>>>>> I fixed the remote code execution exploit by implementing
> > >>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
> > >>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
> > >>>>>>> er/WhitelistObjectInputStream.java
> > >>>>>>>
> > >>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
> > >>>>>>> malicious data) but is close enough for me, mainly because fixing the
> > >>>>>>> DoS issues would really have to be handled by Java, not by user code.
> > >>>>>>> Manually re-implementing deserialization would likely be fragile and
> > >>>>>>> error prone, possibly even introducing other security issues on its
> > >>>> own.
> > >>>>>>>
> > >>>>>>> I'd be interested how other tools are tackling this issue. Is Chainsaw
> > >>>>>>> officially dead or are they implementing receivers for event streams
> > >>>>>>> like this?
> > >>>>>>>
> > >>>>>>> I'm totally fine with implementing a new receiver as a replacement for
> > >>>>>>> log4j2 SerializedLayout (even though I won't be able to get rid of the
> > >>>>>>> Serializable receiver anyway) but the current JsonLayout just isn't up
> > >>>>>>> to the job at the moment.
> > >>>>>>>
> > >>>>>>> Feel free to share this mail with your fellow developers. I'm open for
> > >>>>>>> suggestions.
> > >>>>>>>
> > >>>>>>> I hope this doesn't come across in a harsh way. It isn't meant to be.
> > >>>>>>>
> > >>>>>>> Cheers,
> > >>>>>>> Jörn.
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>
> > >>>>
> > >>>>
> > >>
> > >>
> > >>
> > >
> >
> >
>  
>  

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

I’ll give this a shot over the next view days.

Thanks!


On 19. July 2017 at 21:33:49, Mikael Ståldal (mikes@apache.org) wrote:
> JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte
> termination of log events. Will be part of the upcoming 2.9 release.
>  
>  
> On 2017-07-18 11:48, Jörn Huxhorn wrote:
> > A general event header and footer would probably be overkill.
> >
> > I’m using the following interface in Lilith:
> >
> > public interface WriteByteStrategy {
> > void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
> > throws IOException;
> > }
> >
> > One implementation writes an int with the length of bytes before “bytes", the other  
> implementation writes a 0-byte after “bytes".
> >
> > The problem with that approach is that it requires the intermediate “bytes” array and  
> is thus probably incompatible with your zero garbage approach.
> >
> > I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this with you.  
> I would also have to implement a new receiver type whereas I could just reuse an existing  
> one in case of 0-byte termination.
> >
> > I’d prefer using 0-byte termination since it would be less fragile than end-of-line.  
> The end-of-line approach would break if compact=“false” was set erroneously while  
> you’d still have the option of using either compact=“true” or compact=“false” in case  
> of 0-byte termination. 0-byte termination is also slightly more efficient while reading  
> since the splitting can operate on bytes instead of chars or Strings - another reason  
> why I didn’t just implement it and instead wanted to discuss this with you first.
> >
> > Joern
> >
> >
> > On 18. July 2017 at 03:26:32, Ralph Goers (ralph.goers@dslextreme.com) wrote:
> >> Right now that is all handled by the specific layouts. For example, by default the RFC5424Layout  
> >> doesn’t append newlines so when writing to a file they will all be on the same “line”,  
> but
> >> it has an option to append one if needed. Doing the same would be another option for the  
> >> JSONLayout.
> >>
> >> I’d be ok with event header and footers but ONLY if they have zero overhead when not present.  
> >> IOW, when the Layout is initialized it would have to wire in the appropriate encode  
> method.
> >>
> >> Ralph
> >>
> >>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
> >>>
> >>> Do we want a general event header and footer then?
> >>>
> >>> Gary
> >>>
> >>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
> >>>
> >>>> No. A Footer is only used at end of file. He needs to know how long each
> >>>> event is or when it is the start of a new event.
> >>>>
> >>>> Ralph
> >>>>
> >>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
> >>>> wrote:
> >>>>>
> >>>>> Can't you use a footer for any terminator you wish?
> >>>>>
> >>>>> Gary
> >>>>>
> >>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
> >>>> wrote:
> >>>>>
> >>>>>> Hi.
> >>>>>>
> >>>>>> (Moving this discussion to logging dev mailing list.)
> >>>>>>
> >>>>>> Have you tried to use:
> >>>>>>
> >>>>>>
> >>>>>> Then each log event will be terminated by end-of-line (\r\n).
> >>>>>>
> >>>>>> I think it would be easy to implement 0-byte terminated log events in
> >>>>>> JsonLayout, and that would make sense since we have implemented support
> >>>> for
> >>>>>> that in GelfLayout. Created a JIRA issue for it:
> >>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
> >>>>>>
> >>>>>> As for other tools, the only receivers for Log4j 2 SerializedLayout we
> >>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
> >>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
> >>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
> >>>> JsonLayout
> >>>>>> by default.
> >>>>>>
> >>>>>>
> >>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
> >>>>>>
> >>>>>>> Hi Mikael,
> >>>>>>>
> >>>>>>> I've just taken a look at the JsonLayout and see no way to parse its
> >>>>>>> output event by event.
> >>>>>>>
> >>>>>>> It's semi-fine for files because then it will result in a well-formed
> >>>>>>> array at the root level that can be read in one big swoop. It isn't
> >>>>>>> really ideal for that use case either since it means that the whole
> >>>>>>> array containing all events needs to be read into memory in one piece,
> >>>>>>> causing OOM in case of huge files.
> >>>>>>>
> >>>>>>> But in Lilith, I really need to process events one by one. This is
> >>>>>>> absolutely crucial.
> >>>>>>>
> >>>>>>> The only way to achieve this would essentially require me to
> >>>>>>> re-implement a JSON parser that can cope with data like this.
> >>>>>>>
> >>>>>>> I implemented a similar kludge to support reading of log4j xml files
> >>>>>>> (see
> >>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
> >>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java  
> >>>>>>> ) but this was way easier since i could simply search for
> >>>>>>> "" to find the position at which I could split the stream
> >>>>>>> of events into a valid XML document for every single event . This was
> >>>>>>> still kind of nasty but doable, especially since processing an
> >>>> "offline"
> >>>>>>> XML file doesn't have the same performance restriction as handling of
> >>>>>>> live events.
> >>>>>>>
> >>>>>>> I wouldn't have the luxury of such a unique split signal in JSON,
> >>>> though.
> >>>>>>>
> >>>>>>> I would need to detect the "}" at the level of the root array which
> >>>>>>> would not simply involve counting of opening and closing brackets but
> >>>>>>> also ignoring of brackets inside of strings. This would boil down to
> >>>>>>> implementing a custom JSON reader and I won't be doing this.
> >>>>>>>
> >>>>>>> This isn't just me being lazy either. Such an implementation wouldn't
> >>>> be
> >>>>>>> resource (CPU/memory) friendly anyway.
> >>>>>>>
> >>>>>>> In my own JSON receiver using my own JSON format there are two ways to
> >>>>>>> send those events:
> >>>>>>> - write an int containing the amount of bytes representing the event,
> >>>>>>> then the bytes of the event. This type of events also supports
> >>>>>>> compression.
> >>>>>>> - write the bytes of the event, followed by a 0-byte. This works fine
> >>>>>>> and JavaScript/ActionScript is able to produce events like that while
> >>>>>>> they are unable (or were unable 7 years ago) to count the bytes on
> >>>> their
> >>>>>>> own. This type of events only supports plain text like JSON or XML and
> >>>>>>> no compression since compressed events could contain the 0-byte while
> >>>>>>> XML and JSON both won't allow it.
> >>>>>>>
> >>>>>>> Both are essentially "binary" formats because of either the raw ints or
> >>>>>>> the 0-bytes.
> >>>>>>>
> >>>>>>> I'm not sure why you deprecate SerializedLayout since the security
> >>>>>>> issues arise only while deserializing the events, not while serializing
> >>>>>>> them. Is this just meant to educate the user about the issue?
> >>>>>>>
> >>>>>>> I fixed the remote code execution exploit by implementing
> >>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
> >>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
> >>>>>>> er/WhitelistObjectInputStream.java
> >>>>>>>
> >>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
> >>>>>>> malicious data) but is close enough for me, mainly because fixing the
> >>>>>>> DoS issues would really have to be handled by Java, not by user code.
> >>>>>>> Manually re-implementing deserialization would likely be fragile and
> >>>>>>> error prone, possibly even introducing other security issues on its
> >>>> own.
> >>>>>>>
> >>>>>>> I'd be interested how other tools are tackling this issue. Is Chainsaw
> >>>>>>> officially dead or are they implementing receivers for event streams
> >>>>>>> like this?
> >>>>>>>
> >>>>>>> I'm totally fine with implementing a new receiver as a replacement for
> >>>>>>> log4j2 SerializedLayout (even though I won't be able to get rid of the
> >>>>>>> Serializable receiver anyway) but the current JsonLayout just isn't up
> >>>>>>> to the job at the moment.
> >>>>>>>
> >>>>>>> Feel free to share this mail with your fellow developers. I'm open for
> >>>>>>> suggestions.
> >>>>>>>
> >>>>>>> I hope this doesn't come across in a harsh way. It isn't meant to be.
> >>>>>>>
> >>>>>>> Cheers,
> >>>>>>> Jörn.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>
> >>
> >>
> >
>  
>  

Re: Lilith and Log4j 2

[Mikael Ståldal <mi...@apache.org>]

JsonLayout (and XmlLayout and YamlLayout) now supports 0-byte 
termination of log events. Will be part of the upcoming 2.9 release.


On 2017-07-18 11:48, Jörn Huxhorn wrote:
> A general event header and footer would probably be overkill.
> 
> I’m using the following interface in Lilith:
> 
> public interface WriteByteStrategy {
>      void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
>          throws IOException;
> }
> 
> One implementation writes an int with the length of bytes before “bytes", the other implementation writes a 0-byte after “bytes".
> 
> The problem with that approach is that it requires the intermediate “bytes” array and is thus probably incompatible with your zero garbage approach.
> 
> I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this with you. I would also have to implement a new receiver type whereas I could just reuse an existing one in case of 0-byte termination.
> 
> I’d prefer using 0-byte termination since it would be less fragile than end-of-line. The end-of-line approach would break if compact=“false” was set erroneously while you’d still have the option of using either compact=“true” or compact=“false” in case of 0-byte termination. 0-byte termination is also slightly more efficient while reading since the splitting can operate on bytes instead of chars or Strings - another reason why I didn’t just implement it and instead wanted to discuss this with you first.
> 
> Joern
> 
> 
> On 18. July 2017 at 03:26:32, Ralph Goers (ralph.goers@dslextreme.com) wrote:
>> Right now that is all handled by the specific layouts. For example, by default the RFC5424Layout
>> doesn’t append newlines so when writing to a file they will all be on the same “line”, but
>> it has an option to append one if needed. Doing the same would be another option for the
>> JSONLayout.
>>   
>> I’d be ok with event header and footers but ONLY if they have zero overhead when not present.
>> IOW, when the Layout is initialized it would have to wire in the appropriate encode method.
>>   
>> Ralph
>>   
>>> On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
>>>
>>> Do we want a general event header and footer then?
>>>
>>> Gary
>>>
>>> On Jul 17, 2017 14:43, "Ralph Goers" wrote:
>>>
>>>> No. A Footer is only used at end of file. He needs to know how long each
>>>> event is or when it is the start of a new event.
>>>>
>>>> Ralph
>>>>
>>>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory
>>>> wrote:
>>>>>
>>>>> Can't you use a footer for any terminator you wish?
>>>>>
>>>>> Gary
>>>>>
>>>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal
>>>> wrote:
>>>>>
>>>>>> Hi.
>>>>>>
>>>>>> (Moving this discussion to logging dev mailing list.)
>>>>>>
>>>>>> Have you tried to use:
>>>>>>   
>>>>>>
>>>>>> Then each log event will be terminated by end-of-line (\r\n).
>>>>>>
>>>>>> I think it would be easy to implement 0-byte terminated log events in
>>>>>> JsonLayout, and that would make sense since we have implemented support
>>>> for
>>>>>> that in GelfLayout. Created a JIRA issue for it:
>>>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
>>>>>>
>>>>>> As for other tools, the only receivers for Log4j 2 SerializedLayout we
>>>>>> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
>>>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
>>>>>> JsonLayout and XmlLayout as well, and we are changing it to use
>>>> JsonLayout
>>>>>> by default.
>>>>>>
>>>>>>
>>>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
>>>>>>
>>>>>>> Hi Mikael,
>>>>>>>
>>>>>>> I've just taken a look at the JsonLayout and see no way to parse its
>>>>>>> output event by event.
>>>>>>>
>>>>>>> It's semi-fine for files because then it will result in a well-formed
>>>>>>> array at the root level that can be read in one big swoop. It isn't
>>>>>>> really ideal for that use case either since it means that the whole
>>>>>>> array containing all events needs to be read into memory in one piece,
>>>>>>> causing OOM in case of huge files.
>>>>>>>
>>>>>>> But in Lilith, I really need to process events one by one. This is
>>>>>>> absolutely crucial.
>>>>>>>
>>>>>>> The only way to achieve this would essentially require me to
>>>>>>> re-implement a JSON parser that can cope with data like this.
>>>>>>>
>>>>>>> I implemented a similar kludge to support reading of log4j xml files
>>>>>>> (see
>>>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
>>>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
>>>>>>> ) but this was way easier since i could simply search for
>>>>>>> "" to find the position at which I could split the stream
>>>>>>> of events into a valid XML document for every single event . This was
>>>>>>> still kind of nasty but doable, especially since processing an
>>>> "offline"
>>>>>>> XML file doesn't have the same performance restriction as handling of
>>>>>>> live events.
>>>>>>>
>>>>>>> I wouldn't have the luxury of such a unique split signal in JSON,
>>>> though.
>>>>>>>
>>>>>>> I would need to detect the "}" at the level of the root array which
>>>>>>> would not simply involve counting of opening and closing brackets but
>>>>>>> also ignoring of brackets inside of strings. This would boil down to
>>>>>>> implementing a custom JSON reader and I won't be doing this.
>>>>>>>
>>>>>>> This isn't just me being lazy either. Such an implementation wouldn't
>>>> be
>>>>>>> resource (CPU/memory) friendly anyway.
>>>>>>>
>>>>>>> In my own JSON receiver using my own JSON format there are two ways to
>>>>>>> send those events:
>>>>>>> - write an int containing the amount of bytes representing the event,
>>>>>>> then the bytes of the event. This type of events also supports
>>>>>>> compression.
>>>>>>> - write the bytes of the event, followed by a 0-byte. This works fine
>>>>>>> and JavaScript/ActionScript is able to produce events like that while
>>>>>>> they are unable (or were unable 7 years ago) to count the bytes on
>>>> their
>>>>>>> own. This type of events only supports plain text like JSON or XML and
>>>>>>> no compression since compressed events could contain the 0-byte while
>>>>>>> XML and JSON both won't allow it.
>>>>>>>
>>>>>>> Both are essentially "binary" formats because of either the raw ints or
>>>>>>> the 0-bytes.
>>>>>>>
>>>>>>> I'm not sure why you deprecate SerializedLayout since the security
>>>>>>> issues arise only while deserializing the events, not while serializing
>>>>>>> them. Is this just meant to educate the user about the issue?
>>>>>>>
>>>>>>> I fixed the remote code execution exploit by implementing
>>>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
>>>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
>>>>>>> er/WhitelistObjectInputStream.java
>>>>>>>
>>>>>>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
>>>>>>> malicious data) but is close enough for me, mainly because fixing the
>>>>>>> DoS issues would really have to be handled by Java, not by user code.
>>>>>>> Manually re-implementing deserialization would likely be fragile and
>>>>>>> error prone, possibly even introducing other security issues on its
>>>> own.
>>>>>>>
>>>>>>> I'd be interested how other tools are tackling this issue. Is Chainsaw
>>>>>>> officially dead or are they implementing receivers for event streams
>>>>>>> like this?
>>>>>>>
>>>>>>> I'm totally fine with implementing a new receiver as a replacement for
>>>>>>> log4j2 SerializedLayout (even though I won't be able to get rid of the
>>>>>>> Serializable receiver anyway) but the current JsonLayout just isn't up
>>>>>>> to the job at the moment.
>>>>>>>
>>>>>>> Feel free to share this mail with your fellow developers. I'm open for
>>>>>>> suggestions.
>>>>>>>
>>>>>>> I hope this doesn't come across in a harsh way. It isn't meant to be.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Jörn.
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>   
>>   
>>   
> 

Re: Lilith and Log4j 2

[Jörn Huxhorn <jh...@googlemail.com>]

A general event header and footer would probably be overkill.

I’m using the following interface in Lilith:

public interface WriteByteStrategy {
    void writeBytes(DataOutputStream dataOutputStream, byte[] bytes)
        throws IOException;
}

One implementation writes an int with the length of bytes before “bytes", the other implementation writes a 0-byte after “bytes".

The problem with that approach is that it requires the intermediate “bytes” array and is thus probably incompatible with your zero garbage approach.

I didn’t try end-of-line (\r\n) mainly because I first wanted to discuss this with you. I would also have to implement a new receiver type whereas I could just reuse an existing one in case of 0-byte termination.

I’d prefer using 0-byte termination since it would be less fragile than end-of-line. The end-of-line approach would break if compact=“false” was set erroneously while you’d still have the option of using either compact=“true” or compact=“false” in case of 0-byte termination. 0-byte termination is also slightly more efficient while reading since the splitting can operate on bytes instead of chars or Strings - another reason why I didn’t just implement it and instead wanted to discuss this with you first.

Joern


On 18. July 2017 at 03:26:32, Ralph Goers (ralph.goers@dslextreme.com) wrote:
> Right now that is all handled by the specific layouts. For example, by default the RFC5424Layout  
> doesn’t append newlines so when writing to a file they will all be on the same “line”, but  
> it has an option to append one if needed. Doing the same would be another option for the  
> JSONLayout.
>  
> I’d be ok with event header and footers but ONLY if they have zero overhead when not present.  
> IOW, when the Layout is initialized it would have to wire in the appropriate encode method.  
>  
> Ralph
>  
> > On Jul 17, 2017, at 3:06 PM, Gary Gregory wrote:
> >
> > Do we want a general event header and footer then?
> >
> > Gary
> >
> > On Jul 17, 2017 14:43, "Ralph Goers" wrote:
> >
> >> No. A Footer is only used at end of file. He needs to know how long each
> >> event is or when it is the start of a new event.
> >>
> >> Ralph
> >>
> >>> On Jul 17, 2017, at 12:32 PM, Gary Gregory  
> >> wrote:
> >>>
> >>> Can't you use a footer for any terminator you wish?
> >>>
> >>> Gary
> >>>
> >>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal  
> >> wrote:
> >>>
> >>>> Hi.
> >>>>
> >>>> (Moving this discussion to logging dev mailing list.)
> >>>>
> >>>> Have you tried to use:
> >>>>  
> >>>>
> >>>> Then each log event will be terminated by end-of-line (\r\n).
> >>>>
> >>>> I think it would be easy to implement 0-byte terminated log events in
> >>>> JsonLayout, and that would make sense since we have implemented support
> >> for
> >>>> that in GelfLayout. Created a JIRA issue for it:
> >>>> https://issues.apache.org/jira/browse/LOG4J2-1981
> >>>>
> >>>> As for other tools, the only receivers for Log4j 2 SerializedLayout we
> >>>> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
> >>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
> >>>> JsonLayout and XmlLayout as well, and we are changing it to use
> >> JsonLayout
> >>>> by default.
> >>>>
> >>>>
> >>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
> >>>>
> >>>>> Hi Mikael,
> >>>>>
> >>>>> I've just taken a look at the JsonLayout and see no way to parse its
> >>>>> output event by event.
> >>>>>
> >>>>> It's semi-fine for files because then it will result in a well-formed
> >>>>> array at the root level that can be read in one big swoop. It isn't
> >>>>> really ideal for that use case either since it means that the whole
> >>>>> array containing all events needs to be read into memory in one piece,
> >>>>> causing OOM in case of huge files.
> >>>>>
> >>>>> But in Lilith, I really need to process events one by one. This is
> >>>>> absolutely crucial.
> >>>>>
> >>>>> The only way to achieve this would essentially require me to
> >>>>> re-implement a JSON parser that can cope with data like this.
> >>>>>
> >>>>> I implemented a similar kludge to support reading of log4j xml files
> >>>>> (see
> >>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
> >>>>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java  
> >>>>> ) but this was way easier since i could simply search for
> >>>>> "" to find the position at which I could split the stream
> >>>>> of events into a valid XML document for every single event . This was
> >>>>> still kind of nasty but doable, especially since processing an
> >> "offline"
> >>>>> XML file doesn't have the same performance restriction as handling of
> >>>>> live events.
> >>>>>
> >>>>> I wouldn't have the luxury of such a unique split signal in JSON,
> >> though.
> >>>>>
> >>>>> I would need to detect the "}" at the level of the root array which
> >>>>> would not simply involve counting of opening and closing brackets but
> >>>>> also ignoring of brackets inside of strings. This would boil down to
> >>>>> implementing a custom JSON reader and I won't be doing this.
> >>>>>
> >>>>> This isn't just me being lazy either. Such an implementation wouldn't
> >> be
> >>>>> resource (CPU/memory) friendly anyway.
> >>>>>
> >>>>> In my own JSON receiver using my own JSON format there are two ways to
> >>>>> send those events:
> >>>>> - write an int containing the amount of bytes representing the event,
> >>>>> then the bytes of the event. This type of events also supports
> >>>>> compression.
> >>>>> - write the bytes of the event, followed by a 0-byte. This works fine
> >>>>> and JavaScript/ActionScript is able to produce events like that while
> >>>>> they are unable (or were unable 7 years ago) to count the bytes on
> >> their
> >>>>> own. This type of events only supports plain text like JSON or XML and
> >>>>> no compression since compressed events could contain the 0-byte while
> >>>>> XML and JSON both won't allow it.
> >>>>>
> >>>>> Both are essentially "binary" formats because of either the raw ints or
> >>>>> the 0-bytes.
> >>>>>
> >>>>> I'm not sure why you deprecate SerializedLayout since the security
> >>>>> issues arise only while deserializing the events, not while serializing
> >>>>> them. Is this just meant to educate the user about the issue?
> >>>>>
> >>>>> I fixed the remote code execution exploit by implementing
> >>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
> >>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
> >>>>> er/WhitelistObjectInputStream.java
> >>>>>
> >>>>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
> >>>>> malicious data) but is close enough for me, mainly because fixing the
> >>>>> DoS issues would really have to be handled by Java, not by user code.
> >>>>> Manually re-implementing deserialization would likely be fragile and
> >>>>> error prone, possibly even introducing other security issues on its
> >> own.
> >>>>>
> >>>>> I'd be interested how other tools are tackling this issue. Is Chainsaw
> >>>>> officially dead or are they implementing receivers for event streams
> >>>>> like this?
> >>>>>
> >>>>> I'm totally fine with implementing a new receiver as a replacement for
> >>>>> log4j2 SerializedLayout (even though I won't be able to get rid of the
> >>>>> Serializable receiver anyway) but the current JsonLayout just isn't up
> >>>>> to the job at the moment.
> >>>>>
> >>>>> Feel free to share this mail with your fellow developers. I'm open for
> >>>>> suggestions.
> >>>>>
> >>>>> I hope this doesn't come across in a harsh way. It isn't meant to be.
> >>>>>
> >>>>> Cheers,
> >>>>> Jörn.
> >>>>>
> >>>>
> >>>>
> >>
> >>
> >>
>  
>  
>  

Re: Lilith and Log4j 2

[Ralph Goers <ra...@dslextreme.com>]

Right now that is all handled by the specific layouts. For example, by default the RFC5424Layout doesn’t append newlines so when writing to a file they will all be on the same “line”, but it has an option to append one if needed. Doing the same would be another option for the JSONLayout.

I’d be ok with event header and footers but ONLY if they have zero overhead when not present. IOW, when the Layout is initialized it would have to wire in the appropriate encode method.

Ralph

> On Jul 17, 2017, at 3:06 PM, Gary Gregory <ga...@gmail.com> wrote:
> 
> Do we want a general event header and footer then?
> 
> Gary
> 
> On Jul 17, 2017 14:43, "Ralph Goers" <ra...@dslextreme.com> wrote:
> 
>> No. A Footer is only used at end of file. He needs to know how long each
>> event is or when it is the start of a new event.
>> 
>> Ralph
>> 
>>> On Jul 17, 2017, at 12:32 PM, Gary Gregory <ga...@gmail.com>
>> wrote:
>>> 
>>> Can't you use a footer for any terminator you wish?
>>> 
>>> Gary
>>> 
>>> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal <mi...@apache.org>
>> wrote:
>>> 
>>>> Hi.
>>>> 
>>>> (Moving this discussion to logging dev mailing list.)
>>>> 
>>>> Have you tried to use:
>>>> <JsonLayout properties="true" eventEol="true" compact="true"/>
>>>> 
>>>> Then each log event will be terminated by end-of-line (\r\n).
>>>> 
>>>> I think it would be easy to implement 0-byte terminated log events in
>>>> JsonLayout, and that would make sense since we have implemented support
>> for
>>>> that in GelfLayout. Created a JIRA issue for it:
>>>> https://issues.apache.org/jira/browse/LOG4J2-1981
>>>> 
>>>> As for other tools, the only receivers for Log4j 2 SerializedLayout we
>>>> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
>>>> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
>>>> JsonLayout and XmlLayout as well, and we are changing it to use
>> JsonLayout
>>>> by default.
>>>> 
>>>> 
>>>> On 2017-07-17 15:01, Joern Huxhorn wrote:
>>>> 
>>>>> Hi Mikael,
>>>>> 
>>>>> I've just taken a look at the JsonLayout and see no way to parse its
>>>>> output event by event.
>>>>> 
>>>>> It's semi-fine for files because then it will result in a well-formed
>>>>> array at the root level that can be read in one big swoop. It isn't
>>>>> really ideal for that use case either since it means that the whole
>>>>> array containing all events needs to be read into memory in one piece,
>>>>> causing OOM in case of huge files.
>>>>> 
>>>>> But in Lilith, I really need to process events one by one. This is
>>>>> absolutely crucial.
>>>>> 
>>>>> The only way to achieve this would essentially require me to
>>>>> re-implement a JSON parser that can cope with data like this.
>>>>> 
>>>>> I implemented a similar kludge to support reading of log4j xml files
>>>>> (see
>>>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
>>>>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
>>>>> ) but this was way easier since i could simply search for
>>>>> "</log4j:event>" to find the position at which I could split the stream
>>>>> of events into a valid XML document for every single event . This was
>>>>> still kind of nasty but doable, especially since processing an
>> "offline"
>>>>> XML file doesn't have the same performance restriction as handling of
>>>>> live events.
>>>>> 
>>>>> I wouldn't have the luxury of such a unique split signal in JSON,
>> though.
>>>>> 
>>>>> I would need to detect the "}" at the level of the root array which
>>>>> would not simply involve counting of opening and closing brackets but
>>>>> also ignoring of brackets inside of strings. This would boil down to
>>>>> implementing a custom JSON reader and I won't be doing this.
>>>>> 
>>>>> This isn't just me being lazy either. Such an implementation wouldn't
>> be
>>>>> resource (CPU/memory) friendly anyway.
>>>>> 
>>>>> In my own JSON receiver using my own JSON format there are two ways to
>>>>> send those events:
>>>>> - write an int containing the amount of bytes representing the event,
>>>>> then the bytes of the event. This type of events also supports
>>>>> compression.
>>>>> - write the bytes of the event, followed by a 0-byte. This works fine
>>>>> and JavaScript/ActionScript is able to produce events like that while
>>>>> they are unable (or were unable 7 years ago) to count the bytes on
>> their
>>>>> own. This type of events only supports plain text like JSON or XML and
>>>>> no compression since compressed events could contain the 0-byte while
>>>>> XML and JSON both won't allow it.
>>>>> 
>>>>> Both are essentially "binary" formats because of either the raw ints or
>>>>> the 0-bytes.
>>>>> 
>>>>> I'm not sure why you deprecate SerializedLayout since the security
>>>>> issues arise only while deserializing the events, not while serializing
>>>>> them. Is this just meant to educate the user about the issue?
>>>>> 
>>>>> I fixed the remote code execution exploit by implementing
>>>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
>>>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
>>>>> er/WhitelistObjectInputStream.java
>>>>> 
>>>>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
>>>>> malicious data) but is close enough for me, mainly because fixing the
>>>>> DoS issues would really have to be handled by Java, not by user code.
>>>>> Manually re-implementing deserialization would likely be fragile and
>>>>> error prone, possibly even introducing other security issues on its
>> own.
>>>>> 
>>>>> I'd be interested how other tools are tackling this issue. Is Chainsaw
>>>>> officially dead or are they implementing receivers for event streams
>>>>> like this?
>>>>> 
>>>>> I'm totally fine with implementing a new receiver as a replacement for
>>>>> log4j2 SerializedLayout (even though I won't be able to get rid of the
>>>>> Serializable receiver anyway) but the current JsonLayout just isn't up
>>>>> to the job at the moment.
>>>>> 
>>>>> Feel free to share this mail with your fellow developers. I'm open for
>>>>> suggestions.
>>>>> 
>>>>> I hope this doesn't come across in a harsh way. It isn't meant to be.
>>>>> 
>>>>> Cheers,
>>>>> Jörn.
>>>>> 
>>>> 
>>>> 
>> 
>> 
>> 

Re: Lilith and Log4j 2

[Gary Gregory <ga...@gmail.com>]

Do we want a general event header and footer then?

Gary

On Jul 17, 2017 14:43, "Ralph Goers" <ra...@dslextreme.com> wrote:

> No. A Footer is only used at end of file. He needs to know how long each
> event is or when it is the start of a new event.
>
> Ralph
>
> > On Jul 17, 2017, at 12:32 PM, Gary Gregory <ga...@gmail.com>
> wrote:
> >
> > Can't you use a footer for any terminator you wish?
> >
> > Gary
> >
> > On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal <mi...@apache.org>
> wrote:
> >
> >> Hi.
> >>
> >> (Moving this discussion to logging dev mailing list.)
> >>
> >> Have you tried to use:
> >> <JsonLayout properties="true" eventEol="true" compact="true"/>
> >>
> >> Then each log event will be terminated by end-of-line (\r\n).
> >>
> >> I think it would be easy to implement 0-byte terminated log events in
> >> JsonLayout, and that would make sense since we have implemented support
> for
> >> that in GelfLayout. Created a JIRA issue for it:
> >> https://issues.apache.org/jira/browse/LOG4J2-1981
> >>
> >> As for other tools, the only receivers for Log4j 2 SerializedLayout we
> >> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
> >> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
> >> JsonLayout and XmlLayout as well, and we are changing it to use
> JsonLayout
> >> by default.
> >>
> >>
> >> On 2017-07-17 15:01, Joern Huxhorn wrote:
> >>
> >>> Hi Mikael,
> >>>
> >>> I've just taken a look at the JsonLayout and see no way to parse its
> >>> output event by event.
> >>>
> >>> It's semi-fine for files because then it will result in a well-formed
> >>> array at the root level that can be read in one big swoop. It isn't
> >>> really ideal for that use case either since it means that the whole
> >>> array containing all events needs to be read into memory in one piece,
> >>> causing OOM in case of huge files.
> >>>
> >>> But in Lilith, I really need to process events one by one. This is
> >>> absolutely crucial.
> >>>
> >>> The only way to achieve this would essentially require me to
> >>> re-implement a JSON parser that can cope with data like this.
> >>>
> >>> I implemented a similar kludge to support reading of log4j xml files
> >>> (see
> >>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
> >>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
> >>> ) but this was way easier since i could simply search for
> >>> "</log4j:event>" to find the position at which I could split the stream
> >>> of events into a valid XML document for every single event . This was
> >>> still kind of nasty but doable, especially since processing an
> "offline"
> >>> XML file doesn't have the same performance restriction as handling of
> >>> live events.
> >>>
> >>> I wouldn't have the luxury of such a unique split signal in JSON,
> though.
> >>>
> >>> I would need to detect the "}" at the level of the root array which
> >>> would not simply involve counting of opening and closing brackets but
> >>> also ignoring of brackets inside of strings. This would boil down to
> >>> implementing a custom JSON reader and I won't be doing this.
> >>>
> >>> This isn't just me being lazy either. Such an implementation wouldn't
> be
> >>> resource (CPU/memory) friendly anyway.
> >>>
> >>> In my own JSON receiver using my own JSON format there are two ways to
> >>> send those events:
> >>> - write an int containing the amount of bytes representing the event,
> >>> then the bytes of the event. This type of events also supports
> >>> compression.
> >>> - write the bytes of the event, followed by a 0-byte. This works fine
> >>> and JavaScript/ActionScript is able to produce events like that while
> >>> they are unable (or were unable 7 years ago) to count the bytes on
> their
> >>> own. This type of events only supports plain text like JSON or XML and
> >>> no compression since compressed events could contain the 0-byte while
> >>> XML and JSON both won't allow it.
> >>>
> >>> Both are essentially "binary" formats because of either the raw ints or
> >>> the 0-bytes.
> >>>
> >>> I'm not sure why you deprecate SerializedLayout since the security
> >>> issues arise only while deserializing the events, not while serializing
> >>> them. Is this just meant to educate the user about the issue?
> >>>
> >>> I fixed the remote code execution exploit by implementing
> >>> https://github.com/huxi/lilith/blob/master/lilith-engine/
> >>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
> >>> er/WhitelistObjectInputStream.java
> >>>
> >>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
> >>> malicious data) but is close enough for me, mainly because fixing the
> >>> DoS issues would really have to be handled by Java, not by user code.
> >>> Manually re-implementing deserialization would likely be fragile and
> >>> error prone, possibly even introducing other security issues on its
> own.
> >>>
> >>> I'd be interested how other tools are tackling this issue. Is Chainsaw
> >>> officially dead or are they implementing receivers for event streams
> >>> like this?
> >>>
> >>> I'm totally fine with implementing a new receiver as a replacement for
> >>> log4j2 SerializedLayout (even though I won't be able to get rid of the
> >>> Serializable receiver anyway) but the current JsonLayout just isn't up
> >>> to the job at the moment.
> >>>
> >>> Feel free to share this mail with your fellow developers. I'm open for
> >>> suggestions.
> >>>
> >>> I hope this doesn't come across in a harsh way. It isn't meant to be.
> >>>
> >>> Cheers,
> >>> Jörn.
> >>>
> >>
> >>
>
>
>

Re: Lilith and Log4j 2

[Ralph Goers <ra...@dslextreme.com>]

No. A Footer is only used at end of file. He needs to know how long each event is or when it is the start of a new event.

Ralph

> On Jul 17, 2017, at 12:32 PM, Gary Gregory <ga...@gmail.com> wrote:
> 
> Can't you use a footer for any terminator you wish?
> 
> Gary
> 
> On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal <mi...@apache.org> wrote:
> 
>> Hi.
>> 
>> (Moving this discussion to logging dev mailing list.)
>> 
>> Have you tried to use:
>> <JsonLayout properties="true" eventEol="true" compact="true"/>
>> 
>> Then each log event will be terminated by end-of-line (\r\n).
>> 
>> I think it would be easy to implement 0-byte terminated log events in
>> JsonLayout, and that would make sense since we have implemented support for
>> that in GelfLayout. Created a JIRA issue for it:
>> https://issues.apache.org/jira/browse/LOG4J2-1981
>> 
>> As for other tools, the only receivers for Log4j 2 SerializedLayout we
>> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
>> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
>> JsonLayout and XmlLayout as well, and we are changing it to use JsonLayout
>> by default.
>> 
>> 
>> On 2017-07-17 15:01, Joern Huxhorn wrote:
>> 
>>> Hi Mikael,
>>> 
>>> I've just taken a look at the JsonLayout and see no way to parse its
>>> output event by event.
>>> 
>>> It's semi-fine for files because then it will result in a well-formed
>>> array at the root level that can be read in one big swoop. It isn't
>>> really ideal for that use case either since it means that the whole
>>> array containing all events needs to be read into memory in one piece,
>>> causing OOM in case of huge files.
>>> 
>>> But in Lilith, I really need to process events one by one. This is
>>> absolutely crucial.
>>> 
>>> The only way to achieve this would essentially require me to
>>> re-implement a JSON parser that can cope with data like this.
>>> 
>>> I implemented a similar kludge to support reading of log4j xml files
>>> (see
>>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
>>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
>>> ) but this was way easier since i could simply search for
>>> "</log4j:event>" to find the position at which I could split the stream
>>> of events into a valid XML document for every single event . This was
>>> still kind of nasty but doable, especially since processing an "offline"
>>> XML file doesn't have the same performance restriction as handling of
>>> live events.
>>> 
>>> I wouldn't have the luxury of such a unique split signal in JSON, though.
>>> 
>>> I would need to detect the "}" at the level of the root array which
>>> would not simply involve counting of opening and closing brackets but
>>> also ignoring of brackets inside of strings. This would boil down to
>>> implementing a custom JSON reader and I won't be doing this.
>>> 
>>> This isn't just me being lazy either. Such an implementation wouldn't be
>>> resource (CPU/memory) friendly anyway.
>>> 
>>> In my own JSON receiver using my own JSON format there are two ways to
>>> send those events:
>>> - write an int containing the amount of bytes representing the event,
>>> then the bytes of the event. This type of events also supports
>>> compression.
>>> - write the bytes of the event, followed by a 0-byte. This works fine
>>> and JavaScript/ActionScript is able to produce events like that while
>>> they are unable (or were unable 7 years ago) to count the bytes on their
>>> own. This type of events only supports plain text like JSON or XML and
>>> no compression since compressed events could contain the 0-byte while
>>> XML and JSON both won't allow it.
>>> 
>>> Both are essentially "binary" formats because of either the raw ints or
>>> the 0-bytes.
>>> 
>>> I'm not sure why you deprecate SerializedLayout since the security
>>> issues arise only while deserializing the events, not while serializing
>>> them. Is this just meant to educate the user about the issue?
>>> 
>>> I fixed the remote code execution exploit by implementing
>>> https://github.com/huxi/lilith/blob/master/lilith-engine/
>>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
>>> er/WhitelistObjectInputStream.java
>>> 
>>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
>>> malicious data) but is close enough for me, mainly because fixing the
>>> DoS issues would really have to be handled by Java, not by user code.
>>> Manually re-implementing deserialization would likely be fragile and
>>> error prone, possibly even introducing other security issues on its own.
>>> 
>>> I'd be interested how other tools are tackling this issue. Is Chainsaw
>>> officially dead or are they implementing receivers for event streams
>>> like this?
>>> 
>>> I'm totally fine with implementing a new receiver as a replacement for
>>> log4j2 SerializedLayout (even though I won't be able to get rid of the
>>> Serializable receiver anyway) but the current JsonLayout just isn't up
>>> to the job at the moment.
>>> 
>>> Feel free to share this mail with your fellow developers. I'm open for
>>> suggestions.
>>> 
>>> I hope this doesn't come across in a harsh way. It isn't meant to be.
>>> 
>>> Cheers,
>>> Jörn.
>>> 
>> 
>> 

Re: Lilith and Log4j 2

[Gary Gregory <ga...@gmail.com>]

Can't you use a footer for any terminator you wish?

Gary

On Mon, Jul 17, 2017 at 12:13 PM, Mikael Ståldal <mi...@apache.org> wrote:

> Hi.
>
> (Moving this discussion to logging dev mailing list.)
>
> Have you tried to use:
> <JsonLayout properties="true" eventEol="true" compact="true"/>
>
> Then each log event will be terminated by end-of-line (\r\n).
>
> I think it would be easy to implement 0-byte terminated log events in
> JsonLayout, and that would make sense since we have implemented support for
> that in GelfLayout. Created a JIRA issue for it:
> https://issues.apache.org/jira/browse/LOG4J2-1981
>
> As for other tools, the only receivers for Log4j 2 SerializedLayout we
> know are Log4j's own SocketServer and Lilith. Chainsaw currently only
> support Log4j 1 SerializedLayout. Log4j's own SocketServer support
> JsonLayout and XmlLayout as well, and we are changing it to use JsonLayout
> by default.
>
>
> On 2017-07-17 15:01, Joern Huxhorn wrote:
>
>> Hi Mikael,
>>
>> I've just taken a look at the JsonLayout and see no way to parse its
>> output event by event.
>>
>> It's semi-fine for files because then it will result in a well-formed
>> array at the root level that can be read in one big swoop. It isn't
>> really ideal for that use case either since it means that the whole
>> array containing all events needs to be read into memory in one piece,
>> causing OOM in case of huge files.
>>
>> But in Lilith, I really need to process events one by one. This is
>> absolutely crucial.
>>
>> The only way to achieve this would essentially require me to
>> re-implement a JSON parser that can cope with data like this.
>>
>> I implemented a similar kludge to support reading of log4j xml files
>> (see
>> https://github.com/huxi/lilith/blob/master/log4j/log4j-xml/
>> src/main/java/de/huxhorn/lilith/log4j/xml/Log4jImportCallable.java
>> ) but this was way easier since i could simply search for
>> "</log4j:event>" to find the position at which I could split the stream
>> of events into a valid XML document for every single event . This was
>> still kind of nasty but doable, especially since processing an "offline"
>> XML file doesn't have the same performance restriction as handling of
>> live events.
>>
>> I wouldn't have the luxury of such a unique split signal in JSON, though.
>>
>> I would need to detect the "}" at the level of the root array which
>> would not simply involve counting of opening and closing brackets but
>> also ignoring of brackets inside of strings. This would boil down to
>> implementing a custom JSON reader and I won't be doing this.
>>
>> This isn't just me being lazy either. Such an implementation wouldn't be
>> resource (CPU/memory) friendly anyway.
>>
>> In my own JSON receiver using my own JSON format there are two ways to
>> send those events:
>> - write an int containing the amount of bytes representing the event,
>> then the bytes of the event. This type of events also supports
>> compression.
>> - write the bytes of the event, followed by a 0-byte. This works fine
>> and JavaScript/ActionScript is able to produce events like that while
>> they are unable (or were unable 7 years ago) to count the bytes on their
>> own. This type of events only supports plain text like JSON or XML and
>> no compression since compressed events could contain the 0-byte while
>> XML and JSON both won't allow it.
>>
>> Both are essentially "binary" formats because of either the raw ints or
>> the 0-bytes.
>>
>> I'm not sure why you deprecate SerializedLayout since the security
>> issues arise only while deserializing the events, not while serializing
>> them. Is this just meant to educate the user about the issue?
>>
>> I fixed the remote code execution exploit by implementing
>> https://github.com/huxi/lilith/blob/master/lilith-engine/
>> src/main/java/de/huxhorn/lilith/engine/impl/eventproduc
>> er/WhitelistObjectInputStream.java
>>
>> This still doesn't fix DoS scenarios (endless loops or OOM in case of
>> malicious data) but is close enough for me, mainly because fixing the
>> DoS issues would really have to be handled by Java, not by user code.
>> Manually re-implementing deserialization would likely be fragile and
>> error prone, possibly even introducing other security issues on its own.
>>
>> I'd be interested how other tools are tackling this issue. Is Chainsaw
>> officially dead or are they implementing receivers for event streams
>> like this?
>>
>> I'm totally fine with implementing a new receiver as a replacement for
>> log4j2 SerializedLayout (even though I won't be able to get rid of the
>> Serializable receiver anyway) but the current JsonLayout just isn't up
>> to the job at the moment.
>>
>> Feel free to share this mail with your fellow developers. I'm open for
>> suggestions.
>>
>> I hope this doesn't come across in a harsh way. It isn't meant to be.
>>
>> Cheers,
>> Jörn.
>>
>
>