You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@logging.apache.org by Volkan Yazıcı <vo...@gmail.com> on 2019/12/23 22:17:05 UTC
Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Hello,
As the author of log4j2-logstash-layout[1], I would like to share some
thoughts on the plugin and its implications on Log4j 3.0 roadmap.
- LogstashLayout is pretty stable in its feature set for the
last year. That is, the advantages (customization, etc.)
it offers over JsonLayout could very well be a guideline
for the next generation of JsonLayout.
- The demand for LogstashLayout's (almost) garbage-free
nature in combination with the efficiency of initialization-time
compiled schema showed that people are willing to employ
logging in unanticipated domains. In particular, Nexia Home[2]
is leveraging[3] the plugin for distributed tracing. Hence,
performance matters, AFAIC.
- Getting design right in terms of efficiency, can even yield
advantage over future competitors. See how LogstashLayout
beats the official Elastic Log4j 2.0 plugin for ECS[4].
Ralph already delivers a tremendous amount of work for the project. If
he or anyone else shares the blueprints in their minds for the next
generation of {Xml/Json,Yaml}Layout, I would be more than happy to
volunteer for the discussion and, more importantly, implementation.
Best.
[1] https://github.com/vy/log4j2-logstash-layout
[2] https://www.nexiahome.com/
[3] https://github.com/vy/log4j2-logstash-layout/issues/17
[4] https://github.com/vy/log4j2-logstash-layout#performance
On Mon, Dec 23, 2019 at 12:24 AM Ralph Goers <ra...@dslextreme.com> wrote:
> I am not particularly happy with the JsonLayout at the
> moment as I think what it does is too limited. I’d like there
> to be a lot more flexibility as to what the resulting JSON
> should look like, not just dumping the log event.
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Ralph Goers <ra...@dslextreme.com>.
> On Dec 23, 2019, at 7:50 PM, Gary Gregory <ga...@gmail.com> wrote:
>
> On Mon, Dec 23, 2019 at 6:25 PM Ralph Goers <ra...@dslextreme.com>
> wrote:
>
>> Thanks,
>>
>> What I was thinking was that I would just like a more flexible way to
>> specify what should be included in the JSON. In its current form the JSON
>> Layout just creates a serialized version of a LogEvent. I would prefer it
>> if users could define the individual fields they want and what they should
>> contain.
>>
>
> It seems like it would make sense to do the same for the XML and HTML
> layouts.
>
Maybe. My concern is primarily for JSON because that is what the majority of people are using to send logs to aggregators and ElasticSearch these days. That said, I found that after the modifications I made to GelfLayout it worked perfectly when sending logs to an ELK stack.
Ralph
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Gary Gregory <ga...@gmail.com>.
On Mon, Dec 23, 2019 at 6:25 PM Ralph Goers <ra...@dslextreme.com>
wrote:
> Thanks,
>
> What I was thinking was that I would just like a more flexible way to
> specify what should be included in the JSON. In its current form the JSON
> Layout just creates a serialized version of a LogEvent. I would prefer it
> if users could define the individual fields they want and what they should
> contain.
>
It seems like it would make sense to do the same for the XML and HTML
layouts.
Gary
> Ralph
>
>
> > On Dec 23, 2019, at 3:17 PM, Volkan Yazıcı <vo...@gmail.com>
> wrote:
> >
> > Hello,
> >
> > As the author of log4j2-logstash-layout[1], I would like to share some
> > thoughts on the plugin and its implications on Log4j 3.0 roadmap.
> >
> > - LogstashLayout is pretty stable in its feature set for the
> > last year. That is, the advantages (customization, etc.)
> > it offers over JsonLayout could very well be a guideline
> > for the next generation of JsonLayout.
> >
> > - The demand for LogstashLayout's (almost) garbage-free
> > nature in combination with the efficiency of initialization-time
> > compiled schema showed that people are willing to employ
> > logging in unanticipated domains. In particular, Nexia Home[2]
> > is leveraging[3] the plugin for distributed tracing. Hence,
> > performance matters, AFAIC.
> >
> > - Getting design right in terms of efficiency, can even yield
> > advantage over future competitors. See how LogstashLayout
> > beats the official Elastic Log4j 2.0 plugin for ECS[4].
> >
> > Ralph already delivers a tremendous amount of work for the project. If
> > he or anyone else shares the blueprints in their minds for the next
> > generation of {Xml/Json,Yaml}Layout, I would be more than happy to
> > volunteer for the discussion and, more importantly, implementation.
> >
> > Best.
> >
> > [1] https://github.com/vy/log4j2-logstash-layout
> > [2] https://www.nexiahome.com/
> > [3] https://github.com/vy/log4j2-logstash-layout/issues/17
> > [4] https://github.com/vy/log4j2-logstash-layout#performance
> >
> > On Mon, Dec 23, 2019 at 12:24 AM Ralph Goers <ra...@dslextreme.com>
> wrote:
> >> I am not particularly happy with the JsonLayout at the
> >> moment as I think what it does is too limited. I’d like there
> >> to be a lot more flexibility as to what the resulting JSON
> >> should look like, not just dumping the log event.
> >
>
>
>
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Ralph Goers <ra...@dslextreme.com>.
I would integrate it as is. Once it is here you can continue the work on converting it to use templates or create an entirely new Layout.
Ralph
> On Jan 9, 2020, at 6:28 AM, Volkan Yazıcı <vo...@gmail.com> wrote:
>
> Yes. I had the intention that you first wanted to see if a generic
> text-based templating solution would work, since you were the one
> proposing this idea in the first place. Would you rather first see the
> current LogstashLayout replacing JsonLayout in log4j-core? If so, I
> can first focus on integrating LogstashLayout into log4j-core.
>
> On Thu, Jan 9, 2020 at 1:19 AM Ralph Goers <ra...@dslextreme.com> wrote:
>>
>> Are you still considering contributing this?
>>
>> Ralph
>>
>>> On Jan 8, 2020, at 2:42 PM, Volkan Yazıcı <vo...@gmail.com> wrote:
>>>
>>> On Tue, Dec 24, 2019 at 10:48 PM Volkan Yazıcı <vo...@gmail.com> wrote:
>>>> In its current form, LogstashLayout can (almost) very well render the
>>>> GELF layout[1]. There are only two shortcomings: 1) epoch seconds with
>>>> fraction and 2) using PatternLayout while rendering messages. But no
>>>> worries, I will address them[2,3] in a couple of days.
>>>
>>> I have finally released log4j2-logstash-layout v1.0.0[1], which
>>> includes GELF layout[2]. Exposing epoch seconds of type double in the
>>> JSON template variables as efficiently as Log4j 2.0 GelfLayout turned
>>> out to be a tougher cookie than I was anticipating. But I can't
>>> complain, thanks to the feature, I've stepped into the Jackson hall of
>>> fame[3,4]. For those interested in, LogstashLayout is *still* the
>>> fastest JSON layout and I've updated the benchmark[5] accordingly.
>>>
>>> In my next adventure, I will see if I can beat PatternLayout with a
>>> generic template engine, e.g., Rocker[6]. If that experiment turns out
>>> to be positive, I will extend it with JsonLayout.
>>>
>>> [1] https://github.com/vy/log4j2-logstash-layout/blob/master/CHANGELOG.md#2020-01-08-v100
>>> [2] https://github.com/vy/log4j2-logstash-layout#predefined-templates
>>> [3] https://github.com/FasterXML/jackson-core/pull/589
>>> [4] https://github.com/FasterXML/jackson-core/pull/590
>>> [5] https://github.com/vy/log4j2-logstash-layout#performance
>>> [6] https://github.com/fizzed/rocker
>>>
>>
>>
>
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Volkan Yazıcı <vo...@gmail.com>.
Yes. I had the intention that you first wanted to see if a generic
text-based templating solution would work, since you were the one
proposing this idea in the first place. Would you rather first see the
current LogstashLayout replacing JsonLayout in log4j-core? If so, I
can first focus on integrating LogstashLayout into log4j-core.
On Thu, Jan 9, 2020 at 1:19 AM Ralph Goers <ra...@dslextreme.com> wrote:
>
> Are you still considering contributing this?
>
> Ralph
>
> > On Jan 8, 2020, at 2:42 PM, Volkan Yazıcı <vo...@gmail.com> wrote:
> >
> > On Tue, Dec 24, 2019 at 10:48 PM Volkan Yazıcı <vo...@gmail.com> wrote:
> >> In its current form, LogstashLayout can (almost) very well render the
> >> GELF layout[1]. There are only two shortcomings: 1) epoch seconds with
> >> fraction and 2) using PatternLayout while rendering messages. But no
> >> worries, I will address them[2,3] in a couple of days.
> >
> > I have finally released log4j2-logstash-layout v1.0.0[1], which
> > includes GELF layout[2]. Exposing epoch seconds of type double in the
> > JSON template variables as efficiently as Log4j 2.0 GelfLayout turned
> > out to be a tougher cookie than I was anticipating. But I can't
> > complain, thanks to the feature, I've stepped into the Jackson hall of
> > fame[3,4]. For those interested in, LogstashLayout is *still* the
> > fastest JSON layout and I've updated the benchmark[5] accordingly.
> >
> > In my next adventure, I will see if I can beat PatternLayout with a
> > generic template engine, e.g., Rocker[6]. If that experiment turns out
> > to be positive, I will extend it with JsonLayout.
> >
> > [1] https://github.com/vy/log4j2-logstash-layout/blob/master/CHANGELOG.md#2020-01-08-v100
> > [2] https://github.com/vy/log4j2-logstash-layout#predefined-templates
> > [3] https://github.com/FasterXML/jackson-core/pull/589
> > [4] https://github.com/FasterXML/jackson-core/pull/590
> > [5] https://github.com/vy/log4j2-logstash-layout#performance
> > [6] https://github.com/fizzed/rocker
> >
>
>
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Ralph Goers <ra...@dslextreme.com>.
Are you still considering contributing this?
Ralph
> On Jan 8, 2020, at 2:42 PM, Volkan Yazıcı <vo...@gmail.com> wrote:
>
> On Tue, Dec 24, 2019 at 10:48 PM Volkan Yazıcı <vo...@gmail.com> wrote:
>> In its current form, LogstashLayout can (almost) very well render the
>> GELF layout[1]. There are only two shortcomings: 1) epoch seconds with
>> fraction and 2) using PatternLayout while rendering messages. But no
>> worries, I will address them[2,3] in a couple of days.
>
> I have finally released log4j2-logstash-layout v1.0.0[1], which
> includes GELF layout[2]. Exposing epoch seconds of type double in the
> JSON template variables as efficiently as Log4j 2.0 GelfLayout turned
> out to be a tougher cookie than I was anticipating. But I can't
> complain, thanks to the feature, I've stepped into the Jackson hall of
> fame[3,4]. For those interested in, LogstashLayout is *still* the
> fastest JSON layout and I've updated the benchmark[5] accordingly.
>
> In my next adventure, I will see if I can beat PatternLayout with a
> generic template engine, e.g., Rocker[6]. If that experiment turns out
> to be positive, I will extend it with JsonLayout.
>
> [1] https://github.com/vy/log4j2-logstash-layout/blob/master/CHANGELOG.md#2020-01-08-v100
> [2] https://github.com/vy/log4j2-logstash-layout#predefined-templates
> [3] https://github.com/FasterXML/jackson-core/pull/589
> [4] https://github.com/FasterXML/jackson-core/pull/590
> [5] https://github.com/vy/log4j2-logstash-layout#performance
> [6] https://github.com/fizzed/rocker
>
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Volkan Yazıcı <vo...@gmail.com>.
On Tue, Dec 24, 2019 at 10:48 PM Volkan Yazıcı <vo...@gmail.com> wrote:
> In its current form, LogstashLayout can (almost) very well render the
> GELF layout[1]. There are only two shortcomings: 1) epoch seconds with
> fraction and 2) using PatternLayout while rendering messages. But no
> worries, I will address them[2,3] in a couple of days.
I have finally released log4j2-logstash-layout v1.0.0[1], which
includes GELF layout[2]. Exposing epoch seconds of type double in the
JSON template variables as efficiently as Log4j 2.0 GelfLayout turned
out to be a tougher cookie than I was anticipating. But I can't
complain, thanks to the feature, I've stepped into the Jackson hall of
fame[3,4]. For those interested in, LogstashLayout is *still* the
fastest JSON layout and I've updated the benchmark[5] accordingly.
In my next adventure, I will see if I can beat PatternLayout with a
generic template engine, e.g., Rocker[6]. If that experiment turns out
to be positive, I will extend it with JsonLayout.
[1] https://github.com/vy/log4j2-logstash-layout/blob/master/CHANGELOG.md#2020-01-08-v100
[2] https://github.com/vy/log4j2-logstash-layout#predefined-templates
[3] https://github.com/FasterXML/jackson-core/pull/589
[4] https://github.com/FasterXML/jackson-core/pull/590
[5] https://github.com/vy/log4j2-logstash-layout#performance
[6] https://github.com/fizzed/rocker
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Volkan Yazıcı <vo...@gmail.com>.
On Tue, Dec 24, 2019 at 4:38 PM Ralph Goers <ra...@dslextreme.com> wrote:
> Personally, I would have no problem accepting a contribution that is
> backwards compatible and meets ASF requirements; it has to use the
> Apache License and the contribution needs to be signed off by anyone
> who has copyright claims. That can either be done via one or more ICLAs
> or via a software grant. If it is just you then you would just need to sign an
> ICLA. Also, we would hope you would stick around to provide support for
> the component.
I interpret this as such a contribution will be approved by you from a
technical point of view. Please Correct me if I am wrong. I would not
prefer to invest my time into such a commitment if that is not the
case. My Apache committer username (vy) and track record of support
for log4j2-logstash-layout satisfies your other points, I guess.
> No, I am just thinking that when storing into ElasticSearch user’s typically
> will want the data stored in the manner they want to query it. I know for
> my use case I care about the timestamp, formatted message, thread
> context data and any event properties. I would want the message portion
> formatted using a PatternLayout. I modified the GELF Layout in 2.13.0 to
> do that so it pretty much meets my needs to log with Logstash. In fact, the
> JSON Layout should be flexible enough that one could configure it to
> emulate the GELF format.
In its current form, LogstashLayout can (almost) very well render the
GELF layout[1]. There are only two shortcomings: 1) epoch seconds with
fraction and 2) using PatternLayout while rendering messages. But no
worries, I will address them[2,3] in a couple of days.
[1] https://docs.graylog.org/en/3.1/pages/gelf.html#gelf-payload-specification
[2] https://github.com/vy/log4j2-logstash-layout/issues/48
[3] https://github.com/vy/log4j2-logstash-layout/issues/49
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Ralph Goers <ra...@dslextreme.com>.
Comments inline
> On Dec 24, 2019, at 2:09 AM, Volkan Yazıcı <vo...@gmail.com> wrote:
>
> This is easier said than done. Customizing the schema and injected
> content is the raison d'être of LogstashLayout. Would you people
> accept a PR re-branding LogstashLayout as the new JsonLayout where the
> default JSON schema conforms with the JsonLayout of Log4j 2.0? This
> would provide 100% backward-compatibility with the existing JsonLayout
> and introduce all the benefits (i.e., configuration knobs, zero TLA)
> of LogstashLayout.
Personally, I would have no problem accepting a contribution that is backwards compatible and meets ASF requirements; it has to use the Apache License and the contribution needs to be signed off by anyone who has copyright claims. That can either be done via one or more ICLAs or via a software grant. If it is just you then you would just need to sign an ICLA. Also, we would hope you would stick around to provide support for the component.
>
> @Ralph, do you have any plans on your mind for the API to expose to
> the user for customizing the JSON schema? In LogstashLayout, I use an
> ad-hoc JSON templating -- see Log4j2JsonLayout.json[1], for instance.
> I am not proud of this way of templating, you can't express every kind
> of transformation, e.g., "message": "foo bar baz ${json:message}" will
> not yield the output what one would expect it to. That said, it is
> simple and powerful enough to empower thousands of users so far.
> Further, I have never received any single complaints about its
> shortcomings. (If there are bright minds in the room with some
> JsonT[2] and/or fn:xml-to-json[3], please chime in.)
No, I am just thinking that when storing into ElasticSearch user’s typically will want the data stored in the manner they want to query it. I know for my use case I care about the timestamp, formatted message, thread context data and any event properties. I would want the message portion formatted using a PatternLayout. I modified the GELF Layout in 2.13.0 to do that so it pretty much meets my needs to log with Logstash. In fact, the JSON Layout should be flexible enough that one could configure it to emulate the GELF format.
>
> For XML layout, we can add a configuration knob for XSLT
> transformation that kicks in at rendering time. How to perform this
> "efficiently" (e.g., can we compile the transformation once and reuse
> it as in LogstashLayout) is an open question. We can work that detail
> out in later releases since it will be a user-invisible enhancement.
> (Given both XML and HTML are SGML derivatives, can't we employ the
> same XSLT transformation magic in HtmlLayout design too?)
>
> For YAML layout... Is anybody using it at all? If so, I am really
> curious about the use cases. If there is a demand, we can introduce
> custom schema templating (ala LogstashLayout) to YamlLayout too.
> Though judging from my experience in LogstashLayout, this is quite
> some work. The absence of any YAML schema transformation tools in the
> F/OSS market is also a strong indication that this is not trivial and
> implies many subtle corner cases that needs to be brushed.
As I recall the JSON Layout originally did not use Jackson. I believe Gary converted it to use it. I don’t recall why. Once it was converted I believe he decided he could convert the XML Layout and create a YAML Layout almost for free by leveraging the Jackson support.
>
> I would like to share some remarks about introducing dependencies
> (e.g., Jackson) here.
>
> - For JSON rendering, we can do without Jackson. For
> instance log4j2-ecs-layout[4] renders JSON using its own
> simple single-class-file tooling and *optionally* falls back
> to Jackson for serializing custom Message objects.
>
> - Ever single extra dependency on top of Log4j core
> increments the exponent of dev(ops) nightmare.I find
> myself explaining how to properly package
> log4j2-logstash-layout to dev(ops) people a couple
> of times every year.
As I said, I don’t believe the Layout has to be married to Jackson. As far as I am concerned, the fewer dependencies the better. In fact, this would mean the JSONLayout could remain as part of log4j-core in 3.0.
>
> In case you wonder, my intention is to collect feedback, conclude on a
> blueprint blessed by existing contributors, break the blueprint down
> into structured JIRA stories, and start working on them.
>
> [1] https://github.com/vy/log4j2-logstash-layout/blob/master/layout/src/test/resources/Log4j2JsonLayout.json
> [2] https://goessner.net/articles/jsont/
> [3] https://www.w3.org/TR/xslt-30/#func-xml-to-json
> [4] https://github.com/elastic/java-ecs-logging/tree/master/log4j2-ecs-layout
>
Ralph
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Matt Sicker <bo...@gmail.com>.
The JSON pattern escape was added originally to form generic JSON layouts
if desired. I think I originally used that in a similar scenario, tho the
layout definition is huge.
On Wed, Dec 25, 2019 at 08:47 Carter Kozak <ck...@ckozak.net> wrote:
> On Wed, Dec 25, 2019, at 01:38, Ralph Goers wrote:
> >
> >
> > > On Dec 24, 2019, at 2:06 PM, Volkan Yazıcı <vo...@gmail.com>
> wrote:
> > >
> > > Indeed ColumnMapping is an alternative approach. I believe due to its
> > > origin, i.e., RDBMSes, it facilitates a 1-to-1 mapping particularly
> > > targeted at table column names, e.g.,
> > >
> > > timestamp -> instant
> > > message -> summary
> > >
> > > One can borrow the idea and extend it a little bit to support the
> > > structured form of a JSON:
> > >
> > > timestamp.millis -> instant
> > > throwable -> error.message
> > >
> > > That said, I believe current JSON-based templating in LogstashLayout
> > > communicates the intent more concisely (the template is WYSIWYG) and
> > > allows extra (JSON-specific) flexibility, e.g.,
> > >
> > > "user": ["${json:mdc:user}", "${json:mdc:fallbackUser}”]
> > >
> >
> > Looking at
> https://github.com/vy/log4j2-logstash-layout/blob/master/layout/src/main/resources/LogstashJsonEventLayoutV1.json
> <
> https://github.com/vy/log4j2-logstash-layout/blob/master/layout/src/main/resources/LogstashJsonEventLayoutV1.json>
> makes me think that you could accomplish the same thing with a few
> enhancements to the PatternLayout. In fact, all the expressions with
> ${json:…} look like they could be a LogEvent Lookup where you want the
> output to be formatted as JSON. I am not sure why you didn’t implement it
> that way.
> >
> > I could also see creating a new pattern converter to cause formatting to
> be in JSON so when any pattern converter is called it knows to output JSON.
> So %d would result in “timestamp”: “2019-12-25 01:02:03,123” etc. So a
> pattern of “%json{%d, %p, %c, %C, %M, %L, %m}” would generate
> “{“timestamp”: “2019-12-25 01:02:03,123”, “level”: “debug”, “logger”:
> “com.mycorp.MyClass”, “class”: “com.mycorp.MyClass”, “method”:
> “doSomething”: “lineNumber”: 123, “message”: “Hello, world!”}
> >
> > FWIW, I am not necessarily proposing we implement that. I am trying to
> point out that your templating approach doesn’t really seem to be tied to
> JSON, so calling it JsonLogstashLayout seems a bit misleading. I could just
> as easily imagine an XML template with xml resolvers.
> >
> > Also, I must admit I probably would have implemented the resolvers
> differently. It seems to me that each “entity" is going to return either a
> String, a number, a Map, or an array. So I would have just had the resolver
> return its key and its data and left calling the jsonGenerator to the class
> that calls all the resolvers. Then you could have reused the same resolvers
> to generate XML. At least, that is the path I would have tried to go down.
> Obviously I have only glanced at your code so it could be more challenging
> than it appears.
> >
> > Anyway, those are just my first random thoughts after looking at what
> the layout does.
> >
> > Ralph
> >
>
> I've had a lot of success using pattern layout with the
> EncodingPatternConverter to escape json strings. The layout definition can
> be difficult to read, but it's incredibly powerful. For example:
>
> {"timestamp":"%d", "level":"%p", "origin":"%encode{%c}{JSON}",
> "thread":"%encode{%t}{JSON}", "message":%encode{m}{JSON}}%n
>
> There are a few pattern converters that don't map particularly well to
> this method, for example the MdcPatternConverter with multiple keys doesn't
> currently have a JSON formatting option. It would be great if we could make
> this functionality easier for users, likely through some combination of
> documentation and features. I've found it helpful that we don't require
> jackson for this approach to work.
>
> -ck
>
--
Matt Sicker <bo...@gmail.com>
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature request/possible contribution)
Posted by Carter Kozak <ck...@ckozak.net>.
On Wed, Dec 25, 2019, at 01:38, Ralph Goers wrote:
>
>
> > On Dec 24, 2019, at 2:06 PM, Volkan Yazıcı <vo...@gmail.com> wrote:
> >
> > Indeed ColumnMapping is an alternative approach. I believe due to its
> > origin, i.e., RDBMSes, it facilitates a 1-to-1 mapping particularly
> > targeted at table column names, e.g.,
> >
> > timestamp -> instant
> > message -> summary
> >
> > One can borrow the idea and extend it a little bit to support the
> > structured form of a JSON:
> >
> > timestamp.millis -> instant
> > throwable -> error.message
> >
> > That said, I believe current JSON-based templating in LogstashLayout
> > communicates the intent more concisely (the template is WYSIWYG) and
> > allows extra (JSON-specific) flexibility, e.g.,
> >
> > "user": ["${json:mdc:user}", "${json:mdc:fallbackUser}”]
> >
>
> Looking at https://github.com/vy/log4j2-logstash-layout/blob/master/layout/src/main/resources/LogstashJsonEventLayoutV1.json <https://github.com/vy/log4j2-logstash-layout/blob/master/layout/src/main/resources/LogstashJsonEventLayoutV1.json> makes me think that you could accomplish the same thing with a few enhancements to the PatternLayout. In fact, all the expressions with ${json:…} look like they could be a LogEvent Lookup where you want the output to be formatted as JSON. I am not sure why you didn’t implement it that way.
>
> I could also see creating a new pattern converter to cause formatting to be in JSON so when any pattern converter is called it knows to output JSON. So %d would result in “timestamp”: “2019-12-25 01:02:03,123” etc. So a pattern of “%json{%d, %p, %c, %C, %M, %L, %m}” would generate “{“timestamp”: “2019-12-25 01:02:03,123”, “level”: “debug”, “logger”: “com.mycorp.MyClass”, “class”: “com.mycorp.MyClass”, “method”: “doSomething”: “lineNumber”: 123, “message”: “Hello, world!”}
>
> FWIW, I am not necessarily proposing we implement that. I am trying to point out that your templating approach doesn’t really seem to be tied to JSON, so calling it JsonLogstashLayout seems a bit misleading. I could just as easily imagine an XML template with xml resolvers.
>
> Also, I must admit I probably would have implemented the resolvers differently. It seems to me that each “entity" is going to return either a String, a number, a Map, or an array. So I would have just had the resolver return its key and its data and left calling the jsonGenerator to the class that calls all the resolvers. Then you could have reused the same resolvers to generate XML. At least, that is the path I would have tried to go down. Obviously I have only glanced at your code so it could be more challenging than it appears.
>
> Anyway, those are just my first random thoughts after looking at what the layout does.
>
> Ralph
>
I've had a lot of success using pattern layout with the EncodingPatternConverter to escape json strings. The layout definition can be difficult to read, but it's incredibly powerful. For example:
{"timestamp":"%d", "level":"%p", "origin":"%encode{%c}{JSON}", "thread":"%encode{%t}{JSON}", "message":%encode{m}{JSON}}%n
There are a few pattern converters that don't map particularly well to this method, for example the MdcPatternConverter with multiple keys doesn't currently have a JSON formatting option. It would be great if we could make this functionality easier for users, likely through some combination of documentation and features. I've found it helpful that we don't require jackson for this approach to work.
-ck
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Volkan Yazıcı <vo...@gmail.com>.
On Wed, Dec 25, 2019 at 7:38 AM Ralph Goers <ra...@dslextreme.com> wrote:
> Also, I must admit I probably would have implemented the
> resolvers differently. It seems to me that each “entity" is going
> to return either a String, a number, a Map, or an array. So I
> would have just had the resolver return its key and its data and
> left calling the jsonGenerator to the class that calls all the resolvers.
That was my initial attempt as well. But later figured out that not
every resolver is returning an "entity". For optimization purposes,
some reach out to nasty JsonGenerator tricks such as writeRawValue(),
etc.
> Then you could have reused the same resolvers to generate XML.
How would you differentiate between XML attributes and values in such a case?
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Ralph Goers <ra...@dslextreme.com>.
It would be an interesting experiment to find out how they perform. That said it feels like it might be overkill to use a template engine for such a simple thing, but if it performs well then I guess it may not matter.
What most users want to log is pretty simple. The vast majority of the overhead is retrieving and formatting the data, especially the location information. I wouldn’t think a “pattern” for JSON, XML or any other format would be much more complicated, and could be very similar to, what we have implemented for the pattern layout.
If I was implementing this I would just say users should specify the pattern converters that they want (i.e. - the logger name, level, marker, etc) which just return the data, and they use a specified format to generate the output. I suspect this could even work for a binary format if that was desired.
Ralph
> On Dec 25, 2019, at 3:17 PM, Matt Sicker <bo...@gmail.com> wrote:
>
> I’m not sure if any existing template libraries are garbage free, though I
> like the idea in general at least.
>
> On Wed, Dec 25, 2019 at 14:42 Volkan Yazıcı <vo...@gmail.com> wrote:
>
>> On Wed, Dec 25, 2019 at 7:38 AM Ralph Goers <ra...@dslextreme.com>
>> wrote:
>>> FWIW, I am not necessarily proposing we implement that. I am
>>> trying to point out that your templating approach doesn’t really
>>> seem to be tied to JSON, [...] I could just as easily imagine an
>>> XML template with xml resolvers.
>>
>> This is a striking observation Ralph! (I think Carter Kozak is
>> pointing in the same direction in the thread.) Consider the following:
>> if we provide a fully-fledged template engine (e.g., Velocity,
>> FreeMarker, Thymeleaf, Pippo) with sufficient expressiveness (here
>> consider someone trying to emit epoch seconds in floating point format
>> from LogEvent#getInstant()) and escaping utilities (e.g.,
>> {Json,Sgml,Yaml}Helpers.escapeString()), users can render whatever
>> text-based output they want, be it XML, JSON, YAML, HTML, etc. If we
>> would look deep enough, all what LogstashLayout, PatternLayout, etc.
>> are trying to achieve is a short-sighted attempt to fill in this
>> template engine gap.
>>
>> Here is a thought experiment: Given a VelocityLayout with all its
>> whistles and bells (e.g., EscapeTool[1]), are there any text-based
>> layouts we know of that is not expressible by VelocityLayout? Further,
>> I expect VelocityLayout to outperform the existing individual
>> text-based layouts due to its template compilation. (You can repeat
>> the same experiment by replacing "Velocity" with your favorite Java
>> templating engine.)
>>
>> I am strongly in favor of moving towards this direction rather than
>> introducing ad-hoc layouts for each text-based format that Log4j needs
>> to emit.
>>
>> [Thanks for the rest of the comments Ralph. I skipped replying to them
>> on purpose to just focus on this significant observation.]
>>
>> [1]
>> http://velocity.apache.org/tools/devel/apidocs/org/apache/velocity/tools/generic/EscapeTool.html
>>
> --
> Matt Sicker <bo...@gmail.com>
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Volkan Yazıcı <vo...@gmail.com>.
That is a valid concern. Googl'ing bumped me into Rocker[1]. Its
garbage generation appears to be tractable. For the rest (Velocity,
FreeMarker, etc.), I am curious if we can create a memory pool to
reuse contextual objects.
[1] https://github.com/fizzed/rocker
On Wed, Dec 25, 2019 at 11:17 PM Matt Sicker <bo...@gmail.com> wrote:
>
> I’m not sure if any existing template libraries are garbage free, though I
> like the idea in general at least.
>
> On Wed, Dec 25, 2019 at 14:42 Volkan Yazıcı <vo...@gmail.com> wrote:
>
> > On Wed, Dec 25, 2019 at 7:38 AM Ralph Goers <ra...@dslextreme.com>
> > wrote:
> > > FWIW, I am not necessarily proposing we implement that. I am
> > > trying to point out that your templating approach doesn’t really
> > > seem to be tied to JSON, [...] I could just as easily imagine an
> > > XML template with xml resolvers.
> >
> > This is a striking observation Ralph! (I think Carter Kozak is
> > pointing in the same direction in the thread.) Consider the following:
> > if we provide a fully-fledged template engine (e.g., Velocity,
> > FreeMarker, Thymeleaf, Pippo) with sufficient expressiveness (here
> > consider someone trying to emit epoch seconds in floating point format
> > from LogEvent#getInstant()) and escaping utilities (e.g.,
> > {Json,Sgml,Yaml}Helpers.escapeString()), users can render whatever
> > text-based output they want, be it XML, JSON, YAML, HTML, etc. If we
> > would look deep enough, all what LogstashLayout, PatternLayout, etc.
> > are trying to achieve is a short-sighted attempt to fill in this
> > template engine gap.
> >
> > Here is a thought experiment: Given a VelocityLayout with all its
> > whistles and bells (e.g., EscapeTool[1]), are there any text-based
> > layouts we know of that is not expressible by VelocityLayout? Further,
> > I expect VelocityLayout to outperform the existing individual
> > text-based layouts due to its template compilation. (You can repeat
> > the same experiment by replacing "Velocity" with your favorite Java
> > templating engine.)
> >
> > I am strongly in favor of moving towards this direction rather than
> > introducing ad-hoc layouts for each text-based format that Log4j needs
> > to emit.
> >
> > [Thanks for the rest of the comments Ralph. I skipped replying to them
> > on purpose to just focus on this significant observation.]
> >
> > [1]
> > http://velocity.apache.org/tools/devel/apidocs/org/apache/velocity/tools/generic/EscapeTool.html
> >
> --
> Matt Sicker <bo...@gmail.com>
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Matt Sicker <bo...@gmail.com>.
I’m not sure if any existing template libraries are garbage free, though I
like the idea in general at least.
On Wed, Dec 25, 2019 at 14:42 Volkan Yazıcı <vo...@gmail.com> wrote:
> On Wed, Dec 25, 2019 at 7:38 AM Ralph Goers <ra...@dslextreme.com>
> wrote:
> > FWIW, I am not necessarily proposing we implement that. I am
> > trying to point out that your templating approach doesn’t really
> > seem to be tied to JSON, [...] I could just as easily imagine an
> > XML template with xml resolvers.
>
> This is a striking observation Ralph! (I think Carter Kozak is
> pointing in the same direction in the thread.) Consider the following:
> if we provide a fully-fledged template engine (e.g., Velocity,
> FreeMarker, Thymeleaf, Pippo) with sufficient expressiveness (here
> consider someone trying to emit epoch seconds in floating point format
> from LogEvent#getInstant()) and escaping utilities (e.g.,
> {Json,Sgml,Yaml}Helpers.escapeString()), users can render whatever
> text-based output they want, be it XML, JSON, YAML, HTML, etc. If we
> would look deep enough, all what LogstashLayout, PatternLayout, etc.
> are trying to achieve is a short-sighted attempt to fill in this
> template engine gap.
>
> Here is a thought experiment: Given a VelocityLayout with all its
> whistles and bells (e.g., EscapeTool[1]), are there any text-based
> layouts we know of that is not expressible by VelocityLayout? Further,
> I expect VelocityLayout to outperform the existing individual
> text-based layouts due to its template compilation. (You can repeat
> the same experiment by replacing "Velocity" with your favorite Java
> templating engine.)
>
> I am strongly in favor of moving towards this direction rather than
> introducing ad-hoc layouts for each text-based format that Log4j needs
> to emit.
>
> [Thanks for the rest of the comments Ralph. I skipped replying to them
> on purpose to just focus on this significant observation.]
>
> [1]
> http://velocity.apache.org/tools/devel/apidocs/org/apache/velocity/tools/generic/EscapeTool.html
>
--
Matt Sicker <bo...@gmail.com>
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Volkan Yazıcı <vo...@gmail.com>.
On Wed, Dec 25, 2019 at 7:38 AM Ralph Goers <ra...@dslextreme.com> wrote:
> FWIW, I am not necessarily proposing we implement that. I am
> trying to point out that your templating approach doesn’t really
> seem to be tied to JSON, [...] I could just as easily imagine an
> XML template with xml resolvers.
This is a striking observation Ralph! (I think Carter Kozak is
pointing in the same direction in the thread.) Consider the following:
if we provide a fully-fledged template engine (e.g., Velocity,
FreeMarker, Thymeleaf, Pippo) with sufficient expressiveness (here
consider someone trying to emit epoch seconds in floating point format
from LogEvent#getInstant()) and escaping utilities (e.g.,
{Json,Sgml,Yaml}Helpers.escapeString()), users can render whatever
text-based output they want, be it XML, JSON, YAML, HTML, etc. If we
would look deep enough, all what LogstashLayout, PatternLayout, etc.
are trying to achieve is a short-sighted attempt to fill in this
template engine gap.
Here is a thought experiment: Given a VelocityLayout with all its
whistles and bells (e.g., EscapeTool[1]), are there any text-based
layouts we know of that is not expressible by VelocityLayout? Further,
I expect VelocityLayout to outperform the existing individual
text-based layouts due to its template compilation. (You can repeat
the same experiment by replacing "Velocity" with your favorite Java
templating engine.)
I am strongly in favor of moving towards this direction rather than
introducing ad-hoc layouts for each text-based format that Log4j needs
to emit.
[Thanks for the rest of the comments Ralph. I skipped replying to them
on purpose to just focus on this significant observation.]
[1] http://velocity.apache.org/tools/devel/apidocs/org/apache/velocity/tools/generic/EscapeTool.html
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Ralph Goers <ra...@dslextreme.com>.
> On Dec 24, 2019, at 2:06 PM, Volkan Yazıcı <vo...@gmail.com> wrote:
>
> Indeed ColumnMapping is an alternative approach. I believe due to its
> origin, i.e., RDBMSes, it facilitates a 1-to-1 mapping particularly
> targeted at table column names, e.g.,
>
> timestamp -> instant
> message -> summary
>
> One can borrow the idea and extend it a little bit to support the
> structured form of a JSON:
>
> timestamp.millis -> instant
> throwable -> error.message
>
> That said, I believe current JSON-based templating in LogstashLayout
> communicates the intent more concisely (the template is WYSIWYG) and
> allows extra (JSON-specific) flexibility, e.g.,
>
> "user": ["${json:mdc:user}", "${json:mdc:fallbackUser}”]
>
Looking at https://github.com/vy/log4j2-logstash-layout/blob/master/layout/src/main/resources/LogstashJsonEventLayoutV1.json <https://github.com/vy/log4j2-logstash-layout/blob/master/layout/src/main/resources/LogstashJsonEventLayoutV1.json> makes me think that you could accomplish the same thing with a few enhancements to the PatternLayout. In fact, all the expressions with ${json:…} look like they could be a LogEvent Lookup where you want the output to be formatted as JSON. I am not sure why you didn’t implement it that way.
I could also see creating a new pattern converter to cause formatting to be in JSON so when any pattern converter is called it knows to output JSON. So %d would result in “timestamp”: “2019-12-25 01:02:03,123” etc. So a pattern of “%json{%d, %p, %c, %C, %M, %L, %m}” would generate “{“timestamp”: “2019-12-25 01:02:03,123”, “level”: “debug”, “logger”: “com.mycorp.MyClass”, “class”: “com.mycorp.MyClass”, “method”: “doSomething”: “lineNumber”: 123, “message”: “Hello, world!”}
FWIW, I am not necessarily proposing we implement that. I am trying to point out that your templating approach doesn’t really seem to be tied to JSON, so calling it JsonLogstashLayout seems a bit misleading. I could just as easily imagine an XML template with xml resolvers.
Also, I must admit I probably would have implemented the resolvers differently. It seems to me that each “entity" is going to return either a String, a number, a Map, or an array. So I would have just had the resolver return its key and its data and left calling the jsonGenerator to the class that calls all the resolvers. Then you could have reused the same resolvers to generate XML. At least, that is the path I would have tried to go down. Obviously I have only glanced at your code so it could be more challenging than it appears.
Anyway, those are just my first random thoughts after looking at what the layout does.
Ralph
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Volkan Yazıcı <vo...@gmail.com>.
Indeed ColumnMapping is an alternative approach. I believe due to its
origin, i.e., RDBMSes, it facilitates a 1-to-1 mapping particularly
targeted at table column names, e.g.,
timestamp -> instant
message -> summary
One can borrow the idea and extend it a little bit to support the
structured form of a JSON:
timestamp.millis -> instant
throwable -> error.message
That said, I believe current JSON-based templating in LogstashLayout
communicates the intent more concisely (the template is WYSIWYG) and
allows extra (JSON-specific) flexibility, e.g.,
"user": ["${json:mdc:user}", "${json:mdc:fallbackUser}"]
On Tue, Dec 24, 2019 at 3:54 PM Matt Sicker <bo...@gmail.com> wrote:
>
> Perhaps the ColumnMapping plugin from the database appenders could be
> useful here?
>
> On Tue, Dec 24, 2019 at 03:09 Volkan Yazıcı <vo...@gmail.com> wrote:
>
> > This is easier said than done. Customizing the schema and injected
> > content is the raison d'être of LogstashLayout. Would you people
> > accept a PR re-branding LogstashLayout as the new JsonLayout where the
> > default JSON schema conforms with the JsonLayout of Log4j 2.0? This
> > would provide 100% backward-compatibility with the existing JsonLayout
> > and introduce all the benefits (i.e., configuration knobs, zero TLA)
> > of LogstashLayout.
> >
> > @Ralph, do you have any plans on your mind for the API to expose to
> > the user for customizing the JSON schema? In LogstashLayout, I use an
> > ad-hoc JSON templating -- see Log4j2JsonLayout.json[1], for instance.
> > I am not proud of this way of templating, you can't express every kind
> > of transformation, e.g., "message": "foo bar baz ${json:message}" will
> > not yield the output what one would expect it to. That said, it is
> > simple and powerful enough to empower thousands of users so far.
> > Further, I have never received any single complaints about its
> > shortcomings. (If there are bright minds in the room with some
> > JsonT[2] and/or fn:xml-to-json[3], please chime in.)
> >
> > For XML layout, we can add a configuration knob for XSLT
> > transformation that kicks in at rendering time. How to perform this
> > "efficiently" (e.g., can we compile the transformation once and reuse
> > it as in LogstashLayout) is an open question. We can work that detail
> > out in later releases since it will be a user-invisible enhancement.
> > (Given both XML and HTML are SGML derivatives, can't we employ the
> > same XSLT transformation magic in HtmlLayout design too?)
> >
> > For YAML layout... Is anybody using it at all? If so, I am really
> > curious about the use cases. If there is a demand, we can introduce
> > custom schema templating (ala LogstashLayout) to YamlLayout too.
> > Though judging from my experience in LogstashLayout, this is quite
> > some work. The absence of any YAML schema transformation tools in the
> > F/OSS market is also a strong indication that this is not trivial and
> > implies many subtle corner cases that needs to be brushed.
> >
> > I would like to share some remarks about introducing dependencies
> > (e.g., Jackson) here.
> >
> > - For JSON rendering, we can do without Jackson. For
> > instance log4j2-ecs-layout[4] renders JSON using its own
> > simple single-class-file tooling and *optionally* falls back
> > to Jackson for serializing custom Message objects.
> >
> > - Ever single extra dependency on top of Log4j core
> > increments the exponent of dev(ops) nightmare.I find
> > myself explaining how to properly package
> > log4j2-logstash-layout to dev(ops) people a couple
> > of times every year.
> >
> > In case you wonder, my intention is to collect feedback, conclude on a
> > blueprint blessed by existing contributors, break the blueprint down
> > into structured JIRA stories, and start working on them.
> >
> > [1]
> > https://github.com/vy/log4j2-logstash-layout/blob/master/layout/src/test/resources/Log4j2JsonLayout.json
> > [2] https://goessner.net/articles/jsont/
> > [3] https://www.w3.org/TR/xslt-30/#func-xml-to-json
> > [4]
> > https://github.com/elastic/java-ecs-logging/tree/master/log4j2-ecs-layout
> >
> > On Tue, Dec 24, 2019 at 12:25 AM Ralph Goers <ra...@dslextreme.com>
> > wrote:
> > > What I was thinking was that I would just like a more
> > > flexible way to specify what should be included in the
> > > JSON. In its current form the JSON Layout just creates
> > > a serialized version of a LogEvent. I would prefer it if
> > > users could define the individual fields they want and
> > > what they should contain.
> >
> --
> Matt Sicker <bo...@gmail.com>
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Matt Sicker <bo...@gmail.com>.
Perhaps the ColumnMapping plugin from the database appenders could be
useful here?
On Tue, Dec 24, 2019 at 03:09 Volkan Yazıcı <vo...@gmail.com> wrote:
> This is easier said than done. Customizing the schema and injected
> content is the raison d'être of LogstashLayout. Would you people
> accept a PR re-branding LogstashLayout as the new JsonLayout where the
> default JSON schema conforms with the JsonLayout of Log4j 2.0? This
> would provide 100% backward-compatibility with the existing JsonLayout
> and introduce all the benefits (i.e., configuration knobs, zero TLA)
> of LogstashLayout.
>
> @Ralph, do you have any plans on your mind for the API to expose to
> the user for customizing the JSON schema? In LogstashLayout, I use an
> ad-hoc JSON templating -- see Log4j2JsonLayout.json[1], for instance.
> I am not proud of this way of templating, you can't express every kind
> of transformation, e.g., "message": "foo bar baz ${json:message}" will
> not yield the output what one would expect it to. That said, it is
> simple and powerful enough to empower thousands of users so far.
> Further, I have never received any single complaints about its
> shortcomings. (If there are bright minds in the room with some
> JsonT[2] and/or fn:xml-to-json[3], please chime in.)
>
> For XML layout, we can add a configuration knob for XSLT
> transformation that kicks in at rendering time. How to perform this
> "efficiently" (e.g., can we compile the transformation once and reuse
> it as in LogstashLayout) is an open question. We can work that detail
> out in later releases since it will be a user-invisible enhancement.
> (Given both XML and HTML are SGML derivatives, can't we employ the
> same XSLT transformation magic in HtmlLayout design too?)
>
> For YAML layout... Is anybody using it at all? If so, I am really
> curious about the use cases. If there is a demand, we can introduce
> custom schema templating (ala LogstashLayout) to YamlLayout too.
> Though judging from my experience in LogstashLayout, this is quite
> some work. The absence of any YAML schema transformation tools in the
> F/OSS market is also a strong indication that this is not trivial and
> implies many subtle corner cases that needs to be brushed.
>
> I would like to share some remarks about introducing dependencies
> (e.g., Jackson) here.
>
> - For JSON rendering, we can do without Jackson. For
> instance log4j2-ecs-layout[4] renders JSON using its own
> simple single-class-file tooling and *optionally* falls back
> to Jackson for serializing custom Message objects.
>
> - Ever single extra dependency on top of Log4j core
> increments the exponent of dev(ops) nightmare.I find
> myself explaining how to properly package
> log4j2-logstash-layout to dev(ops) people a couple
> of times every year.
>
> In case you wonder, my intention is to collect feedback, conclude on a
> blueprint blessed by existing contributors, break the blueprint down
> into structured JIRA stories, and start working on them.
>
> [1]
> https://github.com/vy/log4j2-logstash-layout/blob/master/layout/src/test/resources/Log4j2JsonLayout.json
> [2] https://goessner.net/articles/jsont/
> [3] https://www.w3.org/TR/xslt-30/#func-xml-to-json
> [4]
> https://github.com/elastic/java-ecs-logging/tree/master/log4j2-ecs-layout
>
> On Tue, Dec 24, 2019 at 12:25 AM Ralph Goers <ra...@dslextreme.com>
> wrote:
> > What I was thinking was that I would just like a more
> > flexible way to specify what should be included in the
> > JSON. In its current form the JSON Layout just creates
> > a serialized version of a LogEvent. I would prefer it if
> > users could define the individual fields they want and
> > what they should contain.
>
--
Matt Sicker <bo...@gmail.com>
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Volkan Yazıcı <vo...@gmail.com>.
This is easier said than done. Customizing the schema and injected
content is the raison d'être of LogstashLayout. Would you people
accept a PR re-branding LogstashLayout as the new JsonLayout where the
default JSON schema conforms with the JsonLayout of Log4j 2.0? This
would provide 100% backward-compatibility with the existing JsonLayout
and introduce all the benefits (i.e., configuration knobs, zero TLA)
of LogstashLayout.
@Ralph, do you have any plans on your mind for the API to expose to
the user for customizing the JSON schema? In LogstashLayout, I use an
ad-hoc JSON templating -- see Log4j2JsonLayout.json[1], for instance.
I am not proud of this way of templating, you can't express every kind
of transformation, e.g., "message": "foo bar baz ${json:message}" will
not yield the output what one would expect it to. That said, it is
simple and powerful enough to empower thousands of users so far.
Further, I have never received any single complaints about its
shortcomings. (If there are bright minds in the room with some
JsonT[2] and/or fn:xml-to-json[3], please chime in.)
For XML layout, we can add a configuration knob for XSLT
transformation that kicks in at rendering time. How to perform this
"efficiently" (e.g., can we compile the transformation once and reuse
it as in LogstashLayout) is an open question. We can work that detail
out in later releases since it will be a user-invisible enhancement.
(Given both XML and HTML are SGML derivatives, can't we employ the
same XSLT transformation magic in HtmlLayout design too?)
For YAML layout... Is anybody using it at all? If so, I am really
curious about the use cases. If there is a demand, we can introduce
custom schema templating (ala LogstashLayout) to YamlLayout too.
Though judging from my experience in LogstashLayout, this is quite
some work. The absence of any YAML schema transformation tools in the
F/OSS market is also a strong indication that this is not trivial and
implies many subtle corner cases that needs to be brushed.
I would like to share some remarks about introducing dependencies
(e.g., Jackson) here.
- For JSON rendering, we can do without Jackson. For
instance log4j2-ecs-layout[4] renders JSON using its own
simple single-class-file tooling and *optionally* falls back
to Jackson for serializing custom Message objects.
- Ever single extra dependency on top of Log4j core
increments the exponent of dev(ops) nightmare.I find
myself explaining how to properly package
log4j2-logstash-layout to dev(ops) people a couple
of times every year.
In case you wonder, my intention is to collect feedback, conclude on a
blueprint blessed by existing contributors, break the blueprint down
into structured JIRA stories, and start working on them.
[1] https://github.com/vy/log4j2-logstash-layout/blob/master/layout/src/test/resources/Log4j2JsonLayout.json
[2] https://goessner.net/articles/jsont/
[3] https://www.w3.org/TR/xslt-30/#func-xml-to-json
[4] https://github.com/elastic/java-ecs-logging/tree/master/log4j2-ecs-layout
On Tue, Dec 24, 2019 at 12:25 AM Ralph Goers <ra...@dslextreme.com> wrote:
> What I was thinking was that I would just like a more
> flexible way to specify what should be included in the
> JSON. In its current form the JSON Layout just creates
> a serialized version of a LogEvent. I would prefer it if
> users could define the individual fields they want and
> what they should contain.
Re: Thoughts on XML/JSON/YAML layout in Log4j 3.0 (Was: Feature
request/possible contribution)
Posted by Ralph Goers <ra...@dslextreme.com>.
Thanks,
What I was thinking was that I would just like a more flexible way to specify what should be included in the JSON. In its current form the JSON Layout just creates a serialized version of a LogEvent. I would prefer it if users could define the individual fields they want and what they should contain.
Ralph
> On Dec 23, 2019, at 3:17 PM, Volkan Yazıcı <vo...@gmail.com> wrote:
>
> Hello,
>
> As the author of log4j2-logstash-layout[1], I would like to share some
> thoughts on the plugin and its implications on Log4j 3.0 roadmap.
>
> - LogstashLayout is pretty stable in its feature set for the
> last year. That is, the advantages (customization, etc.)
> it offers over JsonLayout could very well be a guideline
> for the next generation of JsonLayout.
>
> - The demand for LogstashLayout's (almost) garbage-free
> nature in combination with the efficiency of initialization-time
> compiled schema showed that people are willing to employ
> logging in unanticipated domains. In particular, Nexia Home[2]
> is leveraging[3] the plugin for distributed tracing. Hence,
> performance matters, AFAIC.
>
> - Getting design right in terms of efficiency, can even yield
> advantage over future competitors. See how LogstashLayout
> beats the official Elastic Log4j 2.0 plugin for ECS[4].
>
> Ralph already delivers a tremendous amount of work for the project. If
> he or anyone else shares the blueprints in their minds for the next
> generation of {Xml/Json,Yaml}Layout, I would be more than happy to
> volunteer for the discussion and, more importantly, implementation.
>
> Best.
>
> [1] https://github.com/vy/log4j2-logstash-layout
> [2] https://www.nexiahome.com/
> [3] https://github.com/vy/log4j2-logstash-layout/issues/17
> [4] https://github.com/vy/log4j2-logstash-layout#performance
>
> On Mon, Dec 23, 2019 at 12:24 AM Ralph Goers <ra...@dslextreme.com> wrote:
>> I am not particularly happy with the JsonLayout at the
>> moment as I think what it does is too limited. I’d like there
>> to be a lot more flexibility as to what the resulting JSON
>> should look like, not just dumping the log event.
>