You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@logging.apache.org by GitBox <gi...@apache.org> on 2021/03/03 23:14:44 UTC
[GitHub] [logging-log4j2] radut opened a new pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
radut opened a new pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472
This PR adds ability to register custom resolvers for JsonTemplateLayout
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] vy closed pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
vy closed pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] vy commented on pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
vy commented on pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472#issuecomment-790586776
> `JsonTemplateLayout` is a good idea, you could define specific constants like `"@version": 1`(number) or lets say `"source" : "log4j"`(string). What I am looking for is rewrite the mdc resolver to be able to write numbers as json numbers, instead of string literals, usefull for indexing and searching in ElasticSearch with function like between x and y, without having another transformer on the way.
MDC resolver is capable of rendering JSON objects, you don't necessarily need to _stringify_ them. You can toggle `stringified` flag of the `mdc` resolver. How doesn't this address your case?
> But most importantly I want to be able to pass specific `Markers` with a `JSON/map` attached to logging event, which is very usefull for that specific logging event. Currently it is possible to write only the name of the marker in my case something like `LS_MARKER` (https://github.com/radut/log4j2-logstash-layout/blob/67f3ea74bf061ee27a4c5405c59745a64a663a91/layout/src/main/java/org/apache/logging/log4j/core/LogStashMarker.java) , but instead I want to override the resolver and pull the `JSON` info and write it (eventually flatten) into the same json log event.
>
> Something similar I have done in the past here : https://github.com/radut/log4j2-logstash-layout/blob/67f3ea74bf061ee27a4c5405c59745a64a663a91/demo/src/main/java/com/radut/log4j/demo/LoggerMarkerTest.java#L52 with the serializer : https://github.com/radut/log4j2-logstash-layout/blob/67f3ea74bf061ee27a4c5405c59745a64a663a91/layout/src/main/java/org/apache/logging/log4j/core/jackson/MarkerJsonSerializer.java a fork from https://github.com/majikthys/log4j2-logstash-jsonevent-layout.
>
> basically there is a `Map<String, Object>` which is serialized as JSON within a field or flattened.
Yeah... That is really a custom use case. Though good to know; appreciated that you have shared it, thank you. I will try to prioritize LOG4J2-3004 in my backlog.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] vy commented on pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
vy commented on pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472#issuecomment-791330965
So, I guess, both of us are right.
1. Yes, `JsonTemplateLayout` can serialize any object type, not only strings. `JTL` accesses to the thread context data using `LogEvent#getContextData()` method of return type `ReadOnlyStringMap`. There it uses the following method of the `ROSM`: `<V, S> void forEach(final TriConsumer<String, ? super V, S> action, final S state)`. See that here the `value` can be any `Object`. Check out MDC tests in `JsonTemplateLayoutTest` to see how there we test serialization for different types in addition to strings.
2. Yes, you cannot _easily_ put anything into MDC using `ThreadContext` class provided methods.
Note my _"easily"_ remark. I think you can do the following to put non-String values into MDC:
```java
ObjectThreadContextMap contextMap = (ObjectThreadContextMap) ThreadContext.getThreadContextMap();
contextMap.putValue("string", "foo");
contextMap.putValue("integer", 1);
contextMap.putValue("map", Collections.singletonMap("foo", "bar"));
contextMap.putValue("array", new Object[]{"foo", 0xDEADBEEF});
```
Though this will only work if `Constants.ENABLE_THREADLOCALS` is `true`:
```
boolean IS_WEB_APP = PropertiesUtil
.getProperties()
.getBooleanProperty("log4j2.is.webapp", isClassAvailable("javax.servlet.Servlet"));
boolean ENABLE_THREADLOCALS = !IS_WEB_APP &&
PropertiesUtil
.getProperties()
.getBooleanProperty("log4j2.enable.threadlocals", true);
```
**Would you mind checking if this works for you?** In the meantime, I will raise this shortcoming of `ThreadContext` in the Log4j `dev` mailing list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] radut edited a comment on pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
radut edited a comment on pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472#issuecomment-790592929
stringified doesn't seem to be working(or at least, not as I expect it to):
log4j2.xml
```
<Console name="console" target="SYSTEM_OUT" follow="true">
<JsonTemplateLayout eventTemplateUri="classpath:LogstashJsonEventLayout.json" stackTraceEnabled="true" />
<Console>
```
```json template
{
"mdc" : {
"$resolver" : "mdc",
"flatten": false,
"stringified" : false
},
"marker" : {
"$resolver" : "marker",
"field" : "name"
},
"exception" : {
"exception_class" : {
"$resolver" : "exception",
"field" : "className"
},
"exception_message" : {
"$resolver" : "exception",
"field" : "message",
"stringified" : true
},
"stacktrace" : {
"$resolver" : "exception",
"field" : "stackTrace",
"stringified" : true
}
},
"line_number" : {
"$resolver" : "source",
"field" : "lineNumber"
},
"class" : {
"$resolver" : "source",
"field" : "className"
},
"@version" : 2,
"source_host" : "${sys:local-ip}",
"message" : {
"$resolver" : "message",
"stringified" : true
},
"thread_name" : {
"$resolver" : "thread",
"field" : "name"
},
"@timestamp" : {
"$resolver" : "timestamp"
},
"level" : {
"$resolver" : "level",
"field" : "name"
},
"file" : {
"$resolver" : "source",
"field" : "fileName"
},
"method" : {
"$resolver" : "source",
"field" : "methodName"
},
"logger_name" : {
"$resolver" : "logger",
"field" : "name"
}
}
```
```
ThreadContext.push("tcVal2", 2);
```
doesn't get outputed at all
```
ThreadContext.put("tcVal2", "2"); // allows only string as value
```
the mdc outputs "tcVal2":"2" as string even if `stringified` is `true` or `false`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] radut commented on pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
radut commented on pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472#issuecomment-790592929
stringified doesn't seem to be working(or at least, not as I expect it to):
log4j2.xml
```
<Console name="console" target="SYSTEM_OUT" follow="true">
<JsonTemplateLayout eventTemplateUri="classpath:LogstashJsonEventLayout.json" stackTraceEnabled="true" />
<Console>
```
```
ThreadContext.push("tcVal2", 2);
```
doesn't get outputed at all
```
ThreadContext.put("tcVal2", "2"); // allows only string as value
```
the mdc outputs "tcVal2":"2" as string even if `stringified` is `true` or `false`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] vy commented on pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
vy commented on pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472#issuecomment-790522511
Hey @radut! This feature is something definitely on my radar, see [LOG4J2-3004](https://issues.apache.org/jira/browse/LOG4J2-3004). The reason we haven't made resolver registry public is indeed to disallow tampering this static list, until we migrate these to Log4j 2 plugins similar to the one in `PatternLayout`.
Compared to plugins, there are a couple of disadvantages of mutating a static list of resolvers.
1. They will affect all `JsonTemplateLayout` instantiations, which might not be desirable. Put another way, you cannot change them per `JsonTemplateLayout` instance.
2. It is not Log4j idiomatic – plugins are the way for extending an existing functionality.
Hence, I will sadly need to reject this change. Though feel free to submit a new PR (or re-open and update this one) adding plugin support to `JsonTemplateLayout`.
Curious about one thing: **What new resolvers were you planing to introduce?** If they address a common need, it is easier to land that into a release compared to a fully-fledged plugin support.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] radut edited a comment on pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
radut edited a comment on pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472#issuecomment-790543060
`JsonTemplateLayout` is a good idea, you could define specific constants like `"@version": 1`(number) or lets say `"source" : "log4j"`(string). What I am looking for is rewrite the mdc resolver to be able to write numbers as json numbers, instead of string literals, usefull for indexing and searching in ElasticSearch with function like between x and y, without having another transformer on the way.
But most importantly I want to be able to pass specific `Markers` with a `JSON/map` attached to logging event, which is very usefull for that specific logging event. Currently it is possible to write only the name of the marker in my case something like `LS_MARKER` (https://github.com/radut/log4j2-logstash-layout/blob/67f3ea74bf061ee27a4c5405c59745a64a663a91/layout/src/main/java/org/apache/logging/log4j/core/LogStashMarker.java) , but instead I want to override the resolver and pull the `JSON` info and write it (eventually flatten) into the same json log event.
Something similar I have done in the past here : https://github.com/radut/log4j2-logstash-layout/blob/67f3ea74bf061ee27a4c5405c59745a64a663a91/demo/src/main/java/com/radut/log4j/demo/LoggerMarkerTest.java#L52 with the serializer : https://github.com/radut/log4j2-logstash-layout/blob/67f3ea74bf061ee27a4c5405c59745a64a663a91/layout/src/main/java/org/apache/logging/log4j/core/jackson/MarkerJsonSerializer.java a fork from https://github.com/majikthys/log4j2-logstash-jsonevent-layout.
basically there is a `Map<String, Object>` which is serialized as JSON within a field or flattened.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] radut commented on pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
radut commented on pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472#issuecomment-790543060
`JsonTemplateLayout` is a good idea, you could define specific constants like `"@version": 1`(number) or lets say `"source" : "log4j"`(string). What I am looking for is rewrite the mdc resolver to be able to write numbers as json numbers, instead of string literals, usefull for indexing and searching in ElasticSearch with function like between x and y, without having another transformer on the way.
But most importantly I want to be able to pass specific `Markers` with a `JSON/map` attached to logging event, which is very usefull for that specific logging event. Currently it is possible to write only the name of the marker in my case something like `LS_MARKER` (https://github.com/radut/log4j2-logstash-layout/blob/67f3ea74bf061ee27a4c5405c59745a64a663a91/layout/src/main/java/org/apache/logging/log4j/core/LogStashMarker.java) , but instead I want to override the resolver and pull the `JSON` info and write it (eventually flatten) into the same json log event.
Something similar I have done in the past here : https://github.com/radut/log4j2-logstash-layout/blob/67f3ea74bf061ee27a4c5405c59745a64a663a91/demo/src/main/java/com/radut/log4j/demo/LoggerMarkerTest.java#L52 with the serializerd : https://github.com/radut/log4j2-logstash-layout/blob/67f3ea74bf061ee27a4c5405c59745a64a663a91/layout/src/main/java/org/apache/logging/log4j/core/jackson/MarkerJsonSerializer.java a fork from https://github.com/majikthys/log4j2-logstash-jsonevent-layout.
basically there is a `Map<String, Object>` which is serialized as JSON within a field or flattened.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] vy commented on pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
vy commented on pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472#issuecomment-791402442
I think this shortcoming has already been raised in the past, see [LOG4J2-1648](https://issues.apache.org/jira/browse/LOG4J2-1648). Apparently such a feature would have other implications. I will try to dig further into this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] vy commented on pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
vy commented on pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472#issuecomment-792336434
Yes, just set both of the `log4j2.is.webapp` and `log4j2.enable.threadlocals` system properties to `true`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] radut commented on pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
radut commented on pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472#issuecomment-792049028
@vy this `ObjectThreadContextMap` worked. Altough is there any way to still have this in a webApp ?
I didn't tried yet in a spring boot application, but there the `javax.servlet.Servlet` will be available.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [logging-log4j2] vy edited a comment on pull request #472: Allow custom resolvers to be registered for JsonTemplateLayout
Posted by GitBox <gi...@apache.org>.
vy edited a comment on pull request #472:
URL: https://github.com/apache/logging-log4j2/pull/472#issuecomment-792336434
Yes, just set `log4j2.is.webapp` to `false` and `log4j2.enable.threadlocals` to `true`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org