You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-user@logging.apache.org by Ganapathi Vara Prasad <ga...@gmail.com> on 2022/11/16 10:54:49 UTC
Re: PatternLayout Vs JsonTemplateLayout/JsonLayout Performance Comparision
Hello Team,
Our log4j2 configuration uses *PatternLayout* and we are exploring
switching to JsonLayout for simplifying integration with ELK. I am trying
to find any performance comparison between pattern and json layouts but
couldn't find any. I could find only the JsonTemplateLayout benchmark in
the log4j-perf module.
I wanted to know:
- Is it a reasonable thing to compare the performance of these two
layouts?
- Are there any existing resources comparing the performance of these
two layouts?
- What considerations should we make if we switch to JsonTemplateLayout?
Thank you for the help.
--Ganapati
Re: PatternLayout Vs JsonTemplateLayout/JsonLayout Performance Comparision
Posted by Matt Sicker <ma...@musigma.org>.
The main purpose of the JSON encoding feature in PatternLayout is to escape quotes, backslashes, and other special characters interpreted by JSON (along with encoding characters that require encoding like newlines and certain control characters). It long predates JTL which already handles escaping the output of JSON natively.
> On Nov 17, 2022, at 1:58 AM, Volkan Yazıcı <vo...@yazi.ci> wrote:
>
> Hey Ganapathi!
>
> First, `JsonLayout` (JL) and `JsonTemplateLayout` (JTL) are two different
> beasts, but more importantly, JTL is the successor of JL. We are keeping JL
> around only for backward-compatibility reasons. All JSON-based log needs
> are advised to be sufficed using JTL.
>
> JTL is heavily optimized and battle-tested. I, as one of JTL developers,
> can assure you that it is blazing fast. Similar to `PatternLayout` (PL),
> JTL uses garbage-free formatting constructs wherever possible. Hence, I
> would expect JTLs performance to be on par with PL, if not faster! If you
> or anybody happens to find a case where JTL performs worse than PL, I will
> volunteer to fix it ASAP!
>
>> Is it a reasonable thing to compare the performance of these two layouts?
>
> Not really. Since JTL produces extra JSON-structure which is not there in
> PL. This said, judging from my experience, I would expect JTL to perform on
> par in most cases.
>
>> Are there any existing resources comparing the performance of these two
> layouts?
>
> Nope. But feel free to do your own comparison by tweaking
> `JsonTemplateLayoutBenchmark` in `log4j-perf`.
>
>> What considerations should we make if we switch to JsonTemplateLayout?
>
> The only thing you need to keep in mind while using JTL is that resolvers
> incurring garbage (e.g., using regex in stack trace truncation) will mostly
> determine the overall latency. Since a majority of the resolvers are
> garbage-free, those will complete instantly. Though this advice more or
> less applies to any Java code, including PL.
>
> If you happen to have any questions, feedback, feature/change requests,
> etc. feel free to share them here.
>
> Cheers!
>
> On Wed, Nov 16, 2022 at 11:55 AM Ganapathi Vara Prasad <
> ganapathibasamsetti@gmail.com> wrote:
>
>> Hello Team,
>>
>> Our log4j2 configuration uses *PatternLayout* and we are exploring
>> switching to JsonLayout for simplifying integration with ELK. I am trying
>> to find any performance comparison between pattern and json layouts but
>> couldn't find any. I could find only the JsonTemplateLayout benchmark in
>> the log4j-perf module.
>>
>> I wanted to know:
>>
>> - Is it a reasonable thing to compare the performance of these two
>> layouts?
>> - Are there any existing resources comparing the performance of these
>> two layouts?
>> - What considerations should we make if we switch to JsonTemplateLayout?
>>
>> Thank you for the help.
>>
>> --Ganapati
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: PatternLayout Vs JsonTemplateLayout/JsonLayout Performance Comparision
Posted by Volkan Yazıcı <vo...@yazi.ci>.
Hey Ganapathi!
First, `JsonLayout` (JL) and `JsonTemplateLayout` (JTL) are two different
beasts, but more importantly, JTL is the successor of JL. We are keeping JL
around only for backward-compatibility reasons. All JSON-based log needs
are advised to be sufficed using JTL.
JTL is heavily optimized and battle-tested. I, as one of JTL developers,
can assure you that it is blazing fast. Similar to `PatternLayout` (PL),
JTL uses garbage-free formatting constructs wherever possible. Hence, I
would expect JTLs performance to be on par with PL, if not faster! If you
or anybody happens to find a case where JTL performs worse than PL, I will
volunteer to fix it ASAP!
> Is it a reasonable thing to compare the performance of these two layouts?
Not really. Since JTL produces extra JSON-structure which is not there in
PL. This said, judging from my experience, I would expect JTL to perform on
par in most cases.
> Are there any existing resources comparing the performance of these two
layouts?
Nope. But feel free to do your own comparison by tweaking
`JsonTemplateLayoutBenchmark` in `log4j-perf`.
> What considerations should we make if we switch to JsonTemplateLayout?
The only thing you need to keep in mind while using JTL is that resolvers
incurring garbage (e.g., using regex in stack trace truncation) will mostly
determine the overall latency. Since a majority of the resolvers are
garbage-free, those will complete instantly. Though this advice more or
less applies to any Java code, including PL.
If you happen to have any questions, feedback, feature/change requests,
etc. feel free to share them here.
Cheers!
On Wed, Nov 16, 2022 at 11:55 AM Ganapathi Vara Prasad <
ganapathibasamsetti@gmail.com> wrote:
> Hello Team,
>
> Our log4j2 configuration uses *PatternLayout* and we are exploring
> switching to JsonLayout for simplifying integration with ELK. I am trying
> to find any performance comparison between pattern and json layouts but
> couldn't find any. I could find only the JsonTemplateLayout benchmark in
> the log4j-perf module.
>
> I wanted to know:
>
> - Is it a reasonable thing to compare the performance of these two
> layouts?
> - Are there any existing resources comparing the performance of these
> two layouts?
> - What considerations should we make if we switch to JsonTemplateLayout?
>
> Thank you for the help.
>
> --Ganapati
>
Re: PatternLayout Vs JsonTemplateLayout/JsonLayout Performance Comparision
Posted by Ganapathi Vara Prasad <ga...@gmail.com>.
Hi Christopher,
At least in my experience, I haven't any issues when logging JSON messages
using PatternLayout. What other things should we consider apart from this?
--Ganapati
On Wed, Nov 16, 2022 at 10:39 PM Christopher Schultz <
chris@christopherschultz.net> wrote:
> Ganapati,
>
> On 11/16/22 05:54, Ganapathi Vara Prasad wrote:
> > Hello Team,
> >
> > Our log4j2 configuration uses *PatternLayout* and we are exploring
> > switching to JsonLayout for simplifying integration with ELK. I am trying
> > to find any performance comparison between pattern and json layouts but
> > couldn't find any. I could find only the JsonTemplateLayout benchmark in
> > the log4j-perf module.
> >
> > I wanted to know:
> >
> > - Is it a reasonable thing to compare the performance of these two
> > layouts?
> > - Are there any existing resources comparing the performance of these
> > two layouts?
> > - What considerations should we make if we switch to
> JsonTemplateLayout?
>
> Is it even possible to use PatternLayout safely with JSON-encoded data?
> What if you log a username which contains a " character? Won't that
> completely break your JSON?
>
> I think performance shouldn't be your primary consideration, here.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>
Re: PatternLayout Vs JsonTemplateLayout/JsonLayout Performance Comparision
Posted by Ralph Goers <ra...@dslextreme.com>.
Personally, I wouldn’t recommend using PatternLayout to render JSON. There are too many things that can go wrong and it is very limiting to do that.
Ralph
> On Nov 16, 2022, at 10:22 AM, Carter Kozak <ck...@ckozak.net> wrote:
>
> On Wed, Nov 16, 2022, at 12:09, Christopher Schultz wrote:
>> Is it even possible to use PatternLayout safely with JSON-encoded data?
>> What if you log a username which contains a " character? Won't that
>> completely break your JSON?
>
> This can be done using the %encode pattern described in the documentation:
> https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternLayout
> It provides the following example: {"message": "%enc{%m}{JSON}"}
>
>> I think performance shouldn't be your primary consideration, here.
>
> I agree wholeheartedly with this sentiment, but we should remember
> that some users have a different set of constraints and requirements.
>
> Carter Kozak
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: PatternLayout Vs JsonTemplateLayout/JsonLayout Performance Comparision
Posted by Carter Kozak <ck...@ckozak.net>.
On Wed, Nov 16, 2022, at 12:09, Christopher Schultz wrote:
> Is it even possible to use PatternLayout safely with JSON-encoded data?
> What if you log a username which contains a " character? Won't that
> completely break your JSON?
This can be done using the %encode pattern described in the documentation:
https://logging.apache.org/log4j/2.x/manual/layouts.html#PatternLayout
It provides the following example: {"message": "%enc{%m}{JSON}"}
> I think performance shouldn't be your primary consideration, here.
I agree wholeheartedly with this sentiment, but we should remember
that some users have a different set of constraints and requirements.
Carter Kozak
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: PatternLayout Vs JsonTemplateLayout/JsonLayout Performance Comparision
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Ganapati,
On 11/16/22 05:54, Ganapathi Vara Prasad wrote:
> Hello Team,
>
> Our log4j2 configuration uses *PatternLayout* and we are exploring
> switching to JsonLayout for simplifying integration with ELK. I am trying
> to find any performance comparison between pattern and json layouts but
> couldn't find any. I could find only the JsonTemplateLayout benchmark in
> the log4j-perf module.
>
> I wanted to know:
>
> - Is it a reasonable thing to compare the performance of these two
> layouts?
> - Are there any existing resources comparing the performance of these
> two layouts?
> - What considerations should we make if we switch to JsonTemplateLayout?
Is it even possible to use PatternLayout safely with JSON-encoded data?
What if you log a username which contains a " character? Won't that
completely break your JSON?
I think performance shouldn't be your primary consideration, here.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: PatternLayout Vs JsonTemplateLayout/JsonLayout Performance Comparision
Posted by Ganapathi Vara Prasad <ga...@gmail.com>.
Hi Ralph,
Thank you for the answers.
--Ganapati
On Wed, Nov 16, 2022 at 9:26 PM Ralph Goers <ra...@dslextreme.com>
wrote:
> 1. We need to mark JsonLayout as deprecated. We no longer recommend using
> it. Use JsonTemplateLayout instead.
> 2. The only consideration for JsonTemplateLayout is that you format the
> template correctly.
>
> FWIW, my employer has been using JsonTemplateLayout to log to ELK for
> quite a while now without any problems. Note that you can still use a
> pattern to format the message. This works very well. As you can see by the
> example below you can even include variables to add extra variables to be
> logged from the ThreadContextMap via overrides.
>
> {
> "@timestamp": {
> "$resolver": "timestamp",
> "pattern": {
> "format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
> "timeZone": "UTC"
> }
> },
> "ecs.version": "1.11.0",
> "log.level": {
> "$resolver": "level",
> "field": "name"
> },
> "application": "\${lower:\${spring:spring.application.name}}",
> "short_message": {
> "$resolver": "message",
> "stringified": true
> },
> "message": {
> "$resolver": "pattern",
> "pattern": "[%t] %X{requestId, sessionId, loginId, ipAddress,
> accountNumber, \${extraContextVars}} %C{1.}.%M:%L - %m%n"
> },
> "process.thread.name": {
> "$resolver": "thread",
> "field": "name"
> },
> "log.logger": {
> "$resolver": "logger",
> "field": "name"
> },
> "event.action": {
> "$resolver": "marker",
> "field": "name"
> },
> "event.data": {
> "$resolver": "map",
> "stringified": true
> },
> "labels": {
> "$resolver": "mdc",
> "flatten": true,
> "stringified": true,
> "pattern": "(?!(?i:token)).+"
> },
> "tags": {
> "$resolver": "ndc"
> },
> "error.type": {
> "$resolver": "exception",
> "field": "className"
> },
> "error.message": {
> "$resolver": "exception",
> "field": "message"
> },
> "error.stack_trace": {
> "$resolver": "exception",
> "field": "stackTrace",
> "stackTrace": {
> "stringified": true
> }
> }
> }
>
>
> Ralph
>
> > On Nov 16, 2022, at 3:54 AM, Ganapathi Vara Prasad <
> ganapathibasamsetti@gmail.com> wrote:
> >
> > Hello Team,
> >
> > Our log4j2 configuration uses *PatternLayout* and we are exploring
> > switching to JsonLayout for simplifying integration with ELK. I am trying
> > to find any performance comparison between pattern and json layouts but
> > couldn't find any. I could find only the JsonTemplateLayout benchmark in
> > the log4j-perf module.
> >
> > I wanted to know:
> >
> > - Is it a reasonable thing to compare the performance of these two
> > layouts?
> > - Are there any existing resources comparing the performance of these
> > two layouts?
> > - What considerations should we make if we switch to
> JsonTemplateLayout?
> >
> > Thank you for the help.
> >
> > --Ganapati
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>
Re: PatternLayout Vs JsonTemplateLayout/JsonLayout Performance Comparision
Posted by Ralph Goers <ra...@dslextreme.com>.
1. We need to mark JsonLayout as deprecated. We no longer recommend using it. Use JsonTemplateLayout instead.
2. The only consideration for JsonTemplateLayout is that you format the template correctly.
FWIW, my employer has been using JsonTemplateLayout to log to ELK for quite a while now without any problems. Note that you can still use a pattern to format the message. This works very well. As you can see by the example below you can even include variables to add extra variables to be logged from the ThreadContextMap via overrides.
{
"@timestamp": {
"$resolver": "timestamp",
"pattern": {
"format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
"timeZone": "UTC"
}
},
"ecs.version": "1.11.0",
"log.level": {
"$resolver": "level",
"field": "name"
},
"application": "\${lower:\${spring:spring.application.name}}",
"short_message": {
"$resolver": "message",
"stringified": true
},
"message": {
"$resolver": "pattern",
"pattern": "[%t] %X{requestId, sessionId, loginId, ipAddress, accountNumber, \${extraContextVars}} %C{1.}.%M:%L - %m%n"
},
"process.thread.name": {
"$resolver": "thread",
"field": "name"
},
"log.logger": {
"$resolver": "logger",
"field": "name"
},
"event.action": {
"$resolver": "marker",
"field": "name"
},
"event.data": {
"$resolver": "map",
"stringified": true
},
"labels": {
"$resolver": "mdc",
"flatten": true,
"stringified": true,
"pattern": "(?!(?i:token)).+"
},
"tags": {
"$resolver": "ndc"
},
"error.type": {
"$resolver": "exception",
"field": "className"
},
"error.message": {
"$resolver": "exception",
"field": "message"
},
"error.stack_trace": {
"$resolver": "exception",
"field": "stackTrace",
"stackTrace": {
"stringified": true
}
}
}
Ralph
> On Nov 16, 2022, at 3:54 AM, Ganapathi Vara Prasad <ga...@gmail.com> wrote:
>
> Hello Team,
>
> Our log4j2 configuration uses *PatternLayout* and we are exploring
> switching to JsonLayout for simplifying integration with ELK. I am trying
> to find any performance comparison between pattern and json layouts but
> couldn't find any. I could find only the JsonTemplateLayout benchmark in
> the log4j-perf module.
>
> I wanted to know:
>
> - Is it a reasonable thing to compare the performance of these two
> layouts?
> - Are there any existing resources comparing the performance of these
> two layouts?
> - What considerations should we make if we switch to JsonTemplateLayout?
>
> Thank you for the help.
>
> --Ganapati
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org