How to handle stack trace serialization errors

[Volkan Yazıcı <vo...@gmail.com>]

Hello,

Yesterday I've paged by a colleague for a really strange logging
behaviour on production. The logging stops working after 15m-2h.
Diving into Tomcat catalina logs pointed out that there is a
particular exception whose Throwable#printStackTrace(PrintWriter)
method is throwing an exception -- causing the death of the appender
thread, which is another issue actually. That is, trying to dump an
exception is causing an exception. This got me thinking about the
question in the subject: How shall a Layout handle stack trace
serialization failures? Dump the 2nd exception instead? Any ideas?

Kind regards.

Re: How to handle stack trace serialization errors

[Volkan Yazıcı <vo...@gmail.com>]

I gave this one a try. Go in a loop while trying to print a stack
trace, in case of an exception, set is as the effective exception, and
retry. Though when I started thinking more about this. I need to
repeat the same for class name, file name, line number, etc. Though
all these are independent resolvers, in every layout, including
PatternLayout. Consider the following snippet:

    Throwable unprintableException =
            new RuntimeException() {
                @Override
                public String getMessage() {
                    throw new RuntimeException("Doh!");
                }
            };

    LogEvent logEvent = Log4jLogEvent
            .newBuilder()
            .setLoggerName("foo")
            .setThrown(unprintableException)
            .build();

    PatternLayout layout = PatternLayout
            .newBuilder()
            .setConfiguration(new DefaultConfiguration())
            .setPattern("%ex{short.message} %ex{short.className}")
            .build();

    String message = new String(layout.toByteArray(logEvent));

Let's assume we have fixed the relevant PatternLayout converters to
execute the retry logic I have aforementioned. Then %ex{short.message}
will emit the message of RuntimeException("Doh!"), whilst
%ex{short.className} will emit the class name of unprintableException.
There is also no way to get an indication of this misalignment from
the layout output.

Hence I am really puzzled about the right approach to "the code should
be modified to catch the exception and gracefully handle it".

On Wed, Jun 17, 2020 at 5:43 PM Ralph Goers <ra...@dslextreme.com> wrote:
> I would suggest that the code should be modified to catch the exception and gracefully handle it.

Re: How to handle stack trace serialization errors

[Ralph Goers <ra...@dslextreme.com>]

Stacktraces need to provide value, otherwise they just waste log space. So whether or not a stack trace should be printed should be based on whether it is going to provide value or not. Otherwise, just logging the exception message and class would be fine.

Remember, the original message was going through the “normal” logging system. Errors in an appender should get logged via StatusLogger, which does not have the flexibility the main logging system does in formatting output, so you can’t print the stack trace and then configure it to just print a few lines.

I would suggest that the code should be modified to catch the exception and gracefully handle it. The part I don’t understand is that the AppenderControl should be catching the exception so I don’t know why that would kill the thread, unless the Throwable is an Error.

Ralph

> On Jun 17, 2020, at 7:34 AM, Volkan Yazıcı <vo...@gmail.com> wrote:
> 
> Hello,
> 
> Yesterday I've paged by a colleague for a really strange logging
> behaviour on production. The logging stops working after 15m-2h.
> Diving into Tomcat catalina logs pointed out that there is a
> particular exception whose Throwable#printStackTrace(PrintWriter)
> method is throwing an exception -- causing the death of the appender
> thread, which is another issue actually. That is, trying to dump an
> exception is causing an exception. This got me thinking about the
> question in the subject: How shall a Layout handle stack trace
> serialization failures? Dump the 2nd exception instead? Any ideas?
> 
> Kind regards.
>