You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-user@logging.apache.org by dh...@yahoo.com on 2008/11/16 23:47:17 UTC
Problems with rollover of RollingFileAppender - data loss
Hi,
I'm running log4j-1.2.15 and have a problem that I'd like to put forward to see if anyone has a solution.
I have an application that is very short lived but is run many times. It's invoked by an external scheduling and orchestration application that can only run the application by constructing a fresh JVM via JNI every time it wants to run it. The application logs using log4j and its RollingFileAppender with append=true.
The problem that I have encountered is that I have observed that sometimes the log files are getting partially wiped. I am putting this down to the fact that the JVM might be shutting down whilst the rolling activity is taking place - resulting in partially rolled files. I have tried adding a shutdown hook to invoke LogManager.shutdown() with the hope that this would block until the appenders had finished doing their work before the JVM shutdown but this doesn't appear to work. I was able to replicate this by invoking the application many times continually. It doesn't happen often (maybe 1% with my setup) - I think the conditions have to be right, i.e. start rolling just before the app shuts down.
Has anyone got any ideas, solutions or workarounds?
Many thanks,
Dave
Re: Problems with rollover of RollingFileAppender - data loss
Posted by Maarten Bosteels <mb...@gmail.com>.
what about using two processes:
* a long-lived process that receives all logging events via a SocketNode,
see [1] and writes them file via a FileAppender
* the short lived program that you already have, using a SocketAppender to
send events to the above process
[1]
http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/net/SimpleSocketServer.html
Maarten
On Sun, Nov 16, 2008 at 11:47 PM, <dh...@yahoo.com> wrote:
> Hi,
>
> I'm running log4j-1.2.15 and have a problem that I'd like to put forward to
> see if anyone has a solution.
>
> I have an application that is very short lived but is run many times. It's
> invoked by an external scheduling and orchestration application that can
> only run the application by constructing a fresh JVM via JNI every time it
> wants to run it. The application logs using log4j and its
> RollingFileAppender with append=true.
>
> The problem that I have encountered is that I have observed that sometimes
> the log files are getting partially wiped. I am putting this down to the
> fact that the JVM might be shutting down whilst the rolling activity is
> taking place - resulting in partially rolled files. I have tried adding a
> shutdown hook to invoke LogManager.shutdown() with the hope that this would
> block until the appenders had finished doing their work before the JVM
> shutdown but this doesn't appear to work. I was able to replicate this by
> invoking the application many times continually. It doesn't happen often
> (maybe 1% with my setup) - I think the conditions have to be right, i.e.
> start rolling just before the app shuts down.
>
> Has anyone got any ideas, solutions or workarounds?
>
> Many thanks,
>
> Dave
>
>
>
>
Re: Problems with rollover of RollingFileAppender - data loss
Posted by dh...@yahoo.com.
Thanks for your replies. I was trying to avoid having another dependency in the system by having an external server. Also, should that server go down, the logs will be lost as the appender will just ditch the messages that it can't send so it doesn't block the application - and we don't want to lose data - it has to be reliable.
I'll try the -Dlog4j.debug property and see what I can get out of it and try and create a standalone test harness to do it.
Many thanks,
Dave
________________________________
From: Thorbjørn Ravn Andersen <th...@gmail.com>
To: Log4J Users List <lo...@logging.apache.org>
Sent: Monday, 17 November, 2008 12:11:01
Subject: Re: Problems with rollover of RollingFileAppender - data loss
dhallammail-log4j@yahoo.com skrev den 16-11-2008 23:47:
> The problem that I have encountered is that I have observed that sometimes the log files are getting partially wiped. I am putting this down to the fact that the JVM might be shutting down whilst the rolling activity is taking place - resulting in partially rolled files. I have tried adding a shutdown
The rolling happens by renaming current files, not by copying contents from one file to another. (See RollingFileAppender.rollOver() for code).
To me this sounds like that the log file is not properly closed when the application shuts down.
An idea might be running the application with the "log.debug" property set, so you can see what is actually happening (this means you need to save the console output somewhere)? The suggested approach with doing socket appends might also be quite usable for you.
/Thorbjørn
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: Problems with rollover of RollingFileAppender - data loss
Posted by Thorbjørn Ravn Andersen <th...@gmail.com>.
dhallammail-log4j@yahoo.com skrev den 16-11-2008 23:47:
> The problem that I have encountered is that I have observed that sometimes the log files are getting partially wiped. I am putting this down to the fact that the JVM might be shutting down whilst the rolling activity is taking place - resulting in partially rolled files. I have tried adding a shutdown
The rolling happens by renaming current files, not by copying contents
from one file to another. (See RollingFileAppender.rollOver() for code).
To me this sounds like that the log file is not properly closed when the
application shuts down.
An idea might be running the application with the "log.debug" property
set, so you can see what is actually happening (this means you need to
save the console output somewhere)? The suggested approach with doing
socket appends might also be quite usable for you.
/Thorbjørn
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org