You are viewing a plain text version of this content. The canonical link for it is here.
Posted to log4j-user@logging.apache.org by Shahnaz Ali <sh...@gmail.com> on 2010/04/07 14:19:11 UTC
JVM crash with Log4J.
Hi,
We have IVR application running on tomcat web application server on Redhat
linux OS, using Log4J. This is a mission critical server application with
good amount of call load.
Recently we faced JVM (tomcat) crash and major of errors before crash
pointing to Log4J. We are not expert in Log4J, though of sharing with you
and take your feedback. If you can point out the reason and corrective
action, will be great.
Required logs are uploaded here: http://www.sendspace.com/file/pnxzs6 with
password: 123=-0
Regards,
*Shahnaz Ali*
We lost in production some hourly rolled log since we use org.apache.log4j.rolling.TimeBasedRollingPolicy with gz
Posted by Sebastien Tardif <Se...@ncf.ca>.
We lost in production some hourly rolled log since we use
org.apache.log4j.rolling.TimeBasedRollingPolicy with gz. This happens for
the appender named "File".
Anybody saw that? Any solution?
It seems to happen only under load. Log4j do not leave any useful trace of
the reason other than: log4j:WARN Failure in post-close rollover action
<appender name="File" class="org.apache.log4j.rolling.RollingFileAppender">
<param name="File" value="c:/logs/ct/ct_info.log"/>
<param name="Append" value="true"/>
<param name="ImmediateFlush" value="true"/>
<param name="Encoding" value="UTF-8" />
<param name="BufferedIO" value="true"/>
<param name="BufferSize" value="1"/>
<rollingPolicy
class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
<param name="FileNamePattern"
value="c:/logs/ct/ct_info.%d{yyyy-MM-dd-HH}.log.gz"/>
</rollingPolicy>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="[%-5p] [%d{yyyy-MM-dd
HH:mm:ss,SSS}] [%x] [%m] %n"/>
</layout>
</appender>
<appender name="ThreadDumpsFile"
class="org.apache.log4j.rolling.RollingFileAppender">
<param name="File" value="c:/logs/ct/threadDumps.log"/>
<param name="Append" value="true"/>
<param name="ImmediateFlush" value="true"/>
<param name="Encoding" value="UTF-8" />
<param name="BufferedIO" value="true"/>
<param name="BufferSize" value="1"/>
<rollingPolicy
class="org.apache.log4j.rolling.TimeBasedRollingPolicy">
<param name="FileNamePattern"
value="c:/logs/ct/threadDumps.%d{yyyy-MM-dd}.log.gz"/>
</rollingPolicy>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="[%-5p] [%d{yyyy-MM-dd
HH:mm:ss,SSS}] [%x] [%m] %n"/>
</layout>
</appender>
<appender name="SocketAsyncAppender"
class="org.apache.log4j.AsyncAppender">
<param name="BufferSize" value="256"/>
<param name="Blocking" value="true"/>
<!--filter class="org.apache.log4j.filter.DenyAllFilter"/-->
<appender-ref ref="SocketAppender"/>
</appender>
<appender name="SocketAppender"
class="com.saic.ct.sys.framework.logging.CTSocketAppender">
<param name="Port" value="4445"/>
<param name="RemoteHost" value="localhost"/>
<param name="LocationInfo" value="false"/>
</appender>
<!-- This is expected to be use only to send stuff to RAD console, so
only in development -->
<appender name="ConsoleAppender"
class="org.apache.log4j.ConsoleAppender">
<param name="ImmediateFlush" value="true"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="[%-5p] [%d{yyyy-MM-dd
HH:mm:ss,SSS}] [%x] [%m] %n"/>
</layout>
</appender>
<appender name="AsyncAppender" class="org.apache.log4j.AsyncAppender">
<param name="BufferSize" value="256"/>
<param name="Blocking" value="true"/>
<appender-ref ref="File"/>
<appender-ref ref="ConsoleAppender"/>
</appender>
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: JVM crash with Log4J.
Posted by Shahnaz Ali <sh...@gmail.com>.
We have test server but how we will create actual load on that. Recreating
this problem is a challenge. After restarting, its working fine from last
around 3 weeks. One server receives 80,000 call requests in a day on
average. Since then no issue.... It is basically serving VXML for telephony
calls.
We can upgrade tomcat, Java.... but who will give guarantee this problem
wont occur again. I am going clueless. :o
And dont tell me please, forget this crash then. I can forget, customer
wont. :P
Any suggestion?
On Thu, Apr 8, 2010 at 3:09 PM, Michael Erskine <ms...@googlemail.com>wrote:
> On 8 April 2010 10:27, Shahnaz Ali <sh...@gmail.com> wrote:
> > Its not possible to do these kind of experiments in running production
> > server. But i am sure there is nothing happened like redeploying new war,
> > jar or any kind of change. Server was untouched for last several hours.
>
> Don't you have an identical test/backup server running your production
> code? :o
>
> > The reason of crash even could be out of Log4J, but i m here to discuss,
> why
> > logs are pointing to only Log4J. And more over all the errors it is
> pointing
> > should not even work after tomcat restart. Infact everything is working
> fine
> > after OS restart without any single change.
>
> Log4j logs it's problems so when underlying problems occur, they get
> logged! If your platform (JVM, application server, host OS) is
> unstable (or suspect it is), set up some test conditions and try to
> recreate the problem. I can't suggest anything more than that without
> doing your job for you :)
>
>
> Regards,
> Michael Erskine.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>
Re: JVM crash with Log4J.
Posted by Michael Erskine <ms...@googlemail.com>.
On 8 April 2010 10:27, Shahnaz Ali <sh...@gmail.com> wrote:
> Its not possible to do these kind of experiments in running production
> server. But i am sure there is nothing happened like redeploying new war,
> jar or any kind of change. Server was untouched for last several hours.
Don't you have an identical test/backup server running your production code? :o
> The reason of crash even could be out of Log4J, but i m here to discuss, why
> logs are pointing to only Log4J. And more over all the errors it is pointing
> should not even work after tomcat restart. Infact everything is working fine
> after OS restart without any single change.
Log4j logs it's problems so when underlying problems occur, they get
logged! If your platform (JVM, application server, host OS) is
unstable (or suspect it is), set up some test conditions and try to
recreate the problem. I can't suggest anything more than that without
doing your job for you :)
Regards,
Michael Erskine.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: JVM crash with Log4J.
Posted by Shahnaz Ali <sh...@gmail.com>.
Hi,
Its not possible to do these kind of experiments in running production
server. But i am sure there is nothing happened like redeploying new war,
jar or any kind of change. Server was untouched for last several hours.
Logs are not fabricated, yeah i took only the portion when problem started.
and the JVM crash dump.
The reason of crash even could be out of Log4J, but i m here to discuss, why
logs are pointing to only Log4J. And more over all the errors it is pointing
should not even work after tomcat restart. Infact everything is working fine
after OS restart without any single change.
BR,
Shahnaz Ali.
On Thu, Apr 8, 2010 at 1:47 PM, Michael Erskine <ms...@googlemail.com>wrote:
> On 8 April 2010 07:47, Shahnaz Ali <sh...@gmail.com> wrote:
> > No, there was no any hot deployment.
>
> But the evidence points to a change of class so something equivalent
> to a redeployment must have happened. Something caused a deployed
> class to be reloaded and that operation failed due to a class
> incompatibility. That is what happened. Unless the logs are somehow
> fabricated :)
>
> The cause of the crash however could be anything! Have a try at
> repeating it by dropping newer Wars and Jars. Upgrade your Java
> runtime to apply all the officially available fixes. Quit running
> Tomcat as root :)
>
> Regards,
> Michael Erskine.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>
Re: JVM crash with Log4J.
Posted by Michael Erskine <ms...@googlemail.com>.
On 8 April 2010 07:47, Shahnaz Ali <sh...@gmail.com> wrote:
> No, there was no any hot deployment.
But the evidence points to a change of class so something equivalent
to a redeployment must have happened. Something caused a deployed
class to be reloaded and that operation failed due to a class
incompatibility. That is what happened. Unless the logs are somehow
fabricated :)
The cause of the crash however could be anything! Have a try at
repeating it by dropping newer Wars and Jars. Upgrade your Java
runtime to apply all the officially available fixes. Quit running
Tomcat as root :)
Regards,
Michael Erskine.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: JVM crash with Log4J.
Posted by Shahnaz Ali <sh...@gmail.com>.
No, there was no any hot deployment. We deploy our application WAR file and
restart tomcat. This is a must for us. And there was no any change to Log4J
library jar file.
BR,
Shahnaz Ali.
On Thu, Apr 8, 2010 at 12:13 PM, Yair Ogen <ya...@gmail.com> wrote:
> Hot deploy is when you load a new version of a war for example while the
> server is still running. This triggers a chain of init actions on the
> server
> side.
>
> I see from the logs the doGet method in the servlet caused a init of log4j
> configuration.
>
> try looking in that direction. I think this should not happen.
>
> The exception logged indicates a class file was changed in the middle of
> using it (or something like that). these things sometime happen when using
> hot deploy.
>
> On Wed, Apr 7, 2010 at 4:40 PM, Shahnaz Ali <sh...@gmail.com> wrote:
>
> > Hi,
> >
> > Thanks for your quick response.
> >
> > This crash happens only once so far in a year. After restart of OS and
> > Tomcat, without any change system is working fine again. This is a telco
> > system, so they need to know reason of crash. And if possible solution of
> > that.
> >
> > Actually this is a server side application which receives calls. At one
> > time
> > it may receive around 150 concurrent calls. When I check logs, i notice
> > there were around 900 call sessions. No clue how it happened.
> >
> > Application is in production, so we just cant change the configuration so
> > easily. We haven't tried with Java 6.
> >
> > Current Log4J version is: log4j-1.2.8.jar
> > Java Version is: 1.5
> > OS: Redhat Linux
> >
> > Please explain, what do you mean by "hot deploy" here?
> >
> >
> > BR,
> > Shahnaz Ali.
> >
> >
> >
> > On Wed, Apr 7, 2010 at 6:10 PM, Michael Erskine <msemtd@googlemail.com
> > >wrote:
> >
> > > On 7 April 2010 13:19, Shahnaz Ali <sh...@gmail.com> wrote:
> > > > We have IVR application running on tomcat web application server on
> > > Redhat
> > > > linux OS, using Log4J. This is a mission critical server application
> > with
> > > > good amount of call load.
> > > >
> > > > Recently we faced JVM (tomcat) crash and major of errors before crash
> > > > pointing to Log4J. We are not expert in Log4J, though of sharing with
> > you
> > > > and take your feedback. If you can point out the reason and
> corrective
> > > > action, will be great.
> > >
> > > Tomcat running as root? 1080 threads? :o
> > >
> > > I can see the PropertyConfigurator failing repeatedly - once for each
> > > HTTP daemon?
> > > Is this something to do with...
> > > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> > >
> > > Does this Log4j configuration usually work?
> > > Have you tried running on a recent version of Java 6?
> > >
> > > Regards,
> > > Michael Erskine.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> > > For additional commands, e-mail: log4j-user-help@logging.apache.org
> > >
> > >
> >
>
Re: JVM crash with Log4J.
Posted by Yair Ogen <ya...@gmail.com>.
Hot deploy is when you load a new version of a war for example while the
server is still running. This triggers a chain of init actions on the server
side.
I see from the logs the doGet method in the servlet caused a init of log4j
configuration.
try looking in that direction. I think this should not happen.
The exception logged indicates a class file was changed in the middle of
using it (or something like that). these things sometime happen when using
hot deploy.
On Wed, Apr 7, 2010 at 4:40 PM, Shahnaz Ali <sh...@gmail.com> wrote:
> Hi,
>
> Thanks for your quick response.
>
> This crash happens only once so far in a year. After restart of OS and
> Tomcat, without any change system is working fine again. This is a telco
> system, so they need to know reason of crash. And if possible solution of
> that.
>
> Actually this is a server side application which receives calls. At one
> time
> it may receive around 150 concurrent calls. When I check logs, i notice
> there were around 900 call sessions. No clue how it happened.
>
> Application is in production, so we just cant change the configuration so
> easily. We haven't tried with Java 6.
>
> Current Log4J version is: log4j-1.2.8.jar
> Java Version is: 1.5
> OS: Redhat Linux
>
> Please explain, what do you mean by "hot deploy" here?
>
>
> BR,
> Shahnaz Ali.
>
>
>
> On Wed, Apr 7, 2010 at 6:10 PM, Michael Erskine <msemtd@googlemail.com
> >wrote:
>
> > On 7 April 2010 13:19, Shahnaz Ali <sh...@gmail.com> wrote:
> > > We have IVR application running on tomcat web application server on
> > Redhat
> > > linux OS, using Log4J. This is a mission critical server application
> with
> > > good amount of call load.
> > >
> > > Recently we faced JVM (tomcat) crash and major of errors before crash
> > > pointing to Log4J. We are not expert in Log4J, though of sharing with
> you
> > > and take your feedback. If you can point out the reason and corrective
> > > action, will be great.
> >
> > Tomcat running as root? 1080 threads? :o
> >
> > I can see the PropertyConfigurator failing repeatedly - once for each
> > HTTP daemon?
> > Is this something to do with...
> > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> >
> > Does this Log4j configuration usually work?
> > Have you tried running on a recent version of Java 6?
> >
> > Regards,
> > Michael Erskine.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> > For additional commands, e-mail: log4j-user-help@logging.apache.org
> >
> >
>
Re: JVM crash with Log4J.
Posted by Shahnaz Ali <sh...@gmail.com>.
Hi,
Thanks for your quick response.
This crash happens only once so far in a year. After restart of OS and
Tomcat, without any change system is working fine again. This is a telco
system, so they need to know reason of crash. And if possible solution of
that.
Actually this is a server side application which receives calls. At one time
it may receive around 150 concurrent calls. When I check logs, i notice
there were around 900 call sessions. No clue how it happened.
Application is in production, so we just cant change the configuration so
easily. We haven't tried with Java 6.
Current Log4J version is: log4j-1.2.8.jar
Java Version is: 1.5
OS: Redhat Linux
Please explain, what do you mean by "hot deploy" here?
BR,
Shahnaz Ali.
On Wed, Apr 7, 2010 at 6:10 PM, Michael Erskine <ms...@googlemail.com>wrote:
> On 7 April 2010 13:19, Shahnaz Ali <sh...@gmail.com> wrote:
> > We have IVR application running on tomcat web application server on
> Redhat
> > linux OS, using Log4J. This is a mission critical server application with
> > good amount of call load.
> >
> > Recently we faced JVM (tomcat) crash and major of errors before crash
> > pointing to Log4J. We are not expert in Log4J, though of sharing with you
> > and take your feedback. If you can point out the reason and corrective
> > action, will be great.
>
> Tomcat running as root? 1080 threads? :o
>
> I can see the PropertyConfigurator failing repeatedly - once for each
> HTTP daemon?
> Is this something to do with...
> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
>
> Does this Log4j configuration usually work?
> Have you tried running on a recent version of Java 6?
>
> Regards,
> Michael Erskine.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
> For additional commands, e-mail: log4j-user-help@logging.apache.org
>
>
Re: JVM crash with Log4J.
Posted by Michael Erskine <ms...@googlemail.com>.
On 7 April 2010 13:19, Shahnaz Ali <sh...@gmail.com> wrote:
> We have IVR application running on tomcat web application server on Redhat
> linux OS, using Log4J. This is a mission critical server application with
> good amount of call load.
>
> Recently we faced JVM (tomcat) crash and major of errors before crash
> pointing to Log4J. We are not expert in Log4J, though of sharing with you
> and take your feedback. If you can point out the reason and corrective
> action, will be great.
Tomcat running as root? 1080 threads? :o
I can see the PropertyConfigurator failing repeatedly - once for each
HTTP daemon?
Is this something to do with...
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
Does this Log4j configuration usually work?
Have you tried running on a recent version of Java 6?
Regards,
Michael Erskine.
---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-user-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-user-help@logging.apache.org
Re: JVM crash with Log4J.
Posted by Yair Ogen <ya...@gmail.com>.
are you using hot deploy on Tomcat?
which version of log4j are you using?
On Wed, Apr 7, 2010 at 3:19 PM, Shahnaz Ali <sh...@gmail.com> wrote:
> Hi,
>
> We have IVR application running on tomcat web application server on Redhat
> linux OS, using Log4J. This is a mission critical server application with
> good amount of call load.
>
> Recently we faced JVM (tomcat) crash and major of errors before crash
> pointing to Log4J. We are not expert in Log4J, though of sharing with you
> and take your feedback. If you can point out the reason and corrective
> action, will be great.
>
> Required logs are uploaded here: http://www.sendspace.com/file/pnxzs6
> with
> password: 123=-0
>
> Regards,
> *Shahnaz Ali*
>