You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Elias Kopsiaftis <ye...@gmail.com> on 2014/04/02 14:15:00 UTC
tomcat randomly undeploys and redeploys the applications
Im having a strange problem and I have googled it for a while but have
found nothing. I deploy the application, then in the log file catalina.out
i get many messages from WebappClassLoader clearReferencesThreads saying
threads appear to have started but have failed to stop, then finally, Ill
get a message from HostConfig checkResources that says its undeploying the
context, and then it redeploys. I have no idea why this is happening or how
to trouble shoot it. Any suggestions would be greatly appreciated
Re: tomcat randomly undeploys and redeploys the applications
Posted by Elias Kopsiaftis <ye...@gmail.com>.
Chuck,
Thanks for your response. I will get you all the information you asked for
later tonight when I get out of work. In the meantime, the only threads our
application creates are those created by a ScheduledThreadPoolExecutor, and
in fact, when I do a thread dump to analyze what is going on, there are 30
threads associated with this executor that are in a BLOCKED state. We have
had trouble with executors in the past, but these threads are supposed to
be running for the lifecycle of the application. Is there a problem using
an executor in a tomcat application? I will provide you with the following
later today: Tomcat version, JRE version, platform, server.xml (with
comments removed), context.xml for the webapp, etc. Would you like me to
also attach the thread dump?
On Wed, Apr 2, 2014 at 8:21 AM, Caldarale, Charles R <
Chuck.Caldarale@unisys.com> wrote:
> > From: Elias Kopsiaftis [mailto:yemista@gmail.com]
> > Subject: tomcat randomly undeploys and redeploys the applications
>
> > I deploy the application, then in the log file catalina.out i get many
> > messages from WebappClassLoader clearReferencesThreads saying threads
> > appear to have started but have failed to stop
>
> This is an indication that your webapp is not managing its threads
> properly.
>
> > then finally, Ill get a message from HostConfig checkResources that
> > says its undeploying the context, and then it redeploys.
>
> This is sometimes caused by incorrect timestamps on the bits of the webapp
> that Tomcat monitors, or an incorrect clock setting on the system Tomcat is
> running on. The mismatch makes it appear that the webapp is being updated
> continuously.
>
> If you want anything more than speculation, you'll need to provide real
> information: Tomcat version, JRE version, platform, server.xml (with
> comments removed), context.xml for the webapp, etc.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
> MATERIAL and is thus for use only by the intended recipient. If you
> received this in error, please contact the sender and delete the e-mail and
> its attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: tomcat randomly undeploys and redeploys the applications
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Elias,
On 4/2/14, 4:12 PM, Elias Kopsiaftis wrote:
> Thats very interesting. When you say update, what do you mean
> technically?
I mean a build that triggers new class files, libraries, or some other
"watched" resource to trigger an automatic reload. If you want to
disable automatic reloads completely, set reloadable="false" on your
<Context> (which is actually the default).
If you want to still be able to reload your web application sometimes,
consider installing the Manager webapp, which will let you do reloads,
etc. via a text/HTTP or GUI interface.
- -chris
> On Wed, Apr 2, 2014 at 4:05 PM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
> Chuck,
>
> On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
>>>>> From: Elias Kopsiaftis [mailto:yemista@gmail.com] Subject:
>>>>> tomcat randomly undeploys and redeploys the applications
>>>>
>>>>> I deploy the application, then in the log file catalina.out
>>>>> i get many messages from WebappClassLoader
>>>>> clearReferencesThreads saying threads appear to have
>>>>> started but have failed to stop
>>>>
>>>> This is an indication that your webapp is not managing its
>>>> threads properly.
>>>>
>>>>> then finally, Ill get a message from HostConfig
>>>>> checkResources that says its undeploying the context, and
>>>>> then it redeploys.
>>>>
>>>> This is sometimes caused by incorrect timestamps on the bits
>>>> of the webapp that Tomcat monitors, or an incorrect clock
>>>> setting on the system Tomcat is running on. The mismatch
>>>> makes it appear that the webapp is being updated
>>>> continuously.
>
> I've found that in development, a single update can cause Tomcat to
> go into a loop of redeployments, re-deploying my web application
> every few seconds or so. If I let it go, it does finally stop
> reloading and settle down.
>
> I believe Tomcat will emit a log message informing the
> administrator which file has triggered the reload, but I think it's
> at the DEBUG log level.
>
> -chris
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTPHFUAAoJEBzwKT+lPKRYwKEQALm1X3IqSHJLcLRyeM41gDeh
vcnu1zgUP1KqGJeV1BZHq3aREUBELdcpCncvVALd4xuJT+1JR6qDbHRINNSUAEbF
JNx8j0r+/Y2eYFFwWt4gNa+7USSxvy7xzz0Sj726U74MCoiSQULKB/53VNn8tPPA
GhNsZcIcclYaaaDyEV5OQ0joyLwrVXo7AWbvsh82AJmpoQ0WnkKmsX40u4rTbMfb
bb3JLnDEs93WGbYzKd7UfXxs9gaVFLg+TFeBP+hNiAfRz2JFiBiQnpwsNBURA0lo
SySFylSVXI2JjHexaT4N+OR3ZB4d2p4eEwOA1Cv9ZGEU/9wH6cTIykDsL2SYWubj
6+3VeGL2xepq6PYxaniDaRLx+dxcEOs97dVxHS/w+M5VzUm2LZm7O9vXuhtQTLGt
WIOhp5ffpvYVvLhuDoxm7XmPMnMH4kBOjc5Cz4v3NgL9eVUJIRWJxWPkdqnZFiU/
S9pQmWl6BBgt5gOUUjLu8almvYWoi8ogGt8Sv4iX1EYdGrGpU1eyqxlHV3C2pR58
eBUqaVE4GLaa0H8ZknMm0leLRdzdhMaEUDbPBdHl2XfCo9+CkqX059zqp6UPFiFr
ap5UgkFl8llobJoxLM1hbE635Wxg26LpxagQ9kUR1wWHrlyIeldrEV0dQtfuvqf5
b7KIADcRBXm5hTpRTXap
=EZOG
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Posted by Elias Kopsiaftis <ye...@gmail.com>.
Thats very interesting. When you say update, what do you mean technically?
On Wed, Apr 2, 2014 at 4:05 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Chuck,
>
> On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
> >> From: Elias Kopsiaftis [mailto:yemista@gmail.com] Subject: tomcat
> >> randomly undeploys and redeploys the applications
> >
> >> I deploy the application, then in the log file catalina.out i get
> >> many messages from WebappClassLoader clearReferencesThreads
> >> saying threads appear to have started but have failed to stop
> >
> > This is an indication that your webapp is not managing its threads
> > properly.
> >
> >> then finally, Ill get a message from HostConfig checkResources
> >> that says its undeploying the context, and then it redeploys.
> >
> > This is sometimes caused by incorrect timestamps on the bits of the
> > webapp that Tomcat monitors, or an incorrect clock setting on the
> > system Tomcat is running on. The mismatch makes it appear that the
> > webapp is being updated continuously.
>
> I've found that in development, a single update can cause Tomcat to go
> into a loop of redeployments, re-deploying my web application every
> few seconds or so. If I let it go, it does finally stop reloading and
> settle down.
>
> I believe Tomcat will emit a log message informing the administrator
> which file has triggered the reload, but I think it's at the DEBUG log
> level.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJTPG2nAAoJEBzwKT+lPKRY9oIQAIFI0aUQ33he8KqNQgwBaigY
> otTiL3Cr6dvVfYi2pLP9YK/2BqOkjtmCXAZ99gOhV0h3QntMWrBbYmkcu6BuibPH
> gnm2vME/B8XdNhGatPQgs1dW3OWa/OTra6req0CcLX3IucGERJWrMotOSLrsylUH
> lVkVWoRXZ3m6NikExKCmB/DUzy9dsecHGZM3hETEloye6p7EVez+RHFg/7VWDU42
> GyPZqiGFVRDELobbc4FVAy2R0OLPVSXzbsnuM3ZFS/TRqX+PKItfsnYczzrck3rK
> ozeYjmgG5IKD4eL/Nl0iUcZ/CjmHxTJCSlVwRhnQ7InFoNjR0TYur2dyw/+mqdlj
> U0u61XYJiUqHUnk5LUh5pFvxfKOyRSy34YlHAHTOBg6Nl/TZoAMtGBXQ5gsLssHC
> uf1U/oTl1hfhgIynQ3ijn10TXXXskEux5GHegUGVv8HqZNEcPVtuPw+3fEhGsXEu
> vgHmXUJIbLBDwIVD9WGC7iYHR4zNwHetlFwUE+sxAxP/zLwPd+Ps4TcoKabUh0nG
> sE7oj/qZxBHRxGIXg6BnGuzerHJPJT8dcaAo+U1zSmQdlAy1mCdvUrFpA7i63RdT
> XU8mwimmr93WBBOkhnWg4iJjTIl2LKEo1GmV0JiB/eqZaWIFNFy8oBnW56KnEhLV
> coHAGbSi2o7XZUdVpn0v
> =kMSx
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: tomcat randomly undeploys and redeploys the applications
Posted by Elias Kopsiaftis <ye...@gmail.com>.
Thanks Chris! I will look into this and let you guys know what I find
On Wed, Apr 2, 2014 at 4:19 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Elias,
>
> On 4/2/14, 4:05 PM, Christopher Schultz wrote:
> > Chuck,
> >
> > On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
> >>> From: Elias Kopsiaftis [mailto:yemista@gmail.com] Subject:
> >>> tomcat randomly undeploys and redeploys the applications
> >
> >>> I deploy the application, then in the log file catalina.out i
> >>> get many messages from WebappClassLoader
> >>> clearReferencesThreads saying threads appear to have started
> >>> but have failed to stop
> >
> >> This is an indication that your webapp is not managing its
> >> threads properly.
> >
> >>> then finally, Ill get a message from HostConfig checkResources
> >>> that says its undeploying the context, and then it redeploys.
> >
> >> This is sometimes caused by incorrect timestamps on the bits of
> >> the webapp that Tomcat monitors, or an incorrect clock setting on
> >> the system Tomcat is running on. The mismatch makes it appear
> >> that the webapp is being updated continuously.
> >
> > I've found that in development, a single update can cause Tomcat to
> > go into a loop of redeployments, re-deploying my web application
> > every few seconds or so. If I let it go, it does finally stop
> > reloading and settle down.
> >
> > I believe Tomcat will emit a log message informing the
> > administrator which file has triggered the reload, but I think it's
> > at the DEBUG log level.
>
> It's WebappClassLoader. You'll want to add this to your
> logging.properties in order to emit those messages:
>
> org.apache.catalina.loader.WebappClassLoader.level=DEBUG
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJTPHDjAAoJEBzwKT+lPKRYr4wP/3AzjqhtU8MKyuEoHSwyrwTt
> 3z0GaYZEolH+kW46+QOOnFUOXvWaK8MHjy5VgRU1rZP9DK/dwGYGbKrA7sMd9iuG
> nvjo7v/hLrPxtvvt8LKiH+vQslhyMU86uNReXu90QUWD0shj3twfC78IEWJKp0rl
> NR2/+mIardJv4KeZ06HQVgD/+xWpWVSxeDOeJJ3i1/alxhh7P7ybs8M3v9QTmaHC
> XFQEKSdb4MEn1tgEl88ucbhaibKLJKizOYRBfG2k0J3CcVsRtfsefmh0KDXSSBT+
> C6cOKULYCf4JimHdLAo8Ap0C4d6dBIu2MwIXRR+voV0rEtsn2uf43VaJmp9XsQie
> 9Gb6PQz7LUHUBAMb4FNrTRAWqfAb0LSq9Q1wXVOa9Ai6HMkYKnIZaO9pHfnAaQKK
> 34Zc5JMahtsbPtRj2jKSWiss92N5EDcKZHPLnhz1Z0q+zKRfCiBxDC3aS/bb4vIY
> HuYLc1MZq9+2PXjj6Mp9TP/eYOLkGgmT9OLsN+sdNSEmwpFQd/p5zQ9He7OW05O9
> q0wMjNmx0SRfjV5BrtG2k4eSvVaJpUP0FNKWWhaBAkreIyLs3P5C2QMjZ36NnJkA
> zO6jDsCJjwc7QOroSFyB++qm3FISDxGTuestkv0UAi92I+ePgjxooOPqSLmaVv5i
> 7UiJvtiB6XzAMxpn8wny
> =cKnh
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: tomcat randomly undeploys and redeploys the applications
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Elias,
On 4/2/14, 4:05 PM, Christopher Schultz wrote:
> Chuck,
>
> On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
>>> From: Elias Kopsiaftis [mailto:yemista@gmail.com] Subject:
>>> tomcat randomly undeploys and redeploys the applications
>
>>> I deploy the application, then in the log file catalina.out i
>>> get many messages from WebappClassLoader
>>> clearReferencesThreads saying threads appear to have started
>>> but have failed to stop
>
>> This is an indication that your webapp is not managing its
>> threads properly.
>
>>> then finally, Ill get a message from HostConfig checkResources
>>> that says its undeploying the context, and then it redeploys.
>
>> This is sometimes caused by incorrect timestamps on the bits of
>> the webapp that Tomcat monitors, or an incorrect clock setting on
>> the system Tomcat is running on. The mismatch makes it appear
>> that the webapp is being updated continuously.
>
> I've found that in development, a single update can cause Tomcat to
> go into a loop of redeployments, re-deploying my web application
> every few seconds or so. If I let it go, it does finally stop
> reloading and settle down.
>
> I believe Tomcat will emit a log message informing the
> administrator which file has triggered the reload, but I think it's
> at the DEBUG log level.
It's WebappClassLoader. You'll want to add this to your
logging.properties in order to emit those messages:
org.apache.catalina.loader.WebappClassLoader.level=DEBUG
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTPHDjAAoJEBzwKT+lPKRYr4wP/3AzjqhtU8MKyuEoHSwyrwTt
3z0GaYZEolH+kW46+QOOnFUOXvWaK8MHjy5VgRU1rZP9DK/dwGYGbKrA7sMd9iuG
nvjo7v/hLrPxtvvt8LKiH+vQslhyMU86uNReXu90QUWD0shj3twfC78IEWJKp0rl
NR2/+mIardJv4KeZ06HQVgD/+xWpWVSxeDOeJJ3i1/alxhh7P7ybs8M3v9QTmaHC
XFQEKSdb4MEn1tgEl88ucbhaibKLJKizOYRBfG2k0J3CcVsRtfsefmh0KDXSSBT+
C6cOKULYCf4JimHdLAo8Ap0C4d6dBIu2MwIXRR+voV0rEtsn2uf43VaJmp9XsQie
9Gb6PQz7LUHUBAMb4FNrTRAWqfAb0LSq9Q1wXVOa9Ai6HMkYKnIZaO9pHfnAaQKK
34Zc5JMahtsbPtRj2jKSWiss92N5EDcKZHPLnhz1Z0q+zKRfCiBxDC3aS/bb4vIY
HuYLc1MZq9+2PXjj6Mp9TP/eYOLkGgmT9OLsN+sdNSEmwpFQd/p5zQ9He7OW05O9
q0wMjNmx0SRfjV5BrtG2k4eSvVaJpUP0FNKWWhaBAkreIyLs3P5C2QMjZ36NnJkA
zO6jDsCJjwc7QOroSFyB++qm3FISDxGTuestkv0UAi92I+ePgjxooOPqSLmaVv5i
7UiJvtiB6XzAMxpn8wny
=cKnh
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Posted by Elias Kopsiaftis <ye...@gmail.com>.
Im sorry about that. The reason I did not post same day was that through
the messages in this thread someone suggested adding a debug statement to
debug WebappClassLoader, and I wanted to wait to see if it happened again
so I could see the debug output, then I came up with the idea in my last
message. yes we are on Linux and we are using java 7 and tomcat 7 and we
are on linux 3.11.6 on a 64 bit build for x86. When I say crash, I mean
that when we use the client application to connect to the web app it wont
connect and we have to go into tomcat manager and start it back up. We use
an executor with 20 threads in it to process things on the server, and that
was never shut down, so that would explain the memory leak and the extra
threads. We need them to run for the duration of the application life
cycle, but they should be cleaned up on undeployment. Thanks for the
reference to the presentation. I will go through it now. I think I may have
my answer and hopefully this wont happen again after we fix our web app to
shut down those threads.
On Fri, Apr 4, 2014 at 11:17 AM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Elias,
>
> On 4/4/14, 11:03 AM, Elias Kopsiaftis wrote:
> > Ive done some more research into this problem with my co developer,
> > and we have a question. Is it possible that memory leaks would
> > cause tomcat to crash?
>
> Yes, but you'll likely get an OutOfMemoryError in your log, and then
> everything will go crazy. Some requests will work, others will not.
> Things will get slow. For us, the background thread often quits and
> sessions never expire, exacerbating the problem.
>
> But it won't happen silently (you may have to look for it) and it
> won't spontaneously undeploy and redeploy your web application.
>
> > Silly question I know, but heres technically what the situation
> > is.
> >
> > We did not add code to shut down all threads when application is
> > undeployed through tomcat manager.
>
> Tomcat will do this itself: you don't need to add any code. Or did you
> mean that you have Threads created by your application that continue
> to tun after your web app has been shut down?
>
> > When undeploying and redeploying we cause memory leaks due to our
> > application.
>
> Yes, this can happen. The Threads cause the WebappClassLoader to stay
> in memory, which means that all Class objects for the previous
> deployment are still hanging around. It can be several tens of MiB per
> reload.
>
> You should read this excellent presentation which explains the
> problem(s) and the solution(s):
>
> http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
>
> > Now, I guess the real question is, we seem to always deploy it
> > fine, but randomly at some point during the night it will crash,
> > and we only know that from checking next morning.
>
> What do you mean when you say "crash"? Exception? Spontaneous
> re-deployment? OutOfMemory? JVM crash (process exits)?
>
> > We have nothing logged that indicates why.
>
> If the process is gone without any indication of why (no shutdown
> message, no nothing), my only guess is Linux OOM-killer. You never
> actually came back to describe the environment you are in after saying
> you'd post that information "later today." If you're not on Linux....
> I'm not sure.
>
> > Would memory leaks cause it to crash randomly even though nothing
> > is trying to connect to it, yet it would seem to deploy fine?
>
> I've never seen an OOME actually cause a JVM to quit. It just slogs
> onward, often not able to accomplish any useful work, but the process
> stays running.
>
> > Could that be the issue we are seeing? Is there any specific class
> > to debug in logging.properties that might indicate whats going on?
> > I added that line you guys mentioned about setting log level to
> > DEBUG from that class loader class, but it hasnt crashed since
> > then....
>
> Describe what you mean by "crash", including as much details about it,
> and definitely read the presentation referenced above. Also definitely
> fix your web application to shut down its Threads. Whatever component
> launches them (e.g. ServletContextListener, Servlet.init, etc.) should
> stop them in the appropriate callback method
> (ServletContextListener.contextDestroyed, Servlet.destroy, etc.).
>
> - -chris
>
> > On Thu, Apr 3, 2014 at 12:06 PM, Christopher Schultz <
> > chris@christopherschultz.net> wrote:
> >
> > Mark,
> >
> > On 4/2/14, 5:20 PM, Mark Eggers wrote:
> >>>> Chris,
> >>>>
> >>>> On 4/2/2014 1:54 PM, Christopher Schultz wrote:
> >>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
> >>>>>
> >>>>> Mark,
> >>>>>
> >>>>> On 4/2/14, 4:30 PM, Mark Eggers wrote:
> >>>>>> Chris,
> >>>>>>
> >>>>>> On 4/2/2014 1:05 PM, Christopher Schultz wrote:
> >>>>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
> >>>>>>>
> >>>>>>> Chuck,
> >>>>>>>
> >>>>>>> On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
> >>>>>>>>> From: Elias Kopsiaftis [mailto:yemista@gmail.com]
> >>>>>>>>> Subject: tomcat randomly undeploys and redeploys
> >>>>>>>>> the applications
> >>>>>>>>
> >>>>>>>>> I deploy the application, then in the log file
> >>>>>>>>> catalina.out i get many messages from
> >>>>>>>>> WebappClassLoader clearReferencesThreads saying
> >>>>>>>>> threads appear to have started but have failed to
> >>>>>>>>> stop
> >>>>>>>>
> >>>>>>>> This is an indication that your webapp is not
> >>>>>>>> managing its threads properly.
> >>>>>>>>
> >>>>>>>>> then finally, Ill get a message from HostConfig
> >>>>>>>>> checkResources that says its undeploying the
> >>>>>>>>> context, and then it redeploys.
> >>>>>>>>
> >>>>>>>> This is sometimes caused by incorrect timestamps on
> >>>>>>>> the bits of the webapp that Tomcat monitors, or an
> >>>>>>>> incorrect clock setting on the system Tomcat is
> >>>>>>>> running on. The mismatch makes it appear that the
> >>>>>>>> webapp is being updated continuously.
> >>>>>>>
> >>>>>>> I've found that in development, a single update can
> >>>>>>> cause Tomcat to go into a loop of redeployments,
> >>>>>>> re-deploying my web application every few seconds or
> >>>>>>> so. If I let it go, it does finally stop reloading and
> >>>>>>> settle down.
> >>>>>>>
> >>>>>>
> >>>>>> Can you describe your development environment a little
> >>>>>> bit, and any thoughts as to what might trigger this loop
> >>>>>> of redeployments?
> >>>>>
> >>>>> I use Eclipse for development, but our "real" build process
> >>>>> is ant-based. We have some watched resources configured
> >>>>> outside the default (stuff like Struts config files,
> >>>>> etc.).
> >>>>>
> >>>>
> >>>> Ah, makes sense.
> >>>>
> >>>>> When I do a build while Tomcat is running, usually I get
> >>>>> one webapp reload, but sometimes I get a series of reloads.
> >>>>> It usually gets so irritating (our webapp takes about 10
> >>>>> seconds to reload) that I just kill Tomcat and immediately
> >>>>> restart it. It starts up once and all is well after that.
> >>>>>
> >>>>
> >>>> Yep, and in the process more files are copied about, and
> >>>> that triggers another reload.
> >>>>
> >>>> Fun, fun.
> >
> > No, the deployment update takes like one or two seconds. It's
> > usually something like copying less than 10 class files or
> > whatever. It's nearly instantaneous. Whatever happens, it's not
> > because I'm updating files during the reload. I could understand
> > that situation.
> >
> > What I observe is that I update my application, I wait maybe 10
> > seconds, and then Tomcat reloads my application multiple times
> > before I just kill it.
> >
> >>>>>> I've not seen this, but it could explain some issues some
> >>>>>> the developers I support are seeing.
> >>>>>
> >>>>> It definitely happens, and I never remember to enable the
> >>>>> DEBUG logging to find out what resource it thinks has been
> >>>>> updated until after it happens, at which point I just don't
> >>>>> care. Perhaps I should enable it right now :)
> >>>>>
> >>>>> - -chris
> >>>>
> >>>> I've managed to make this happen in my environment now
> >>>> (NetBeans 7.4, Maven 3.2.1, Tomcat 7.0.42 - all will be
> >>>> upgraded soon). I just needed an application that takes a bit
> >>>> longer to load. I only managed to trigger two reloads, so
> >>>> it's not much of an issue.
> >>>>
> >>>> Maybe look at adding the backgroundProcessorDelay attribute
> >>>> to the context? I don't know what would happen if the context
> >>>> got a string of reload requests within the delay interval.
> >>>> Would it queue them up one after the other, or would it just
> >>>> execute one?
> >
> > I think it's more important to see what file(s) Tomcat thinks
> > is(are) being updated. I wonder if it's the same file, or if
> > there's some weird timestamp issue happening. Perhaps there is even
> > some kind of edge case where a resource's last-modified date isn't
> > being updated properly.
> >
> > In most cases, Tomcat reloads my application a single time, as
> > expected. These reload-storms are fairly rare.
> >
> > -chris
> >>
> >> ---------------------------------------------------------------------
> >>
> >>
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> >>
> >>
> >
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJTPs0fAAoJEBzwKT+lPKRYYBcQAI+xdEUBXVdylZIWxcJoY097
> v9xgkd6Zud4QoupVhUVrdP7YCjjhFTtS0y9MCOMxLdSyQQFjKS+yFfjE8Xtyah7x
> tnUwf200qnUBVCaBgDY+lfmrx+AuSNogce81qqpQkYUGiVJT+rW0MYnMQfgeCPmg
> RchOe6zo5664bvU3EBRnoPe3UYZcIwVjK9vXS8dy9vMygZJ6zGN+XIs0PTlFE5h/
> HA8Ik6UJoK6yvyHqOfY2wC0xXeKMUsbNAcsIBcuC4H7h/NbDQCoH0YV5vvkpOBhF
> vhMLvCtzOgTEK0oLDBsbuiAXF6g1XzQk+Uxoy0a1Kx5U4TbswFIK2hHtAtmQWC1y
> v8RvvvNYNRO3tzI3Zo/CFO3+4Ha5EHmj96HLJI2itIOefD+By+y+gqqUR35lGdNT
> 7NY4zd0fIYQqkqkBajpDPJX6eGShEHX0obF5dR/46uaehl+Vy9tfUq+fZEILP2Cm
> XZdi6SD0epzpX+J6/7usQErU4QIxAnmmxwHu+ShhvRMF0+5ZsWCkt5RvZxMNuiMQ
> g9aDC1T+RACVOTCe80WDKgOnWmeGhbCycClb2ITSt3ceh/tb+JOShJoIgNl8r1MN
> Hq1h/aNxYLJRsKt3c4P+PXui4bgh8JVe3gtAuilo1s+0Z7gBgaAmwIVtG3gDA/hH
> ERlGxSO1fiY+0/p3Lf+f
> =3VV0
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: tomcat randomly undeploys and redeploys the applications
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Elias,
On 4/4/14, 11:03 AM, Elias Kopsiaftis wrote:
> Ive done some more research into this problem with my co developer,
> and we have a question. Is it possible that memory leaks would
> cause tomcat to crash?
Yes, but you'll likely get an OutOfMemoryError in your log, and then
everything will go crazy. Some requests will work, others will not.
Things will get slow. For us, the background thread often quits and
sessions never expire, exacerbating the problem.
But it won't happen silently (you may have to look for it) and it
won't spontaneously undeploy and redeploy your web application.
> Silly question I know, but heres technically what the situation
> is.
>
> We did not add code to shut down all threads when application is
> undeployed through tomcat manager.
Tomcat will do this itself: you don't need to add any code. Or did you
mean that you have Threads created by your application that continue
to tun after your web app has been shut down?
> When undeploying and redeploying we cause memory leaks due to our
> application.
Yes, this can happen. The Threads cause the WebappClassLoader to stay
in memory, which means that all Class objects for the previous
deployment are still hanging around. It can be several tens of MiB per
reload.
You should read this excellent presentation which explains the
problem(s) and the solution(s):
http://people.apache.org/~markt/presentations/2010-11-04-Memory-Leaks-60mins.pdf
> Now, I guess the real question is, we seem to always deploy it
> fine, but randomly at some point during the night it will crash,
> and we only know that from checking next morning.
What do you mean when you say "crash"? Exception? Spontaneous
re-deployment? OutOfMemory? JVM crash (process exits)?
> We have nothing logged that indicates why.
If the process is gone without any indication of why (no shutdown
message, no nothing), my only guess is Linux OOM-killer. You never
actually came back to describe the environment you are in after saying
you'd post that information "later today." If you're not on Linux....
I'm not sure.
> Would memory leaks cause it to crash randomly even though nothing
> is trying to connect to it, yet it would seem to deploy fine?
I've never seen an OOME actually cause a JVM to quit. It just slogs
onward, often not able to accomplish any useful work, but the process
stays running.
> Could that be the issue we are seeing? Is there any specific class
> to debug in logging.properties that might indicate whats going on?
> I added that line you guys mentioned about setting log level to
> DEBUG from that class loader class, but it hasnt crashed since
> then....
Describe what you mean by "crash", including as much details about it,
and definitely read the presentation referenced above. Also definitely
fix your web application to shut down its Threads. Whatever component
launches them (e.g. ServletContextListener, Servlet.init, etc.) should
stop them in the appropriate callback method
(ServletContextListener.contextDestroyed, Servlet.destroy, etc.).
- -chris
> On Thu, Apr 3, 2014 at 12:06 PM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
> Mark,
>
> On 4/2/14, 5:20 PM, Mark Eggers wrote:
>>>> Chris,
>>>>
>>>> On 4/2/2014 1:54 PM, Christopher Schultz wrote:
>>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>>>>>
>>>>> Mark,
>>>>>
>>>>> On 4/2/14, 4:30 PM, Mark Eggers wrote:
>>>>>> Chris,
>>>>>>
>>>>>> On 4/2/2014 1:05 PM, Christopher Schultz wrote:
>>>>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>>>>>>>
>>>>>>> Chuck,
>>>>>>>
>>>>>>> On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
>>>>>>>>> From: Elias Kopsiaftis [mailto:yemista@gmail.com]
>>>>>>>>> Subject: tomcat randomly undeploys and redeploys
>>>>>>>>> the applications
>>>>>>>>
>>>>>>>>> I deploy the application, then in the log file
>>>>>>>>> catalina.out i get many messages from
>>>>>>>>> WebappClassLoader clearReferencesThreads saying
>>>>>>>>> threads appear to have started but have failed to
>>>>>>>>> stop
>>>>>>>>
>>>>>>>> This is an indication that your webapp is not
>>>>>>>> managing its threads properly.
>>>>>>>>
>>>>>>>>> then finally, Ill get a message from HostConfig
>>>>>>>>> checkResources that says its undeploying the
>>>>>>>>> context, and then it redeploys.
>>>>>>>>
>>>>>>>> This is sometimes caused by incorrect timestamps on
>>>>>>>> the bits of the webapp that Tomcat monitors, or an
>>>>>>>> incorrect clock setting on the system Tomcat is
>>>>>>>> running on. The mismatch makes it appear that the
>>>>>>>> webapp is being updated continuously.
>>>>>>>
>>>>>>> I've found that in development, a single update can
>>>>>>> cause Tomcat to go into a loop of redeployments,
>>>>>>> re-deploying my web application every few seconds or
>>>>>>> so. If I let it go, it does finally stop reloading and
>>>>>>> settle down.
>>>>>>>
>>>>>>
>>>>>> Can you describe your development environment a little
>>>>>> bit, and any thoughts as to what might trigger this loop
>>>>>> of redeployments?
>>>>>
>>>>> I use Eclipse for development, but our "real" build process
>>>>> is ant-based. We have some watched resources configured
>>>>> outside the default (stuff like Struts config files,
>>>>> etc.).
>>>>>
>>>>
>>>> Ah, makes sense.
>>>>
>>>>> When I do a build while Tomcat is running, usually I get
>>>>> one webapp reload, but sometimes I get a series of reloads.
>>>>> It usually gets so irritating (our webapp takes about 10
>>>>> seconds to reload) that I just kill Tomcat and immediately
>>>>> restart it. It starts up once and all is well after that.
>>>>>
>>>>
>>>> Yep, and in the process more files are copied about, and
>>>> that triggers another reload.
>>>>
>>>> Fun, fun.
>
> No, the deployment update takes like one or two seconds. It's
> usually something like copying less than 10 class files or
> whatever. It's nearly instantaneous. Whatever happens, it's not
> because I'm updating files during the reload. I could understand
> that situation.
>
> What I observe is that I update my application, I wait maybe 10
> seconds, and then Tomcat reloads my application multiple times
> before I just kill it.
>
>>>>>> I've not seen this, but it could explain some issues some
>>>>>> the developers I support are seeing.
>>>>>
>>>>> It definitely happens, and I never remember to enable the
>>>>> DEBUG logging to find out what resource it thinks has been
>>>>> updated until after it happens, at which point I just don't
>>>>> care. Perhaps I should enable it right now :)
>>>>>
>>>>> - -chris
>>>>
>>>> I've managed to make this happen in my environment now
>>>> (NetBeans 7.4, Maven 3.2.1, Tomcat 7.0.42 - all will be
>>>> upgraded soon). I just needed an application that takes a bit
>>>> longer to load. I only managed to trigger two reloads, so
>>>> it's not much of an issue.
>>>>
>>>> Maybe look at adding the backgroundProcessorDelay attribute
>>>> to the context? I don't know what would happen if the context
>>>> got a string of reload requests within the delay interval.
>>>> Would it queue them up one after the other, or would it just
>>>> execute one?
>
> I think it's more important to see what file(s) Tomcat thinks
> is(are) being updated. I wonder if it's the same file, or if
> there's some weird timestamp issue happening. Perhaps there is even
> some kind of edge case where a resource's last-modified date isn't
> being updated properly.
>
> In most cases, Tomcat reloads my application a single time, as
> expected. These reload-storms are fairly rare.
>
> -chris
>>
>> ---------------------------------------------------------------------
>>
>>
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTPs0fAAoJEBzwKT+lPKRYYBcQAI+xdEUBXVdylZIWxcJoY097
v9xgkd6Zud4QoupVhUVrdP7YCjjhFTtS0y9MCOMxLdSyQQFjKS+yFfjE8Xtyah7x
tnUwf200qnUBVCaBgDY+lfmrx+AuSNogce81qqpQkYUGiVJT+rW0MYnMQfgeCPmg
RchOe6zo5664bvU3EBRnoPe3UYZcIwVjK9vXS8dy9vMygZJ6zGN+XIs0PTlFE5h/
HA8Ik6UJoK6yvyHqOfY2wC0xXeKMUsbNAcsIBcuC4H7h/NbDQCoH0YV5vvkpOBhF
vhMLvCtzOgTEK0oLDBsbuiAXF6g1XzQk+Uxoy0a1Kx5U4TbswFIK2hHtAtmQWC1y
v8RvvvNYNRO3tzI3Zo/CFO3+4Ha5EHmj96HLJI2itIOefD+By+y+gqqUR35lGdNT
7NY4zd0fIYQqkqkBajpDPJX6eGShEHX0obF5dR/46uaehl+Vy9tfUq+fZEILP2Cm
XZdi6SD0epzpX+J6/7usQErU4QIxAnmmxwHu+ShhvRMF0+5ZsWCkt5RvZxMNuiMQ
g9aDC1T+RACVOTCe80WDKgOnWmeGhbCycClb2ITSt3ceh/tb+JOShJoIgNl8r1MN
Hq1h/aNxYLJRsKt3c4P+PXui4bgh8JVe3gtAuilo1s+0Z7gBgaAmwIVtG3gDA/hH
ERlGxSO1fiY+0/p3Lf+f
=3VV0
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Posted by Elias Kopsiaftis <ye...@gmail.com>.
Ive done some more research into this problem with my co developer, and we
have a question. Is it possible that memory leaks would cause tomcat to
crash? Silly question I know, but heres technically what the situation is.
We did not add code to shut down all threads when application is undeployed
through tomcat manager. When undeploying and redeploying we cause memory
leaks due to our application. Now, I guess the real question is, we seem to
always deploy it fine, but randomly at some point during the night it will
crash, and we only know that from checking next morning. We have nothing
logged that indicates why. Would memory leaks cause it to crash randomly
even though nothing is trying to connect to it, yet it would seem to deploy
fine? Could that be the issue we are seeing? Is there any specific class to
debug in logging.properties that might indicate whats going on? I added
that line you guys mentioned about setting log level to DEBUG from that
class loader class, but it hasnt crashed since then....
On Thu, Apr 3, 2014 at 12:06 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Mark,
>
> On 4/2/14, 5:20 PM, Mark Eggers wrote:
> > Chris,
> >
> > On 4/2/2014 1:54 PM, Christopher Schultz wrote:
> >> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
> >>
> >> Mark,
> >>
> >> On 4/2/14, 4:30 PM, Mark Eggers wrote:
> >>> Chris,
> >>>
> >>> On 4/2/2014 1:05 PM, Christopher Schultz wrote:
> >>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
> >>>>
> >>>> Chuck,
> >>>>
> >>>> On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
> >>>>>> From: Elias Kopsiaftis [mailto:yemista@gmail.com]
> >>>>>> Subject: tomcat randomly undeploys and redeploys the
> >>>>>> applications
> >>>>>
> >>>>>> I deploy the application, then in the log file
> >>>>>> catalina.out i get many messages from WebappClassLoader
> >>>>>> clearReferencesThreads saying threads appear to have
> >>>>>> started but have failed to stop
> >>>>>
> >>>>> This is an indication that your webapp is not managing its
> >>>>> threads properly.
> >>>>>
> >>>>>> then finally, Ill get a message from HostConfig
> >>>>>> checkResources that says its undeploying the context,
> >>>>>> and then it redeploys.
> >>>>>
> >>>>> This is sometimes caused by incorrect timestamps on the
> >>>>> bits of the webapp that Tomcat monitors, or an incorrect
> >>>>> clock setting on the system Tomcat is running on. The
> >>>>> mismatch makes it appear that the webapp is being updated
> >>>>> continuously.
> >>>>
> >>>> I've found that in development, a single update can cause
> >>>> Tomcat to go into a loop of redeployments, re-deploying my
> >>>> web application every few seconds or so. If I let it go, it
> >>>> does finally stop reloading and settle down.
> >>>>
> >>>
> >>> Can you describe your development environment a little bit,
> >>> and any thoughts as to what might trigger this loop of
> >>> redeployments?
> >>
> >> I use Eclipse for development, but our "real" build process is
> >> ant-based. We have some watched resources configured outside the
> >> default (stuff like Struts config files, etc.).
> >>
> >
> > Ah, makes sense.
> >
> >> When I do a build while Tomcat is running, usually I get one
> >> webapp reload, but sometimes I get a series of reloads. It
> >> usually gets so irritating (our webapp takes about 10 seconds to
> >> reload) that I just kill Tomcat and immediately restart it. It
> >> starts up once and all is well after that.
> >>
> >
> > Yep, and in the process more files are copied about, and that
> > triggers another reload.
> >
> > Fun, fun.
>
> No, the deployment update takes like one or two seconds. It's usually
> something like copying less than 10 class files or whatever. It's
> nearly instantaneous. Whatever happens, it's not because I'm updating
> files during the reload. I could understand that situation.
>
> What I observe is that I update my application, I wait maybe 10
> seconds, and then Tomcat reloads my application multiple times before
> I just kill it.
>
> >>> I've not seen this, but it could explain some issues some the
> >>> developers I support are seeing.
> >>
> >> It definitely happens, and I never remember to enable the DEBUG
> >> logging to find out what resource it thinks has been updated
> >> until after it happens, at which point I just don't care. Perhaps
> >> I should enable it right now :)
> >>
> >> - -chris
> >
> > I've managed to make this happen in my environment now (NetBeans
> > 7.4, Maven 3.2.1, Tomcat 7.0.42 - all will be upgraded soon). I
> > just needed an application that takes a bit longer to load. I only
> > managed to trigger two reloads, so it's not much of an issue.
> >
> > Maybe look at adding the backgroundProcessorDelay attribute to the
> > context? I don't know what would happen if the context got a string
> > of reload requests within the delay interval. Would it queue them
> > up one after the other, or would it just execute one?
>
> I think it's more important to see what file(s) Tomcat thinks is(are)
> being updated. I wonder if it's the same file, or if there's some
> weird timestamp issue happening. Perhaps there is even some kind of
> edge case where a resource's last-modified date isn't being updated
> properly.
>
> In most cases, Tomcat reloads my application a single time, as
> expected. These reload-storms are fairly rare.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJTPYcLAAoJEBzwKT+lPKRYf/gQAKhUNIp0yIP24g6fctOppPGC
> 0WbacCWGaJwC9Xbuh/lNqtYRZck2943+rOZwOw4sBfwxox3dlVYGl+NHj0C4NRyW
> KJmH11v5gO59Di7S4NZeYGlHpbYWSZt/2HiMZAVQ6zElXN5qkSEa5WbjXJJsduSe
> FHD6RFLST7pvlbOuEj8L/+MldsflYTe7Mu5CykBK52GLZGMAYTFYWqcs6nrsscxc
> ZWEKAt1QU1barvojnoZjk4pZksihi/QwOmCJ1a+rHWUZPmtzp/9gdTon47WHDcXU
> NVEQLOHJgtolJI2XYMVXFZPVOEeD80PWdQ+aIUAozOR954odw9RcZRz71EiNAagB
> YBImgFi0zFNwVKivi4yqKHGex8LPj6qGuDI2Nd48Za4s6gN90fpHwpYFzrq3dnnv
> ep7Jf1qHNiGc70s//TH8iKZToCkN/N2Sythlv729NEFGdXBkn0Ph8RVhPmxiYiwu
> WrIXrk875m2241QBubcuaNFnPPmNA2cj0IkHB8QDM+35wUI40sx+vtMvLV3U674O
> JBATY5Oiq26jj585OAsNJJsNY1y33rkH0tqZNEPTYUfqSW4FUxHe0J6TWf7CMujF
> sPjQqk1kXawsyNJNeGqKxUqW6LHvo+dX40/FmgAl4llkOCU9DFXxdhZVBg14UQdC
> OeWlyGTxwovZZEdJJqtO
> =r81a
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: tomcat randomly undeploys and redeploys the applications
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 4/2/14, 5:20 PM, Mark Eggers wrote:
> Chris,
>
> On 4/2/2014 1:54 PM, Christopher Schultz wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>>
>> Mark,
>>
>> On 4/2/14, 4:30 PM, Mark Eggers wrote:
>>> Chris,
>>>
>>> On 4/2/2014 1:05 PM, Christopher Schultz wrote:
>>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>>>>
>>>> Chuck,
>>>>
>>>> On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
>>>>>> From: Elias Kopsiaftis [mailto:yemista@gmail.com]
>>>>>> Subject: tomcat randomly undeploys and redeploys the
>>>>>> applications
>>>>>
>>>>>> I deploy the application, then in the log file
>>>>>> catalina.out i get many messages from WebappClassLoader
>>>>>> clearReferencesThreads saying threads appear to have
>>>>>> started but have failed to stop
>>>>>
>>>>> This is an indication that your webapp is not managing its
>>>>> threads properly.
>>>>>
>>>>>> then finally, Ill get a message from HostConfig
>>>>>> checkResources that says its undeploying the context,
>>>>>> and then it redeploys.
>>>>>
>>>>> This is sometimes caused by incorrect timestamps on the
>>>>> bits of the webapp that Tomcat monitors, or an incorrect
>>>>> clock setting on the system Tomcat is running on. The
>>>>> mismatch makes it appear that the webapp is being updated
>>>>> continuously.
>>>>
>>>> I've found that in development, a single update can cause
>>>> Tomcat to go into a loop of redeployments, re-deploying my
>>>> web application every few seconds or so. If I let it go, it
>>>> does finally stop reloading and settle down.
>>>>
>>>
>>> Can you describe your development environment a little bit,
>>> and any thoughts as to what might trigger this loop of
>>> redeployments?
>>
>> I use Eclipse for development, but our "real" build process is
>> ant-based. We have some watched resources configured outside the
>> default (stuff like Struts config files, etc.).
>>
>
> Ah, makes sense.
>
>> When I do a build while Tomcat is running, usually I get one
>> webapp reload, but sometimes I get a series of reloads. It
>> usually gets so irritating (our webapp takes about 10 seconds to
>> reload) that I just kill Tomcat and immediately restart it. It
>> starts up once and all is well after that.
>>
>
> Yep, and in the process more files are copied about, and that
> triggers another reload.
>
> Fun, fun.
No, the deployment update takes like one or two seconds. It's usually
something like copying less than 10 class files or whatever. It's
nearly instantaneous. Whatever happens, it's not because I'm updating
files during the reload. I could understand that situation.
What I observe is that I update my application, I wait maybe 10
seconds, and then Tomcat reloads my application multiple times before
I just kill it.
>>> I've not seen this, but it could explain some issues some the
>>> developers I support are seeing.
>>
>> It definitely happens, and I never remember to enable the DEBUG
>> logging to find out what resource it thinks has been updated
>> until after it happens, at which point I just don't care. Perhaps
>> I should enable it right now :)
>>
>> - -chris
>
> I've managed to make this happen in my environment now (NetBeans
> 7.4, Maven 3.2.1, Tomcat 7.0.42 - all will be upgraded soon). I
> just needed an application that takes a bit longer to load. I only
> managed to trigger two reloads, so it's not much of an issue.
>
> Maybe look at adding the backgroundProcessorDelay attribute to the
> context? I don't know what would happen if the context got a string
> of reload requests within the delay interval. Would it queue them
> up one after the other, or would it just execute one?
I think it's more important to see what file(s) Tomcat thinks is(are)
being updated. I wonder if it's the same file, or if there's some
weird timestamp issue happening. Perhaps there is even some kind of
edge case where a resource's last-modified date isn't being updated
properly.
In most cases, Tomcat reloads my application a single time, as
expected. These reload-storms are fairly rare.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTPYcLAAoJEBzwKT+lPKRYf/gQAKhUNIp0yIP24g6fctOppPGC
0WbacCWGaJwC9Xbuh/lNqtYRZck2943+rOZwOw4sBfwxox3dlVYGl+NHj0C4NRyW
KJmH11v5gO59Di7S4NZeYGlHpbYWSZt/2HiMZAVQ6zElXN5qkSEa5WbjXJJsduSe
FHD6RFLST7pvlbOuEj8L/+MldsflYTe7Mu5CykBK52GLZGMAYTFYWqcs6nrsscxc
ZWEKAt1QU1barvojnoZjk4pZksihi/QwOmCJ1a+rHWUZPmtzp/9gdTon47WHDcXU
NVEQLOHJgtolJI2XYMVXFZPVOEeD80PWdQ+aIUAozOR954odw9RcZRz71EiNAagB
YBImgFi0zFNwVKivi4yqKHGex8LPj6qGuDI2Nd48Za4s6gN90fpHwpYFzrq3dnnv
ep7Jf1qHNiGc70s//TH8iKZToCkN/N2Sythlv729NEFGdXBkn0Ph8RVhPmxiYiwu
WrIXrk875m2241QBubcuaNFnPPmNA2cj0IkHB8QDM+35wUI40sx+vtMvLV3U674O
JBATY5Oiq26jj585OAsNJJsNY1y33rkH0tqZNEPTYUfqSW4FUxHe0J6TWf7CMujF
sPjQqk1kXawsyNJNeGqKxUqW6LHvo+dX40/FmgAl4llkOCU9DFXxdhZVBg14UQdC
OeWlyGTxwovZZEdJJqtO
=r81a
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Posted by Mark Eggers <it...@yahoo.com>.
Chris,
On 4/2/2014 1:54 PM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Mark,
>
> On 4/2/14, 4:30 PM, Mark Eggers wrote:
>> Chris,
>>
>> On 4/2/2014 1:05 PM, Christopher Schultz wrote:
>>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>>>
>>> Chuck,
>>>
>>> On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
>>>>> From: Elias Kopsiaftis [mailto:yemista@gmail.com] Subject:
>>>>> tomcat randomly undeploys and redeploys the applications
>>>>
>>>>> I deploy the application, then in the log file catalina.out i
>>>>> get many messages from WebappClassLoader
>>>>> clearReferencesThreads saying threads appear to have started
>>>>> but have failed to stop
>>>>
>>>> This is an indication that your webapp is not managing its
>>>> threads properly.
>>>>
>>>>> then finally, Ill get a message from HostConfig
>>>>> checkResources that says its undeploying the context, and
>>>>> then it redeploys.
>>>>
>>>> This is sometimes caused by incorrect timestamps on the bits of
>>>> the webapp that Tomcat monitors, or an incorrect clock setting
>>>> on the system Tomcat is running on. The mismatch makes it
>>>> appear that the webapp is being updated continuously.
>>>
>>> I've found that in development, a single update can cause Tomcat
>>> to go into a loop of redeployments, re-deploying my web
>>> application every few seconds or so. If I let it go, it does
>>> finally stop reloading and settle down.
>>>
>>
>> Can you describe your development environment a little bit, and
>> any thoughts as to what might trigger this loop of redeployments?
>
> I use Eclipse for development, but our "real" build process is
> ant-based. We have some watched resources configured outside the
> default (stuff like Struts config files, etc.).
>
Ah, makes sense.
> When I do a build while Tomcat is running, usually I get one webapp
> reload, but sometimes I get a series of reloads. It usually gets so
> irritating (our webapp takes about 10 seconds to reload) that I just
> kill Tomcat and immediately restart it. It starts up once and all is
> well after that.
>
Yep, and in the process more files are copied about, and that triggers
another reload.
Fun, fun.
>> I've not seen this, but it could explain some issues some the
>> developers I support are seeing.
>
> It definitely happens, and I never remember to enable the DEBUG
> logging to find out what resource it thinks has been updated until
> after it happens, at which point I just don't care. Perhaps I should
> enable it right now :)
>
> - -chris
I've managed to make this happen in my environment now (NetBeans 7.4,
Maven 3.2.1, Tomcat 7.0.42 - all will be upgraded soon). I just needed
an application that takes a bit longer to load. I only managed to
trigger two reloads, so it's not much of an issue.
Maybe look at adding the backgroundProcessorDelay attribute to the
context? I don't know what would happen if the context got a string of
reload requests within the delay interval. Would it queue them up one
after the other, or would it just execute one?
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 4/2/14, 4:30 PM, Mark Eggers wrote:
> Chris,
>
> On 4/2/2014 1:05 PM, Christopher Schultz wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
>>
>> Chuck,
>>
>> On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
>>>> From: Elias Kopsiaftis [mailto:yemista@gmail.com] Subject:
>>>> tomcat randomly undeploys and redeploys the applications
>>>
>>>> I deploy the application, then in the log file catalina.out i
>>>> get many messages from WebappClassLoader
>>>> clearReferencesThreads saying threads appear to have started
>>>> but have failed to stop
>>>
>>> This is an indication that your webapp is not managing its
>>> threads properly.
>>>
>>>> then finally, Ill get a message from HostConfig
>>>> checkResources that says its undeploying the context, and
>>>> then it redeploys.
>>>
>>> This is sometimes caused by incorrect timestamps on the bits of
>>> the webapp that Tomcat monitors, or an incorrect clock setting
>>> on the system Tomcat is running on. The mismatch makes it
>>> appear that the webapp is being updated continuously.
>>
>> I've found that in development, a single update can cause Tomcat
>> to go into a loop of redeployments, re-deploying my web
>> application every few seconds or so. If I let it go, it does
>> finally stop reloading and settle down.
>>
>
> Can you describe your development environment a little bit, and
> any thoughts as to what might trigger this loop of redeployments?
I use Eclipse for development, but our "real" build process is
ant-based. We have some watched resources configured outside the
default (stuff like Struts config files, etc.).
When I do a build while Tomcat is running, usually I get one webapp
reload, but sometimes I get a series of reloads. It usually gets so
irritating (our webapp takes about 10 seconds to reload) that I just
kill Tomcat and immediately restart it. It starts up once and all is
well after that.
> I've not seen this, but it could explain some issues some the
> developers I support are seeing.
It definitely happens, and I never remember to enable the DEBUG
logging to find out what resource it thinks has been updated until
after it happens, at which point I just don't care. Perhaps I should
enable it right now :)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTPHjmAAoJEBzwKT+lPKRYtzMP/2cBUk4nK2Ox5IL4BvxT0YF1
xb5WGrh+QvRBuOyPF1M2X2SVIijyh0k9geRD3zWaGl4YZ8dLVZY4mXEmh2lXt4g9
B1jhNrUpeNHUVtb5gVrzwj063H+duabm67OWzXXJMzTY2MlaroPhx9W80Fu7OpJY
Nsh7WNsFxBa5KPAV2H3XjbP6Oskbpv1F329AaBHM6HAH2J4eHahtpEujaIolSQbA
L8rOT1UYw3dblMQRuvryydB42btw1hTpgZX7UHGIsNxtELPy8LkwrmaCxAKaHBvR
zbw2vKj5VWjpaBmWzPT4qIrdIJg2wUAfXYjyAKuZ5JRxIxEPiTWB6SM5224Jzknv
DFwic0FxervIcLudfSb96Xcto0fqNzUJeHHPhRD8eGzSOIdkqiI8Faqml/f207ma
ibXQp6OIo1P8YLGkA21ZWBwSrp3hvKpV0+Nq+LarVTueODv4Mig1E8S3B5n9CzZQ
8N6vVZolnZlpkXC88VuQk6Dct0u8kmx5sqR/HUY7ACSVZHskRABVi5C+AxWg89TN
ElXmPydGyjgvOp1Ldf5S6jBSdTY9qI7LEl7kBrEUKMz4aBzwfmTw2/08yozAjaQW
TxJuOderFvslr+C2iQ6g8Qqsk+davD/bZ05REw9vz4/QgGzGCNhAW9pKoH5GoU0x
ZHwMuvTSzwM754whVzKf
=fzrA
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Posted by Mark Eggers <it...@yahoo.com>.
Chris,
On 4/2/2014 1:05 PM, Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Chuck,
>
> On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
>>> From: Elias Kopsiaftis [mailto:yemista@gmail.com] Subject: tomcat
>>> randomly undeploys and redeploys the applications
>>
>>> I deploy the application, then in the log file catalina.out i get
>>> many messages from WebappClassLoader clearReferencesThreads
>>> saying threads appear to have started but have failed to stop
>>
>> This is an indication that your webapp is not managing its threads
>> properly.
>>
>>> then finally, Ill get a message from HostConfig checkResources
>>> that says its undeploying the context, and then it redeploys.
>>
>> This is sometimes caused by incorrect timestamps on the bits of the
>> webapp that Tomcat monitors, or an incorrect clock setting on the
>> system Tomcat is running on. The mismatch makes it appear that the
>> webapp is being updated continuously.
>
> I've found that in development, a single update can cause Tomcat to go
> into a loop of redeployments, re-deploying my web application every
> few seconds or so. If I let it go, it does finally stop reloading and
> settle down.
>
Can you describe your development environment a little bit, and any
thoughts as to what might trigger this loop of redeployments?
I've not seen this, but it could explain some issues some the developers
I support are seeing.
I believe Tomcat will emit a log message informing the administrator
> which file has triggered the reload, but I think it's at the DEBUG log
> level.
>
> - -chris
Thanks,
/mde/
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: tomcat randomly undeploys and redeploys the applications
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Chuck,
On 4/2/14, 8:21 AM, Caldarale, Charles R wrote:
>> From: Elias Kopsiaftis [mailto:yemista@gmail.com] Subject: tomcat
>> randomly undeploys and redeploys the applications
>
>> I deploy the application, then in the log file catalina.out i get
>> many messages from WebappClassLoader clearReferencesThreads
>> saying threads appear to have started but have failed to stop
>
> This is an indication that your webapp is not managing its threads
> properly.
>
>> then finally, Ill get a message from HostConfig checkResources
>> that says its undeploying the context, and then it redeploys.
>
> This is sometimes caused by incorrect timestamps on the bits of the
> webapp that Tomcat monitors, or an incorrect clock setting on the
> system Tomcat is running on. The mismatch makes it appear that the
> webapp is being updated continuously.
I've found that in development, a single update can cause Tomcat to go
into a loop of redeployments, re-deploying my web application every
few seconds or so. If I let it go, it does finally stop reloading and
settle down.
I believe Tomcat will emit a log message informing the administrator
which file has triggered the reload, but I think it's at the DEBUG log
level.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJTPG2nAAoJEBzwKT+lPKRY9oIQAIFI0aUQ33he8KqNQgwBaigY
otTiL3Cr6dvVfYi2pLP9YK/2BqOkjtmCXAZ99gOhV0h3QntMWrBbYmkcu6BuibPH
gnm2vME/B8XdNhGatPQgs1dW3OWa/OTra6req0CcLX3IucGERJWrMotOSLrsylUH
lVkVWoRXZ3m6NikExKCmB/DUzy9dsecHGZM3hETEloye6p7EVez+RHFg/7VWDU42
GyPZqiGFVRDELobbc4FVAy2R0OLPVSXzbsnuM3ZFS/TRqX+PKItfsnYczzrck3rK
ozeYjmgG5IKD4eL/Nl0iUcZ/CjmHxTJCSlVwRhnQ7InFoNjR0TYur2dyw/+mqdlj
U0u61XYJiUqHUnk5LUh5pFvxfKOyRSy34YlHAHTOBg6Nl/TZoAMtGBXQ5gsLssHC
uf1U/oTl1hfhgIynQ3ijn10TXXXskEux5GHegUGVv8HqZNEcPVtuPw+3fEhGsXEu
vgHmXUJIbLBDwIVD9WGC7iYHR4zNwHetlFwUE+sxAxP/zLwPd+Ps4TcoKabUh0nG
sE7oj/qZxBHRxGIXg6BnGuzerHJPJT8dcaAo+U1zSmQdlAy1mCdvUrFpA7i63RdT
XU8mwimmr93WBBOkhnWg4iJjTIl2LKEo1GmV0JiB/eqZaWIFNFy8oBnW56KnEhLV
coHAGbSi2o7XZUdVpn0v
=kMSx
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: tomcat randomly undeploys and redeploys the applications
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Elias Kopsiaftis [mailto:yemista@gmail.com]
> Subject: tomcat randomly undeploys and redeploys the applications
> I deploy the application, then in the log file catalina.out i get many
> messages from WebappClassLoader clearReferencesThreads saying threads
> appear to have started but have failed to stop
This is an indication that your webapp is not managing its threads properly.
> then finally, Ill get a message from HostConfig checkResources that
> says its undeploying the context, and then it redeploys.
This is sometimes caused by incorrect timestamps on the bits of the webapp that Tomcat monitors, or an incorrect clock setting on the system Tomcat is running on. The mismatch makes it appear that the webapp is being updated continuously.
If you want anything more than speculation, you'll need to provide real information: Tomcat version, JRE version, platform, server.xml (with comments removed), context.xml for the webapp, etc.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org