You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jeff Turner <je...@socialchange.net.au> on 2001/09/10 04:55:27 UTC
The pitfalls in restarting tomcat
Hi,
I was wondering how people who use Tomcat every day) do the stop/start
cycle that's required whenever something in WEB-INF/lib changes.
"./shutdown.sh ; ./startup.sh" is a bad idea, because Tomcat 3.x seems
to shut down it's threads asynchronously. So after shutdown.sh has
returned, Tomcat has not necessarily stopped. Then startup.sh comes
along, finds that your port is in use, and gives you "Address already in
use" errors.
With Tomcat 3.3, it is very easy to confuse Tomcat into thinking that it
has shut down (the ajp12.id file does not exist), but it actually
running. In this (common) situation, there is no way to kill tomcat
other than killing the processes ('killall java'). Killing tomcat in
this way is *very* dangerous, because it sometimes leaves threads in the
state described by 'man ps' as:
D uninterruptible sleep (usually IO)
Then you're plain screwed; the thread is completely unkillable even by
root, and is holding onto your tomcat port (8080). The only option is to
reboot the machine.
So anyway, has anyone got a safer way of restarting tomcat? Perhaps a
script that waits until Tomcat is *really* dead before restarting? It
would be nice if there was direct support in Tomcat for this everyday
task (a restart.sh script).
--Jeff
Re: The pitfalls in restarting tomcat
Posted by Jonathan Eric Miller <je...@uchicago.edu>.
----- Original Message -----
From: "Craig R. McClanahan" <cr...@apache.org>
To: <to...@jakarta.apache.org>
Sent: Thursday, September 13, 2001 9:08 PM
Subject: Re: The pitfalls in restarting tomcat
> On Thu, 13 Sep 2001, Jonathan Eric Miller wrote:
>
> > Date: Thu, 13 Sep 2001 21:00:01 -0500
> > From: Jonathan Eric Miller <je...@uchicago.edu>
> > Reply-To: tomcat-user@jakarta.apache.org
> > To: tomcat-user@jakarta.apache.org
> > Subject: Re: The pitfalls in restarting tomcat
> >
> >
> > ----- Original Message -----
> > From: "Craig R. McClanahan" <cr...@apache.org>
> > To: <to...@jakarta.apache.org>
> > Sent: Thursday, September 13, 2001 7:40 PM
> > Subject: Re: The pitfalls in restarting tomcat
> >
> >
> > >
> > >
> > > On Thu, 13 Sep 2001, Jonathan Eric Miller wrote:
> > >
> > > > Date: Thu, 13 Sep 2001 19:24:08 -0500
> > > > From: Jonathan Eric Miller <je...@uchicago.edu>
> > > > Reply-To: tomcat-user@jakarta.apache.org
> > > > To: tomcat-user@jakarta.apache.org
> > > > Subject: Re: The pitfalls in restarting tomcat
> > > >
> > > > IMHO, it would still be nice to have a true easy way to completely
> > restart
> > > > Tomcat. For example, say you were running into memory leak problems
or
> > > > something like that. You might want to schedule a script to run once
a
> > day
> > > > to restart the server. In a perfect world, one would never have to
do
> > this,
> > > > but, sometimes you run into bugs and you want to make sure that
things
> > are
> > > > completely reset.
> > > >
> > >
> > > On Unix, many /etc/rc.d/init.d scripts I've seen implement their
"restart"
> > > command as a shutdown followed by a startup. Why is it any more
> > > complicated than this?
> >
> > You have to put a pause in there too, otherwise you'll get an error
message
> > about the port already listening. This is because when you start it,
> > sometimes, the previous instance hasn't stopped yet.
> >
> A pause won't cut it -- the amount of time a shutdown takes is
> non-deterministic, because the destroy() method of all the initialized
> servlets, filters, and the contextDestroyed() method of listeners is
> called during the shutdown process. It would take code to do this
> reliably.
My point exactly. ;-)
> > > Note also that, if you don't physically restart the JVM, you don't
give
> > > any of the memory it grabbed back to the operating system.
> >
> > The restart command that I'm thinking of would completely stop Tomcat
and
> > start it again, and also shut down the JVM. It would basically, run
> > shutdown.sh and then startup.sh. However, the difference is that it
would
> > shut it down in a synchronous manner not asynchronous. Therefore, it
would
> > start the server back up immediately after it was shut down rather than
> > pausing for an arbitrary period of time waiting for Tomcat to shut down.
> >
> > I think this may be becoming less of an issue than it was before. Prior
to
> > Tomcat 7, the shutdown process seemed very laggy. As of 7 it was a lot
> > better. Now, with RC1, maybe it's just me, but, it seems even better
yet.
> >
>
> There have been substantial improvements in RC1.
>
> > On a somewhat unrelated note, but, speaking of lag, anyone know if Sun
is
> > planning doing anything to speed up the initialization of JSSE?
> >
>
> I would bet this is related to initializing the random number generator.
Yup.
> Do you *really* want to reduce the security of your cryptography?
No, but, what I want to know is why I'm able to start Apache Web Server with
SSL and I don't get this lag? I'm pretty sure that IIS with SSL doesn't take
that long either. There are also numerous other SSL enabled clients that
don't suffer from this kind of lag. I can see if it was only servers that
suffered from this, but that isn't the case. Say I want to write a console
app that is the rough equivalent of ldapsearch in Java that uses SSL.
Everytime, I run that program I'm going to get 15 seconds of lag before it
does anything. iPlanet's ldapsearch doesn't take that long. IMHO, they
should speed it up using native code if that's what it takes (now that JSSE
comes standard with JDK 1.4).
Jon
> The same issue shows up in Tomcat with initialization of the random number
> generator used for session ids. The current initialization algorithm is
> fast, but subject to predictable session ids if an attacker can read
> server.xml.
>
> > Jon
> >
> > > > Jon
> > > >
> > >
> > > Craig
> >
> >
> >
> Craig
>
>
Re: The pitfalls in restarting tomcat
Posted by "Craig R. McClanahan" <cr...@apache.org>.
On Thu, 13 Sep 2001, Jonathan Eric Miller wrote:
> Date: Thu, 13 Sep 2001 21:00:01 -0500
> From: Jonathan Eric Miller <je...@uchicago.edu>
> Reply-To: tomcat-user@jakarta.apache.org
> To: tomcat-user@jakarta.apache.org
> Subject: Re: The pitfalls in restarting tomcat
>
>
> ----- Original Message -----
> From: "Craig R. McClanahan" <cr...@apache.org>
> To: <to...@jakarta.apache.org>
> Sent: Thursday, September 13, 2001 7:40 PM
> Subject: Re: The pitfalls in restarting tomcat
>
>
> >
> >
> > On Thu, 13 Sep 2001, Jonathan Eric Miller wrote:
> >
> > > Date: Thu, 13 Sep 2001 19:24:08 -0500
> > > From: Jonathan Eric Miller <je...@uchicago.edu>
> > > Reply-To: tomcat-user@jakarta.apache.org
> > > To: tomcat-user@jakarta.apache.org
> > > Subject: Re: The pitfalls in restarting tomcat
> > >
> > > IMHO, it would still be nice to have a true easy way to completely
> restart
> > > Tomcat. For example, say you were running into memory leak problems or
> > > something like that. You might want to schedule a script to run once a
> day
> > > to restart the server. In a perfect world, one would never have to do
> this,
> > > but, sometimes you run into bugs and you want to make sure that things
> are
> > > completely reset.
> > >
> >
> > On Unix, many /etc/rc.d/init.d scripts I've seen implement their "restart"
> > command as a shutdown followed by a startup. Why is it any more
> > complicated than this?
>
> You have to put a pause in there too, otherwise you'll get an error message
> about the port already listening. This is because when you start it,
> sometimes, the previous instance hasn't stopped yet.
>
A pause won't cut it -- the amount of time a shutdown takes is
non-deterministic, because the destroy() method of all the initialized
servlets, filters, and the contextDestroyed() method of listeners is
called during the shutdown process. It would take code to do this
reliably.
> > Note also that, if you don't physically restart the JVM, you don't give
> > any of the memory it grabbed back to the operating system.
>
> The restart command that I'm thinking of would completely stop Tomcat and
> start it again, and also shut down the JVM. It would basically, run
> shutdown.sh and then startup.sh. However, the difference is that it would
> shut it down in a synchronous manner not asynchronous. Therefore, it would
> start the server back up immediately after it was shut down rather than
> pausing for an arbitrary period of time waiting for Tomcat to shut down.
>
> I think this may be becoming less of an issue than it was before. Prior to
> Tomcat 7, the shutdown process seemed very laggy. As of 7 it was a lot
> better. Now, with RC1, maybe it's just me, but, it seems even better yet.
>
There have been substantial improvements in RC1.
> On a somewhat unrelated note, but, speaking of lag, anyone know if Sun is
> planning doing anything to speed up the initialization of JSSE?
>
I would bet this is related to initializing the random number generator.
Do you *really* want to reduce the security of your cryptography?
The same issue shows up in Tomcat with initialization of the random number
generator used for session ids. The current initialization algorithm is
fast, but subject to predictable session ids if an attacker can read
server.xml.
> Jon
>
> > > Jon
> > >
> >
> > Craig
>
>
>
Craig
Re: The pitfalls in restarting tomcat
Posted by Jonathan Eric Miller <je...@uchicago.edu>.
----- Original Message -----
From: "Craig R. McClanahan" <cr...@apache.org>
To: <to...@jakarta.apache.org>
Sent: Thursday, September 13, 2001 7:40 PM
Subject: Re: The pitfalls in restarting tomcat
>
>
> On Thu, 13 Sep 2001, Jonathan Eric Miller wrote:
>
> > Date: Thu, 13 Sep 2001 19:24:08 -0500
> > From: Jonathan Eric Miller <je...@uchicago.edu>
> > Reply-To: tomcat-user@jakarta.apache.org
> > To: tomcat-user@jakarta.apache.org
> > Subject: Re: The pitfalls in restarting tomcat
> >
> > IMHO, it would still be nice to have a true easy way to completely
restart
> > Tomcat. For example, say you were running into memory leak problems or
> > something like that. You might want to schedule a script to run once a
day
> > to restart the server. In a perfect world, one would never have to do
this,
> > but, sometimes you run into bugs and you want to make sure that things
are
> > completely reset.
> >
>
> On Unix, many /etc/rc.d/init.d scripts I've seen implement their "restart"
> command as a shutdown followed by a startup. Why is it any more
> complicated than this?
You have to put a pause in there too, otherwise you'll get an error message
about the port already listening. This is because when you start it,
sometimes, the previous instance hasn't stopped yet.
> Note also that, if you don't physically restart the JVM, you don't give
> any of the memory it grabbed back to the operating system.
The restart command that I'm thinking of would completely stop Tomcat and
start it again, and also shut down the JVM. It would basically, run
shutdown.sh and then startup.sh. However, the difference is that it would
shut it down in a synchronous manner not asynchronous. Therefore, it would
start the server back up immediately after it was shut down rather than
pausing for an arbitrary period of time waiting for Tomcat to shut down.
I think this may be becoming less of an issue than it was before. Prior to
Tomcat 7, the shutdown process seemed very laggy. As of 7 it was a lot
better. Now, with RC1, maybe it's just me, but, it seems even better yet.
On a somewhat unrelated note, but, speaking of lag, anyone know if Sun is
planning doing anything to speed up the initialization of JSSE?
Jon
> > Jon
> >
>
> Craig
Re: The pitfalls in restarting tomcat
Posted by "Craig R. McClanahan" <cr...@apache.org>.
On Thu, 13 Sep 2001, Jonathan Eric Miller wrote:
> Date: Thu, 13 Sep 2001 19:24:08 -0500
> From: Jonathan Eric Miller <je...@uchicago.edu>
> Reply-To: tomcat-user@jakarta.apache.org
> To: tomcat-user@jakarta.apache.org
> Subject: Re: The pitfalls in restarting tomcat
>
> IMHO, it would still be nice to have a true easy way to completely restart
> Tomcat. For example, say you were running into memory leak problems or
> something like that. You might want to schedule a script to run once a day
> to restart the server. In a perfect world, one would never have to do this,
> but, sometimes you run into bugs and you want to make sure that things are
> completely reset.
>
On Unix, many /etc/rc.d/init.d scripts I've seen implement their "restart"
command as a shutdown followed by a startup. Why is it any more
complicated than this?
Note also that, if you don't physically restart the JVM, you don't give
any of the memory it grabbed back to the operating system.
> Jon
>
Craig
Re: The pitfalls in restarting tomcat
Posted by Jonathan Eric Miller <je...@uchicago.edu>.
IMHO, it would still be nice to have a true easy way to completely restart
Tomcat. For example, say you were running into memory leak problems or
something like that. You might want to schedule a script to run once a day
to restart the server. In a perfect world, one would never have to do this,
but, sometimes you run into bugs and you want to make sure that things are
completely reset.
Jon
----- Original Message -----
From: "Craig R. McClanahan" <cr...@apache.org>
To: <to...@jakarta.apache.org>
Sent: Thursday, September 13, 2001 10:57 AM
Subject: Re: The pitfalls in restarting tomcat
>
>
> On Wed, 12 Sep 2001, Jonathan Eric Miller wrote:
>
> > Date: Wed, 12 Sep 2001 23:57:07 -0500
> > From: Jonathan Eric Miller <je...@uchicago.edu>
> > Reply-To: tomcat-user@jakarta.apache.org
> > To: tomcat-user@jakarta.apache.org
> > Subject: Re: The pitfalls in restarting tomcat
> >
> > I agree that there should be a restart.sh. However, it is possible to
> > restart/reload a servlet without having to stop and start Tomcat, in
version
> > 4 that is, which is due for release any day now. I think you can do it
in
> > Tomcat 3 as well. In 4, you can mark a Context as reloadable. Then, it
will
> > check the files when they are requested, and if the time stamp is new,
then,
> > it will automatically reload the servlet. Also, there is a management
> > servlet that allows you to restart other servlets in case you don't want
to
> > have auto-reloading on.
> >
> > Nonetheless, I still think a restart.sh command would be useful.
> >
>
> In Tomcat 4, you can also use the Manager webapp to restart a particular
> app at any time (whether or not you have declared it to be reloadable)
> through an HTTP request like:
>
> http://localhost:8080/manager/reload?path=/exmaples
>
> This can also be scripted into shell scripts if you need to restart
> periodically for some reason (such as to switch log files).
>
> For more info, see
>
> http://jakarta.apache.org/tomcat/tomcat-4.0-doc/manager-howto.html
>
> > Jon
> >
>
> Craig
>
> > ----- Original Message -----
> > From: "Jeff Turner" <je...@socialchange.net.au>
> > To: <to...@jakarta.apache.org>
> > Sent: Sunday, September 09, 2001 9:55 PM
> > Subject: The pitfalls in restarting tomcat
> >
> >
> > > Hi,
> > >
> > > I was wondering how people who use Tomcat every day) do the stop/start
> > > cycle that's required whenever something in WEB-INF/lib changes.
> > >
> > > "./shutdown.sh ; ./startup.sh" is a bad idea, because Tomcat 3.x seems
> > > to shut down it's threads asynchronously. So after shutdown.sh has
> > > returned, Tomcat has not necessarily stopped. Then startup.sh comes
> > > along, finds that your port is in use, and gives you "Address already
in
> > > use" errors.
> > >
> > > With Tomcat 3.3, it is very easy to confuse Tomcat into thinking that
it
> > > has shut down (the ajp12.id file does not exist), but it actually
> > > running. In this (common) situation, there is no way to kill tomcat
> > > other than killing the processes ('killall java'). Killing tomcat in
> > > this way is *very* dangerous, because it sometimes leaves threads in
the
> > > state described by 'man ps' as:
> > >
> > > D uninterruptible sleep (usually IO)
> > >
> > > Then you're plain screwed; the thread is completely unkillable even by
> > > root, and is holding onto your tomcat port (8080). The only option is
to
> > > reboot the machine.
> > >
> > > So anyway, has anyone got a safer way of restarting tomcat? Perhaps a
> > > script that waits until Tomcat is *really* dead before restarting? It
> > > would be nice if there was direct support in Tomcat for this everyday
> > > task (a restart.sh script).
> > >
> > > --Jeff
> > >
> >
> >
>
Re: The pitfalls in restarting tomcat
Posted by "Craig R. McClanahan" <cr...@apache.org>.
On Wed, 12 Sep 2001, Jonathan Eric Miller wrote:
> Date: Wed, 12 Sep 2001 23:57:07 -0500
> From: Jonathan Eric Miller <je...@uchicago.edu>
> Reply-To: tomcat-user@jakarta.apache.org
> To: tomcat-user@jakarta.apache.org
> Subject: Re: The pitfalls in restarting tomcat
>
> I agree that there should be a restart.sh. However, it is possible to
> restart/reload a servlet without having to stop and start Tomcat, in version
> 4 that is, which is due for release any day now. I think you can do it in
> Tomcat 3 as well. In 4, you can mark a Context as reloadable. Then, it will
> check the files when they are requested, and if the time stamp is new, then,
> it will automatically reload the servlet. Also, there is a management
> servlet that allows you to restart other servlets in case you don't want to
> have auto-reloading on.
>
> Nonetheless, I still think a restart.sh command would be useful.
>
In Tomcat 4, you can also use the Manager webapp to restart a particular
app at any time (whether or not you have declared it to be reloadable)
through an HTTP request like:
http://localhost:8080/manager/reload?path=/exmaples
This can also be scripted into shell scripts if you need to restart
periodically for some reason (such as to switch log files).
For more info, see
http://jakarta.apache.org/tomcat/tomcat-4.0-doc/manager-howto.html
> Jon
>
Craig
> ----- Original Message -----
> From: "Jeff Turner" <je...@socialchange.net.au>
> To: <to...@jakarta.apache.org>
> Sent: Sunday, September 09, 2001 9:55 PM
> Subject: The pitfalls in restarting tomcat
>
>
> > Hi,
> >
> > I was wondering how people who use Tomcat every day) do the stop/start
> > cycle that's required whenever something in WEB-INF/lib changes.
> >
> > "./shutdown.sh ; ./startup.sh" is a bad idea, because Tomcat 3.x seems
> > to shut down it's threads asynchronously. So after shutdown.sh has
> > returned, Tomcat has not necessarily stopped. Then startup.sh comes
> > along, finds that your port is in use, and gives you "Address already in
> > use" errors.
> >
> > With Tomcat 3.3, it is very easy to confuse Tomcat into thinking that it
> > has shut down (the ajp12.id file does not exist), but it actually
> > running. In this (common) situation, there is no way to kill tomcat
> > other than killing the processes ('killall java'). Killing tomcat in
> > this way is *very* dangerous, because it sometimes leaves threads in the
> > state described by 'man ps' as:
> >
> > D uninterruptible sleep (usually IO)
> >
> > Then you're plain screwed; the thread is completely unkillable even by
> > root, and is holding onto your tomcat port (8080). The only option is to
> > reboot the machine.
> >
> > So anyway, has anyone got a safer way of restarting tomcat? Perhaps a
> > script that waits until Tomcat is *really* dead before restarting? It
> > would be nice if there was direct support in Tomcat for this everyday
> > task (a restart.sh script).
> >
> > --Jeff
> >
>
>
Re: The pitfalls in restarting tomcat
Posted by Jonathan Eric Miller <je...@uchicago.edu>.
I agree that there should be a restart.sh. However, it is possible to
restart/reload a servlet without having to stop and start Tomcat, in version
4 that is, which is due for release any day now. I think you can do it in
Tomcat 3 as well. In 4, you can mark a Context as reloadable. Then, it will
check the files when they are requested, and if the time stamp is new, then,
it will automatically reload the servlet. Also, there is a management
servlet that allows you to restart other servlets in case you don't want to
have auto-reloading on.
Nonetheless, I still think a restart.sh command would be useful.
Jon
----- Original Message -----
From: "Jeff Turner" <je...@socialchange.net.au>
To: <to...@jakarta.apache.org>
Sent: Sunday, September 09, 2001 9:55 PM
Subject: The pitfalls in restarting tomcat
> Hi,
>
> I was wondering how people who use Tomcat every day) do the stop/start
> cycle that's required whenever something in WEB-INF/lib changes.
>
> "./shutdown.sh ; ./startup.sh" is a bad idea, because Tomcat 3.x seems
> to shut down it's threads asynchronously. So after shutdown.sh has
> returned, Tomcat has not necessarily stopped. Then startup.sh comes
> along, finds that your port is in use, and gives you "Address already in
> use" errors.
>
> With Tomcat 3.3, it is very easy to confuse Tomcat into thinking that it
> has shut down (the ajp12.id file does not exist), but it actually
> running. In this (common) situation, there is no way to kill tomcat
> other than killing the processes ('killall java'). Killing tomcat in
> this way is *very* dangerous, because it sometimes leaves threads in the
> state described by 'man ps' as:
>
> D uninterruptible sleep (usually IO)
>
> Then you're plain screwed; the thread is completely unkillable even by
> root, and is holding onto your tomcat port (8080). The only option is to
> reboot the machine.
>
> So anyway, has anyone got a safer way of restarting tomcat? Perhaps a
> script that waits until Tomcat is *really* dead before restarting? It
> would be nice if there was direct support in Tomcat for this everyday
> task (a restart.sh script).
>
> --Jeff
>
RE: The pitfalls in restarting tomcat
Posted by Aravind Naidu <ar...@eha.net>.
or you can use (if you are on Linux that is) the command
"pstree -p" to easily view the parent process for Tomcat.
-- Aravind
-----Original Message-----
From: Pier Fumagalli [mailto:pier@betaversion.org]
Sent: Monday, 10 September 2001 11:49
To: tomcat-user@jakarta.apache.org
Subject: Re: The pitfalls in restarting tomcat
"Jeff Turner" <je...@socialchange.net.au> wrote:
> With Tomcat 3.3, it is very easy to confuse Tomcat into thinking that it
> has shut down (the ajp12.id file does not exist), but it actually
> running. In this (common) situation, there is no way to kill tomcat
> other than killing the processes ('killall java'). Killing tomcat in
> this way is *very* dangerous, because it sometimes leaves threads in the
> state described by 'man ps' as:
>
> D uninterruptible sleep (usually IO)
Of course you don't have to killall, but kill only the parent process Java,
the real process, as all the others are just threads... (Linux sucks on
that)
Pier
Re: The pitfalls in restarting tomcat
Posted by Pier Fumagalli <pi...@betaversion.org>.
"Jeff Turner" <je...@socialchange.net.au> wrote:
> With Tomcat 3.3, it is very easy to confuse Tomcat into thinking that it
> has shut down (the ajp12.id file does not exist), but it actually
> running. In this (common) situation, there is no way to kill tomcat
> other than killing the processes ('killall java'). Killing tomcat in
> this way is *very* dangerous, because it sometimes leaves threads in the
> state described by 'man ps' as:
>
> D uninterruptible sleep (usually IO)
Of course you don't have to killall, but kill only the parent process Java,
the real process, as all the others are just threads... (Linux sucks on
that)
Pier
Re: The pitfalls in restarting tomcat
Posted by Pier Fumagalli <pi...@betaversion.org>.
"Jeff Turner" <je...@socialchange.net.au> wrote:
> So anyway, has anyone got a safer way of restarting tomcat? Perhaps a
> script that waits until Tomcat is *really* dead before restarting? It
> would be nice if there was direct support in Tomcat for this everyday
> task (a restart.sh script).
BTW, forgot to mention that... At the end, the code in
jakarta-tomcat-service should be able to take care that with no problems.
Pier