You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by "James H. H. Lampert" <ja...@touchtonecorp.com> on 2017/08/18 19:48:43 UTC
Tomcat server apparently bouncing up and down
This is not with the Debian apt-get that I'm having trouble figuring out
where it's finding its webapp contexts.
It's with the downloaded-from-Apache one that I've got running (on port
7070, no HTTPS) on a CentOS 5 box, and that I haven't figured out how to
run as a service (but that's not the immediate problem).
It seems that the Tomcat server was apparently running fine for several
days (albeit unmonitored and without any firewall opening allowing it to
be reached from outside, up until yesterday afternoon), up until a few
hours ago. Then it started going down.
Yesterday afternoon, I put our webapp context on it, and changed the
port forwarding in the firewall, so that the outside access (and
therefore, the Site24x7 monitor on this particular server) pointed to
it, rather than to what it replaced.
And then I got a message from Site24x7 telling me that it had gone down.
After a brief inspection, I shut it down and restarted it.
Less than an hour later, the same story. And a third time.
The fourth time, I had my hands too full to go in and manually restart
it, and to my surprise, without my having restarted it, I got a message
from Site24x7 telling me it was back up.
And it bounced up and down a few times since then. I eventually shut it
down.
There's nothing in catalina.out between our webapp announcing that it
was running (9:14 AM), and the messages from my shutting it down (11:46 AM).
So far as I've been able to determine, when it's reported as down, it's
not accepting requests even from within the LAN.
Anybody ever seen anything like this before?
--
JHHL
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
[OT]Re: Tomcat server apparently bouncing up and down
Posted by Peter Kreuser <lo...@kreuser.name>.
Talking nicely and understandingly to it won't help either, I guess...
Have a nice weekend
Peter
> Am 19.08.2017 um 08:31 schrieb André Warnier (tomcat) <aw...@ice-sa.com>:
>
> 3 kids raised, 30 years of programming talking : slap it.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat server apparently bouncing up and down
Posted by "André Warnier (tomcat)" <aw...@ice-sa.com>.
3 kids raised, 30 years of programming talking : slap it.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat server apparently bouncing up and down
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
James,
Since you told the context is rather huge, have you checked gc times? A
long running full gc can block the machine completely resulting in the
up/down behaviour from outside. GC options depend on JVM version I use:
export JAVA_OPTS="$JAVA_OPTS -XX:+DisableExplicitGC -verbose:GC
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log "
alternatively moskito would also show you avg response times and gcs ;-)
regards
Leon
I agree. Long GC pauses can cause this type of issue. I have seen somewhat
similar. Apache PING failed due to long GC pause and user got 503s.
One question, do you see evidence of Tomcat restart in logs which
correlates with alerts?
Re: Tomcat server apparently bouncing up and down
Posted by Leon Rosenberg <ro...@gmail.com>.
Since you told the context is rather huge, have you checked gc times? A
long running full gc can block the machine completely resulting in the
up/down behaviour from outside. GC options depend on JVM version I use:
export JAVA_OPTS="$JAVA_OPTS -XX:+DisableExplicitGC -verbose:GC
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:logs/gc.log "
alternatively moskito would also show you avg response times and gcs ;-)
regards
Leon
On Fri, Aug 18, 2017 at 11:13 PM, James H. H. Lampert <
jamesl@touchtonecorp.com> wrote:
> On 8/18/17, 1:41 PM, Christopher Schultz wrote:
>
> You say that you aren't running it as a service. How then are you
>> running Tomcat?
>>
>
> startup.sh and shutdown.sh from a command line.
>
> Just starting catalina.sh from the CLI directly? If
>> you run it in the background, are you running it with nohup? If not,
>> your console closing might be killing the Java process. Hmm... but you
>> said that Tomcat does in fact shut down when you login and stop it.
>> Probably not a SIGHUP killing the process.
>>
>
> When it's unresponsive, it's apparently still running. But it's not just
> our context that's unresponsive; manager is also unresponsive. And we run
> with autodeploy disabled: aside from being a huge context that takes a
> while to deploy, it's also one that often needs to be stopped, have
> instance-specific values set in its web.inf, and then get restarted, before
> it can function normally.
>
> If you stop Tomcat (when it's unresponsive), then re-start it, does it
>> appear to work correctly right away, or do you need to do anything
>> else to get it to work again?
>>
>
> It opens up the port immediately, and serves a sign-on page for our webapp
> as soon as it's had a chance to initialize.
>
> I looked in the latest localhost access log, and no sign of anything
> suspicious there.
>
> --
> JHHL
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat server apparently bouncing up and down
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
James,
On 8/18/17 5:13 PM, James H. H. Lampert wrote:
> On 8/18/17, 1:41 PM, Christopher Schultz wrote:
>
>> You say that you aren't running it as a service. How then are
>> you running Tomcat?
>
> startup.sh and shutdown.sh from a command line.
>
>> Just starting catalina.sh from the CLI directly? If you run it in
>> the background, are you running it with nohup? If not, your
>> console closing might be killing the Java process. Hmm... but
>> you said that Tomcat does in fact shut down when you login and
>> stop it. Probably not a SIGHUP killing the process.
>
> When it's unresponsive, it's apparently still running. But it's not
> just our context that's unresponsive; manager is also unresponsive.
> And we run with autodeploy disabled: aside from being a huge
> context that takes a while to deploy, it's also one that often
> needs to be stopped, have instance-specific values set in its
> web.inf, and then get restarted, before it can function normally.
>
>> If you stop Tomcat (when it's unresponsive), then re-start it,
>> does it appear to work correctly right away, or do you need to do
>> anything else to get it to work again?
>
> It opens up the port immediately, and serves a sign-on page for
> our webapp as soon as it's had a chance to initialize.
>
> I looked in the latest localhost access log, and no sign of
> anything suspicious there.
If the JVM is still running, can you take a thread dump?
https://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my
_running_webapp_.3F
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJZm06OAAoJEBzwKT+lPKRYnk8P/0+248VvAIAFl07W5MOSpTk7
UpZqL4/aI21UWHHF61IqjvQyVcLyRR96jHut7crUSt1NpifjhZt/EWgfJTIvQDSs
Q2D7P8RH9OcNwT9v6H2q4eQW0/UW0dubyCcsSfx6Ts2qoCz6RLJr2ZIA7fOz+qT9
jG4ipYD3reUfmgxVrATjY7Lnfuehhnd8YX4FJnBZ1lg7p6zcHZDpqWj6yb/zTKIx
hW03bjy93cI+DWsgUFJCRH2LhA2I6bljihAyTrRfDKem9VF4MgOPmfGpvfZAC/6P
54tDvAAXj1KL7zGhGch479il6ZbkF7LllqV/0o6m9t1MhSes4PAmYWmbSf4CHLzC
8yyTyMAWm0+yavVpSALDtLlNWtmQMc7BbdwS3lYg5RvdM7E6DnmZlIeX0dQ9Nel+
3rg+4kGxYWsP3nx3MKWusTyA1qi9pNypWbY+ztY/+xYIzWj8EoAXIzcSgjZAHet1
jEJC+krV3VyewHYncyuv/aXfCtv4atVvtRHzJ1OWphTTp3/Axt3+5OD92MmfSTYj
5xhSNXu0xQmBZY05SZiWNblF3yyaShU1CTnUnC8cVx6vXGQ+CEZN71KnnH0ZX9lE
G4tJEidiWuCSSLrmxPul8eRWz9dA3vvA/C6eE4QetVOKaLSHY/gKCEUAKt+MQgFU
jugee3DZjdq4S4OUM9U2
=TOFu
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat server apparently bouncing up and down
Posted by "James H. H. Lampert" <ja...@touchtonecorp.com>.
On 8/18/17, 1:41 PM, Christopher Schultz wrote:
> You say that you aren't running it as a service. How then are you
> running Tomcat?
startup.sh and shutdown.sh from a command line.
> Just starting catalina.sh from the CLI directly? If
> you run it in the background, are you running it with nohup? If not,
> your console closing might be killing the Java process. Hmm... but you
> said that Tomcat does in fact shut down when you login and stop it.
> Probably not a SIGHUP killing the process.
When it's unresponsive, it's apparently still running. But it's not just
our context that's unresponsive; manager is also unresponsive. And we
run with autodeploy disabled: aside from being a huge context that takes
a while to deploy, it's also one that often needs to be stopped, have
instance-specific values set in its web.inf, and then get restarted,
before it can function normally.
> If you stop Tomcat (when it's unresponsive), then re-start it, does it
> appear to work correctly right away, or do you need to do anything
> else to get it to work again?
It opens up the port immediately, and serves a sign-on page for our
webapp as soon as it's had a chance to initialize.
I looked in the latest localhost access log, and no sign of anything
suspicious there.
--
JHHL
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat server apparently bouncing up and down
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
James,
On 8/18/17 3:48 PM, James H. H. Lampert wrote:
> This is not with the Debian apt-get that I'm having trouble
> figuring out where it's finding its webapp contexts.
>
> It's with the downloaded-from-Apache one that I've got running (on
> port 7070, no HTTPS) on a CentOS 5 box, and that I haven't figured
> out how to run as a service (but that's not the immediate
> problem).
>
> It seems that the Tomcat server was apparently running fine for
> several days (albeit unmonitored and without any firewall opening
> allowing it to be reached from outside, up until yesterday
> afternoon), up until a few hours ago. Then it started going down.
>
> Yesterday afternoon, I put our webapp context on it, and changed
> the port forwarding in the firewall, so that the outside access
> (and therefore, the Site24x7 monitor on this particular server)
> pointed to it, rather than to what it replaced.
>
> And then I got a message from Site24x7 telling me that it had gone
> down. After a brief inspection, I shut it down and restarted it.
>
> Less than an hour later, the same story. And a third time.
>
> The fourth time, I had my hands too full to go in and manually
> restart it, and to my surprise, without my having restarted it, I
> got a message from Site24x7 telling me it was back up.
>
> And it bounced up and down a few times since then. I eventually
> shut it down.
>
> There's nothing in catalina.out between our webapp announcing that
> it was running (9:14 AM), and the messages from my shutting it down
> (11:46 AM).
>
> So far as I've been able to determine, when it's reported as down,
> it's not accepting requests even from within the LAN.
>
> Anybody ever seen anything like this before?
Do you know how the monitoring service tests the service for its
"liveness"?
Is there anything (else) in the log files? There may be something in
the files other than catalina.out. I know that I still have trouble in
a dev environment with Tomcat repeatedly detecting changes on the
filesystem and re-deploying a context multiple times, even when there
have been no changes after the one that caused the initial reload.
Have you enabled an access log? If so, can you see the requests both
before and after the service appears to go offline?
You say that you aren't running it as a service. How then are you
running Tomcat? Just starting catalina.sh from the CLI directly? If
you run it in the background, are you running it with nohup? If not,
your console closing might be killing the Java process. Hmm... but you
said that Tomcat does in fact shut down when you login and stop it.
Probably not a SIGHUP killing the process.
If you stop Tomcat (when it's unresponsive), then re-start it, does it
appear to work correctly right away, or do you need to do anything
else to get it to work again?
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlmXUOYACgkQHPApP6U8
pFhrNw/+JKhveJc7Zg7gtZ4WBGrWa42jibjIXJOV62VgLRyjaCFIowltwxXpHz9F
S2p91A8MYGOYnvMMHU8QHwdC+KW6gR+SoKBVHImHzz1MUJpC6Oru/Np1dcGaM4Jg
7pn4SoWXeMCWrZqt/3keQ2YvsU8ZSUgh6957AF0pYnBmXNkkYZ3LMJ+2Y1UILWZA
+DBuuHRnn73YGPu0S8vLQvFr6PQwbbVUVmIe+Lp5i069G1FPHa+9zYdwlUjF5P9l
gewCgVidcQE1rSoNyW2FiiNqEtQgths4sV6a34DVi2e5ez6FMGMXk4qcrcZmdP3f
lOL0L+QYBvhfXXH/Vl83Bn42lQI2WMVv0nozb/NTr41kUVIbZBbERWQLRmmxqVGs
irz/L/6YgGcWX/Mj+ib9mP6SJpqSXrJwBlVeudkG8V4uVkEGzFIy3PwURCNScGrJ
9xVeBz+eBEU0kcgJGP8AcGaO6MuhEnvsuVgVLpSpCAv1m2jrB68/9a1LfU+gfnfL
vu6TIaDb0ivqMeD76y3XD4FLOINdAY/eKv6OSRGG+sGI7qCd/K3PQf7nj8qHnDVA
QlAsmE7tA6gJb8ULDOdTYFuLDgOoEpkaCHatMP/QAOZf8CgX4uHKB73r7VA2q5Kc
Kk0qIJ2stmnIah0BJVXk/sLAtYmxLOc7WzYzJf9SUK43tbZ6eM8=
=fmwC
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat server apparently bouncing up and down
Posted by "André Warnier (tomcat)" <aw...@ice-sa.com>.
On 20.08.2017 11:18, Chaitanya Sabbineni wrote:
> Hi All,
>
> If tomcat is not stopped properly that is if any of the demon threads are
> running under tomcat it will prevent jvm from starting as this will already
> hold the threads. So these threads has to be killed in order start it
> properly. Regarding the bouncing of the tomcat I.e starting and stopping
> any
*local*
> application can connedt
*locally*
> to the shutdown port and send a command stop to
> the tomcat which will stop the server.
>
Note: If you have any concern about this, you can change the port, and the address (IP on
which it listens)(*), and the shutdown command (**), in the server.xml -> <Server> tag.
http://tomcat.apache.org/tomcat-7.0-doc/config/server.html#Attributes
(*) which by default, /is/ localhost only, so don't change it.
(**) if you change this, you'll need to change your start/stop scripts too.
> On Sat, 19 Aug 2017 1:18 am James H. H. Lampert, <ja...@touchtonecorp.com>
> wrote:
>
>> This is not with the Debian apt-get that I'm having trouble figuring out
>> where it's finding its webapp contexts.
>>
>> It's with the downloaded-from-Apache one that I've got running (on port
>> 7070, no HTTPS) on a CentOS 5 box, and that I haven't figured out how to
>> run as a service (but that's not the immediate problem).
>>
>> It seems that the Tomcat server was apparently running fine for several
>> days (albeit unmonitored and without any firewall opening allowing it to
>> be reached from outside, up until yesterday afternoon), up until a few
>> hours ago. Then it started going down.
>>
>> Yesterday afternoon, I put our webapp context on it, and changed the
>> port forwarding in the firewall, so that the outside access (and
>> therefore, the Site24x7 monitor on this particular server) pointed to
>> it, rather than to what it replaced.
>>
>> And then I got a message from Site24x7 telling me that it had gone down.
>> After a brief inspection, I shut it down and restarted it.
>>
>> Less than an hour later, the same story. And a third time.
>>
>> The fourth time, I had my hands too full to go in and manually restart
>> it, and to my surprise, without my having restarted it, I got a message
>> from Site24x7 telling me it was back up.
>>
>> And it bounced up and down a few times since then. I eventually shut it
>> down.
>>
>> There's nothing in catalina.out between our webapp announcing that it
>> was running (9:14 AM), and the messages from my shutting it down (11:46
>> AM).
>>
>> So far as I've been able to determine, when it's reported as down, it's
>> not accepting requests even from within the LAN.
>>
>> Anybody ever seen anything like this before?
Yes, but there can be several reasons :
- TCP connections to port 7070 are no longer being accepted, because there is a
TCP/IP-level problem
- TCP connections to port 7070 are no longer being accepted, because there is nothing
listening there (no Tomcat e.g.)
- TCP connections are accepted, and the monitoring system can establish a connection to
it, and can send on it a HTTP request for "/", but Tomcat is not responding to it within
the timeout that the monitoring system expects (because tomcat is too busy e.g.)
- ...
You first need to find out precisely what the monitoring system is really telling you.
(It's certainly not saying that "tomcat is bouncing up and down"). Is it not able to
connect, or is it able to connect, but not getting an answer to a HTTP request ?
On your server, the command
netstat --tcp -aopn
(optionally :
netstat --tcp -aopn | grep 7070
)
will show you if something is listening on port 7070, and if it is really tomcat.
The command-line program "curl" (or perl's "lwp-request") can help you find out if tomcat
is answering requests.
Turning on tomcat's Access Log can also help.
http://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Access_Log_Valve
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat server apparently bouncing up and down
Posted by Chaitanya Sabbineni <sa...@gmail.com>.
Hi All,
If tomcat is not stopped properly that is if any of the demon threads are
running under tomcat it will prevent jvm from starting as this will already
hold the threads. So these threads has to be killed in order start it
properly. Regarding the bouncing of the tomcat I.e starting and stopping
any application can connedt to the shutdown port and send a command stop to
the tomcat which will stop the server.
On Sat, 19 Aug 2017 1:18 am James H. H. Lampert, <ja...@touchtonecorp.com>
wrote:
> This is not with the Debian apt-get that I'm having trouble figuring out
> where it's finding its webapp contexts.
>
> It's with the downloaded-from-Apache one that I've got running (on port
> 7070, no HTTPS) on a CentOS 5 box, and that I haven't figured out how to
> run as a service (but that's not the immediate problem).
>
> It seems that the Tomcat server was apparently running fine for several
> days (albeit unmonitored and without any firewall opening allowing it to
> be reached from outside, up until yesterday afternoon), up until a few
> hours ago. Then it started going down.
>
> Yesterday afternoon, I put our webapp context on it, and changed the
> port forwarding in the firewall, so that the outside access (and
> therefore, the Site24x7 monitor on this particular server) pointed to
> it, rather than to what it replaced.
>
> And then I got a message from Site24x7 telling me that it had gone down.
> After a brief inspection, I shut it down and restarted it.
>
> Less than an hour later, the same story. And a third time.
>
> The fourth time, I had my hands too full to go in and manually restart
> it, and to my surprise, without my having restarted it, I got a message
> from Site24x7 telling me it was back up.
>
> And it bounced up and down a few times since then. I eventually shut it
> down.
>
> There's nothing in catalina.out between our webapp announcing that it
> was running (9:14 AM), and the messages from my shutting it down (11:46
> AM).
>
> So far as I've been able to determine, when it's reported as down, it's
> not accepting requests even from within the LAN.
>
> Anybody ever seen anything like this before?
>
> --
> JHHL
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>