You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Dave Stubbs <da...@stubbs.uk.com> on 2011/09/09 10:54:36 UTC
Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
We're experiencing a problem on both production and test systems that has
only started to occur in the last few weeks (no new version of any software
have been put live to anyone's knowledge) that has us quite stumped.
We are seeing tomcat starting up additional copies of itself, each new copy
is allocating a chunk of storage, it only starts 1 thread, nothing gets
written to any logs and no CPU is being listed as having been used.
This keeps happening, and if it's not monitored the system memory all gets
taken up and we start to get SIGBUS errors.
We've tried turning all logging on, there don't appear to be any software
related events at the heart of the issue, and we can't see any system
triggers.
When we list the system tasks we see that the ghost process PPID is the same
as the PID of the still running main tomcat instance. Here is list of what
happened when it wasn't picked up for a while.
tomcat 21115 21114 0 14:46:44 ? 193:04 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
tomcat 773 21115 0 17:29:27 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
tomcat 4270 21115 0 18:29:29 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
tomcat 5988 21115 0 18:59:32 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
tomcat 27209 21115 0 01:01:02 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
tomcat 13155 21115 0 05:32:09 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
tomcat 16785 21115 0 06:34:52 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
tomcat 12948 21115 0 20:58:50 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
tomcat 20054 21115 0 23:00:21 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
tomcat 23730 21115 0 00:03:47 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
The main process 21115 starts off fine, and is even still running with no
problems the next day, but we keep getting these ghost processes (for want
of a better description).
We have multiple tomcat servers running on the same system, but this is the
only one experiencing the problem, all of the others are unaffected.
The web apps themselves are pretty standard, servlets, JSPs, web services,
Oracle DB connection and a JMX link to an Apache MQ server for monitoring
purposes. This isn't even a heavily used application.
The web has come up empty for us, we can't find any information on this
fault at all, so would really appreciate any help you may be able to give
us.
Regards
Dave
RE: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts
additional copies of itself with only 1 thread and no CPU
Posted by Dave Stubbs <da...@stubbs.uk.com>.
This is on Solaris 10 which doesn't have that issue. Each process is
allocating its own storage so it looks like it's completely separate from
the parent process, threads would share the parents storage space.
No, we definitely aren't starting any additional processes ourselves.
This is happening not only on the test but on the production systems which
are black boxed.
-----Original Message-----
From: Konstantin Kolinko [mailto:knst.kolinko@gmail.com]
Sent: 09 September 2011 12:02
To: Tomcat Users List
Subject: Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
2011/9/9 Dave Stubbs <da...@stubbs.uk.com>:
> We are seeing tomcat starting up additional copies of itself, each new
>copy is allocating a chunk of storage, it only starts 1 thread,
>nothing gets written to any logs and no CPU is being listed as having been
used.
>(...)
1. It might be an issue with "ps" utility that it shows threads as separate
processes:
In it already mentioned in the FAQ:
http://wiki.apache.org/tomcat/FAQ/Linux_Unix#Q1
In Tomcat 6 and later you can configure an <Executor> so that unneeded
request processing threads can be stopped.
Without an <Executor> the threads will be created if concurrent load on your
Tomcat increases, so active thread count will increase until it reaches the
default count of 200 request processing threads on <Connector>.
2. Are you starting any additional processes by yourself? (E.g. when using
CGIServlet).
> tomcat 21115 21114 0 14:46:44 ? 193:04 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
> 773 21115 0 17:29:27 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
Posted by Konstantin Kolinko <kn...@gmail.com>.
2011/9/9 Dave Stubbs <da...@stubbs.uk.com>:
> We are seeing tomcat starting up additional copies of itself, each new copy
> is allocating a chunk of storage, it only starts 1 thread, nothing gets
> written to any logs and no CPU is being listed as having been used.
>(...)
1. It might be an issue with "ps" utility that it shows threads as
separate processes:
In it already mentioned in the FAQ:
http://wiki.apache.org/tomcat/FAQ/Linux_Unix#Q1
In Tomcat 6 and later you can configure an <Executor> so that unneeded
request processing threads can be stopped.
Without an <Executor> the threads will be created if concurrent load
on your Tomcat increases, so active thread count will increase until
it reaches the default count of 200 request processing threads on
<Connector>.
2. Are you starting any additional processes by yourself? (E.g. when
using CGIServlet).
> tomcat 21115 21114 0 14:46:44 ? 193:04 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
> tomcat 773 21115 0 17:29:27 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts
additional copies of itself with only 1 thread and no CPU
Posted by Dave Stubbs <da...@stubbs.uk.com>.
I'll get that checked when it next happens (it's doing it daily at the
moment sometime several times).
We've already tried to match the events with the activity on the parent JVM,
haven't found anything yet though (haven't given up on this).
One thing we did notice is higher CPU usage on the main JVM. We're not sure
yet if this happens before or after is creates the additional JVMs, or if
that's just due to high load on the application. We're monitoring that as
well.
-----Original Message-----
From: Tobias Crefeld [mailto:tc@cataneo.eu]
Sent: 09 September 2011 11:11
To: users@tomcat.apache.org
Subject: Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
Am Fri, 9 Sep 2011 09:54:36 +0100
schrieb "Dave Stubbs" <da...@stubbs.uk.com>:
> When we list the system tasks we see that the ghost process PPID is
> the same as the PID of the still running main tomcat instance. Here is
> list of what happened when it wasn't picked up for a while.
Obviously the "parent"-JVM itself starts the other JVMs, so it must be an
application issue.
Do the "child-JVMs" stop when 21115 shuts down?
Do the child-JVMs terminate on "normal" signals like -SIGTERM?
I would try to log the exact time, when these additional JVM are being
started and compare these with things that were happening on the
"parent"-Tomcat - maybe you find a application dependency that way.
Just an idea: Maybe the parent-JVM gets a signal to restart itself that
isn't handled correct?
(
http://publib.boulder.ibm.com/infocenter/realtime/v1r0/topic/com.ibm.rt.doc.
10/user/signals.html
- sorry, but haven't found a source for Sun-JDK instantly)
Regards,
Tobias.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
Posted by Tobias Crefeld <tc...@cataneo.eu>.
Am Fri, 9 Sep 2011 09:54:36 +0100
schrieb "Dave Stubbs" <da...@stubbs.uk.com>:
> When we list the system tasks we see that the ghost process PPID is
> the same as the PID of the still running main tomcat instance. Here
> is list of what happened when it wasn't picked up for a while.
Obviously the "parent"-JVM itself starts the other JVMs, so it must be
an application issue.
Do the "child-JVMs" stop when 21115 shuts down?
Do the child-JVMs terminate on "normal" signals like -SIGTERM?
I would try to log the exact time, when these additional JVM are being
started and compare these with things that were happening on the
"parent"-Tomcat - maybe you find a application dependency that way.
Just an idea: Maybe the parent-JVM gets a signal to restart itself that
isn't handled correct?
( http://publib.boulder.ibm.com/infocenter/realtime/v1r0/topic/com.ibm.rt.doc.10/user/signals.html
- sorry, but haven't found a source for Sun-JDK instantly)
Regards,
Tobias.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
Posted by Tobias Crefeld <tc...@cataneo.eu>.
Am Fri, 09 Sep 2011 11:25:13 +0200
schrieb André Warnier <aw...@ice-sa.com>:
> What does the command "netstat -pan | grep LISTEN" show when you have
> such multiple copies running ?
Unfortunately Solaris' netstat doesn't know a option to show listeners.
IIRC there are some scripts in Solaris-world that should do the job.
Regards,
Tobias Crefeld.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts
additional copies of itself with only 1 thread and no CPU
Posted by Dave Stubbs <da...@stubbs.uk.com>.
The tomcat service is managed by the Solaris service management system (I
can see references to svcs in their start-up/shutdown logs (it's on a
customer's site so I'm not 100% on how it's configured).
I should say there are 2 other tomcat instances all being managed the same
way and they don't have any problems at all.
We've investigated the shutdown and start-up systems thoroughly and are
happy that it's not an external agent that's trying to fire up another
tomcat instance. In fact we've manually fired up a second instance and it
comes up completely differently to these ghost processes, which as I
mentioned before are allocating a whole new chunk of memory but then only
running 1 thread and taking no CPU at all.
-----Original Message-----
From: André Warnier [mailto:aw@ice-sa.com]
Sent: 09 September 2011 13:35
To: Tomcat Users List
Subject: Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
Ok, fishing in the dark really, but trying to accumulate more clues.
What about the other questions (how is Tomcat started) ?
And, looking at your original "ps" list, some vague thing :
all these ghost processes seem to
a) not be doing much (CPU time 0), even after a long time.
b) all be started around the clock hour:00, or the hour:30, give or take a
couple of minutes.
Got anything particular that would run at approximately 30 minutes interval
?
(Like some monitoring process which sends a request every 30 minutes just to
check ? And if so : has this been added just before the problem started ?
and what request does it send ?)
As an alternative to netstat under Solaris, there is the "lsof" utility.
It has a million options to sort through, but it certainly can tell you
which process owns which TCP socket.
The point here is to try to figure out what these ghosts are doing and which
configuration settings they use. They seem to be using the same
command-line settings as the original, but we only see a small part (there
may be other switches on your version of "ps" which list the whole command
line of each process.).
You cannot have several processes opening the same TCP port in "LISTEN"
mode.
If all these processes use the same config and try to open the same ports
(Tomcat opens several), all but the first one should fail, and that should
normally leave a trace in the Tomcat logs.
Else, they are not real processes and really are threads, no matter what
"ps" is saying.
You could also try taking some thread dumps of the Tomcat JVM (see the
Tomcat FAQ), to see if some thread-id there match the PID's seen in your ps
displays.
Dave Stubbs wrote:
> Solaris environment is....
>
> SunOS ********** 5.10 Generic_127127-11 sun4v sparc
> SUNW,SPARC-Enterprise-T5120
>
> System = SunOS
> Node = **********
> Release = 5.10
> KernelID = Generic_127127-11
> Machine = sun4v
> BusType = <unknown>
> Serial = <unknown>
> Users = <unknown>
> OEM# = 0
> Origin# = 1
> NumCPU = 64
>
> -----Original Message-----
> From: André Warnier [mailto:aw@ice-sa.com]
> Sent: 09 September 2011 10:25
> To: Tomcat Users List
> Subject: Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
> copies of itself with only 1 thread and no CPU
>
> Intriguing.
>
> For a complement of information :
> - what is the exact platform ?
> - is Tomcat started via jsvc ?
> - as a matter of fact, *how* is the "main" Tomcat being started ?
>
> What does the command "netstat -pan | grep LISTEN" show when you have such
> multiple copies running ?
> (the form of the command will depend a bit on the platform; the command
> above, under Linux, shows the ports on which processes are listening, and
> the corresponding PID's and maybe program name)
>
> Dave Stubbs wrote:
>> We're experiencing a problem on both production and test systems that
>> has only started to occur in the last few weeks (no new version of any
>> software have been put live to anyone's knowledge) that has us quite
> stumped.
>>
>>
>> We are seeing tomcat starting up additional copies of itself, each new
>> copy is allocating a chunk of storage, it only starts 1 thread,
>> nothing gets written to any logs and no CPU is being listed as having
been
> used.
>> This keeps happening, and if it's not monitored the system memory all
>> gets taken up and we start to get SIGBUS errors.
>>
>> We've tried turning all logging on, there don't appear to be any
>> software related events at the heart of the issue, and we can't see
>> any system triggers.
>>
>> When we list the system tasks we see that the ghost process PPID is
>> the same as the PID of the still running main tomcat instance. Here is
>> list of what happened when it wasn't picked up for a while.
>>
>>
>>
>> tomcat 21115 21114 0 14:46:44 ? 193:04 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 773 21115 0 17:29:27 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 4270 21115 0 18:29:29 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 5988 21115 0 18:59:32 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 27209 21115 0 01:01:02 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 13155 21115 0 05:32:09 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 16785 21115 0 06:34:52 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 12948 21115 0 20:58:50 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 20054 21115 0 23:00:21 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 23730 21115 0 00:03:47 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
>>
>> The main process 21115 starts off fine, and is even still running with
>> no problems the next day, but we keep getting these ghost processes
>> (for want of a better description).
>>
>> We have multiple tomcat servers running on the same system, but this
>> is the only one experiencing the problem, all of the others are
> unaffected.
>> The web apps themselves are pretty standard, servlets, JSPs, web
>> services, Oracle DB connection and a JMX link to an Apache MQ server
>> for monitoring purposes. This isn't even a heavily used application.
>>
>>
>>
>> The web has come up empty for us, we can't find any information on
>> this fault at all, so would really appreciate any help you may be able
>> to give us.
>>
>>
>>
>> Regards
>>
>> Dave
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
Posted by Tobias Crefeld <tc...@cataneo.eu>.
Am Fri, 09 Sep 2011 14:34:44 +0200
schrieb André Warnier <aw...@ice-sa.com>:
> As an alternative to netstat under Solaris, there is the "lsof"
> utility.
AFAIK there is no lsof in (standard-) Solaris-10. But maybe one can
take it from independent repositories like Sunfreeware
(http://www.sunfreeware.com/ shows a hit for lsof on Solaris10/Sparc).
Actually I never found a standard tool in Solaris-10 that could list
the tcp-listeners (...and meanwhile there are only 2 Solaris-machines
left - dying next month...).
Regards,
Tobias.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
Posted by André Warnier <aw...@ice-sa.com>.
Ok, fishing in the dark really, but trying to accumulate more clues.
What about the other questions (how is Tomcat started) ?
And, looking at your original "ps" list, some vague thing :
all these ghost processes seem to
a) not be doing much (CPU time 0), even after a long time.
b) all be started around the clock hour:00, or the hour:30, give or take a couple of minutes.
Got anything particular that would run at approximately 30 minutes interval ?
(Like some monitoring process which sends a request every 30 minutes just to check ? And
if so : has this been added just before the problem started ? and what request does it send ?)
As an alternative to netstat under Solaris, there is the "lsof" utility.
It has a million options to sort through, but it certainly can tell you which process owns
which TCP socket.
The point here is to try to figure out what these ghosts are doing and which configuration
settings they use. They seem to be using the same command-line settings as the original,
but we only see a small part (there may be other switches on your version of "ps" which
list the whole command line of each process.).
You cannot have several processes opening the same TCP port in "LISTEN" mode.
If all these processes use the same config and try to open the same ports (Tomcat opens
several), all but the first one should fail, and that should normally leave a trace in the
Tomcat logs.
Else, they are not real processes and really are threads, no matter what "ps" is saying.
You could also try taking some thread dumps of the Tomcat JVM (see the Tomcat FAQ), to see
if some thread-id there match the PID's seen in your ps displays.
Dave Stubbs wrote:
> Solaris environment is....
>
> SunOS ********** 5.10 Generic_127127-11 sun4v sparc
> SUNW,SPARC-Enterprise-T5120
>
> System = SunOS
> Node = **********
> Release = 5.10
> KernelID = Generic_127127-11
> Machine = sun4v
> BusType = <unknown>
> Serial = <unknown>
> Users = <unknown>
> OEM# = 0
> Origin# = 1
> NumCPU = 64
>
> -----Original Message-----
> From: André Warnier [mailto:aw@ice-sa.com]
> Sent: 09 September 2011 10:25
> To: Tomcat Users List
> Subject: Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
> copies of itself with only 1 thread and no CPU
>
> Intriguing.
>
> For a complement of information :
> - what is the exact platform ?
> - is Tomcat started via jsvc ?
> - as a matter of fact, *how* is the "main" Tomcat being started ?
>
> What does the command "netstat -pan | grep LISTEN" show when you have such
> multiple copies running ?
> (the form of the command will depend a bit on the platform; the command
> above, under Linux, shows the ports on which processes are listening, and
> the corresponding PID's and maybe program name)
>
> Dave Stubbs wrote:
>> We're experiencing a problem on both production and test systems that
>> has only started to occur in the last few weeks (no new version of any
>> software have been put live to anyone's knowledge) that has us quite
> stumped.
>>
>>
>> We are seeing tomcat starting up additional copies of itself, each new
>> copy is allocating a chunk of storage, it only starts 1 thread,
>> nothing gets written to any logs and no CPU is being listed as having been
> used.
>> This keeps happening, and if it's not monitored the system memory all
>> gets taken up and we start to get SIGBUS errors.
>>
>> We've tried turning all logging on, there don't appear to be any
>> software related events at the heart of the issue, and we can't see
>> any system triggers.
>>
>> When we list the system tasks we see that the ghost process PPID is
>> the same as the PID of the still running main tomcat instance. Here is
>> list of what happened when it wasn't picked up for a while.
>>
>>
>>
>> tomcat 21115 21114 0 14:46:44 ? 193:04 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 773 21115 0 17:29:27 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 4270 21115 0 18:29:29 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 5988 21115 0 18:59:32 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 27209 21115 0 01:01:02 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 13155 21115 0 05:32:09 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 16785 21115 0 06:34:52 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 12948 21115 0 20:58:50 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 20054 21115 0 23:00:21 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
>> 23730 21115 0 00:03:47 ? 0:00 /usr/java/bin/java
>> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
>>
>> The main process 21115 starts off fine, and is even still running with
>> no problems the next day, but we keep getting these ghost processes
>> (for want of a better description).
>>
>> We have multiple tomcat servers running on the same system, but this
>> is the only one experiencing the problem, all of the others are
> unaffected.
>> The web apps themselves are pretty standard, servlets, JSPs, web
>> services, Oracle DB connection and a JMX link to an Apache MQ server
>> for monitoring purposes. This isn't even a heavily used application.
>>
>>
>>
>> The web has come up empty for us, we can't find any information on
>> this fault at all, so would really appreciate any help you may be able
>> to give us.
>>
>>
>>
>> Regards
>>
>> Dave
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts
additional copies of itself with only 1 thread and no CPU
Posted by Dave Stubbs <da...@stubbs.uk.com>.
Solaris environment is....
SunOS ********** 5.10 Generic_127127-11 sun4v sparc
SUNW,SPARC-Enterprise-T5120
System = SunOS
Node = **********
Release = 5.10
KernelID = Generic_127127-11
Machine = sun4v
BusType = <unknown>
Serial = <unknown>
Users = <unknown>
OEM# = 0
Origin# = 1
NumCPU = 64
-----Original Message-----
From: André Warnier [mailto:aw@ice-sa.com]
Sent: 09 September 2011 10:25
To: Tomcat Users List
Subject: Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
Intriguing.
For a complement of information :
- what is the exact platform ?
- is Tomcat started via jsvc ?
- as a matter of fact, *how* is the "main" Tomcat being started ?
What does the command "netstat -pan | grep LISTEN" show when you have such
multiple copies running ?
(the form of the command will depend a bit on the platform; the command
above, under Linux, shows the ports on which processes are listening, and
the corresponding PID's and maybe program name)
Dave Stubbs wrote:
> We're experiencing a problem on both production and test systems that
> has only started to occur in the last few weeks (no new version of any
> software have been put live to anyone's knowledge) that has us quite
stumped.
>
>
>
> We are seeing tomcat starting up additional copies of itself, each new
> copy is allocating a chunk of storage, it only starts 1 thread,
> nothing gets written to any logs and no CPU is being listed as having been
used.
>
> This keeps happening, and if it's not monitored the system memory all
> gets taken up and we start to get SIGBUS errors.
>
> We've tried turning all logging on, there don't appear to be any
> software related events at the heart of the issue, and we can't see
> any system triggers.
>
> When we list the system tasks we see that the ghost process PPID is
> the same as the PID of the still running main tomcat instance. Here is
> list of what happened when it wasn't picked up for a while.
>
>
>
> tomcat 21115 21114 0 14:46:44 ? 193:04 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
> 773 21115 0 17:29:27 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
> 4270 21115 0 18:29:29 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
> 5988 21115 0 18:59:32 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
> 27209 21115 0 01:01:02 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
> 13155 21115 0 05:32:09 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
> 16785 21115 0 06:34:52 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
> 12948 21115 0 20:58:50 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
> 20054 21115 0 23:00:21 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
> 23730 21115 0 00:03:47 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
>
> The main process 21115 starts off fine, and is even still running with
> no problems the next day, but we keep getting these ghost processes
> (for want of a better description).
>
> We have multiple tomcat servers running on the same system, but this
> is the only one experiencing the problem, all of the others are
unaffected.
>
> The web apps themselves are pretty standard, servlets, JSPs, web
> services, Oracle DB connection and a JMX link to an Apache MQ server
> for monitoring purposes. This isn't even a heavily used application.
>
>
>
> The web has come up empty for us, we can't find any information on
> this fault at all, so would really appreciate any help you may be able
> to give us.
>
>
>
> Regards
>
> Dave
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
Posted by André Warnier <aw...@ice-sa.com>.
Intriguing.
For a complement of information :
- what is the exact platform ?
- is Tomcat started via jsvc ?
- as a matter of fact, *how* is the "main" Tomcat being started ?
What does the command "netstat -pan | grep LISTEN" show when you have such multiple copies
running ?
(the form of the command will depend a bit on the platform; the command above, under
Linux, shows the ports on which processes are listening, and the corresponding PID's and
maybe program name)
Dave Stubbs wrote:
> We're experiencing a problem on both production and test systems that has
> only started to occur in the last few weeks (no new version of any software
> have been put live to anyone's knowledge) that has us quite stumped.
>
>
>
> We are seeing tomcat starting up additional copies of itself, each new copy
> is allocating a chunk of storage, it only starts 1 thread, nothing gets
> written to any logs and no CPU is being listed as having been used.
>
> This keeps happening, and if it's not monitored the system memory all gets
> taken up and we start to get SIGBUS errors.
>
> We've tried turning all logging on, there don't appear to be any software
> related events at the heart of the issue, and we can't see any system
> triggers.
>
> When we list the system tasks we see that the ghost process PPID is the same
> as the PID of the still running main tomcat instance. Here is list of what
> happened when it wasn't picked up for a while.
>
>
>
> tomcat 21115 21114 0 14:46:44 ? 193:04 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
> tomcat 773 21115 0 17:29:27 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
> tomcat 4270 21115 0 18:29:29 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
> tomcat 5988 21115 0 18:59:32 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
> tomcat 27209 21115 0 01:01:02 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
> tomcat 13155 21115 0 05:32:09 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
> tomcat 16785 21115 0 06:34:52 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
> tomcat 12948 21115 0 20:58:50 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
> tomcat 20054 21115 0 23:00:21 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
> tomcat 23730 21115 0 00:03:47 ? 0:00 /usr/java/bin/java
> -Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
>
> The main process 21115 starts off fine, and is even still running with no
> problems the next day, but we keep getting these ghost processes (for want
> of a better description).
>
> We have multiple tomcat servers running on the same system, but this is the
> only one experiencing the problem, all of the others are unaffected.
>
> The web apps themselves are pretty standard, servlets, JSPs, web services,
> Oracle DB connection and a JMX link to an Apache MQ server for monitoring
> purposes. This isn't even a heavily used application.
>
>
>
> The web has come up empty for us, we can't find any information on this
> fault at all, so would really appreciate any help you may be able to give
> us.
>
>
>
> Regards
>
> Dave
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org