You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Hunter Hillegas <li...@lastonepicked.com> on 2001/05/21 17:16:46 UTC
3.2.1 Dies
I am running a high traffic site with Tomcat 3.2.1/Sun JVM 1.3/Linux
2.4.4/Apache 1.3.19. After a few days of serving up the hits very well,
Tomcat just dies. It won't respond to shutdown requests and stops serving
pages, giving out only null pointer exceptions. Killing the java processes
manually and restarting Tomcat clears it up.
Anyone else seen this? Any idea how to work around it?
Hunter
Re: 3.2.1 Dies
Posted by Srinadh Karumuri <sk...@bbn.com>.
STDOUT & STDERR messages go to nohup.log. In your case you might be loosing
those messages as you logged off (they will be sent to /dev/null - trash).
Sri
At 10:54 AM 05/21/2001 -0700, Hunter Hillegas wrote:
>What does running with nohup do for you?
>
>I usually start Tomcat using tomcat.sh start and then just log out...
>
>Hunter
>
>> From: Srinadh Karumuri <sk...@bbn.com>
>> Date: Mon, 21 May 2001 13:43:53 -0400
>> To: Hunter Hillegas <li...@lastonepicked.com>, Tomcat User List
>> <to...@jakarta.apache.org>
>> Subject: Re: 3.2.1 Dies
>>
>> - I am running the tomcat using 'nohup' (UNIX command) on Solaris. My
>> OutOfMemory errors were logged in nohup.log
>
________________________________________________________
Srinadh Karumuri
Senior Programmer/Analyst
Business Apps.
BBN Technologies (Verizon)
Ph:(617)873-2841
Load balancing
Posted by Antonio Parolini <to...@employer.com.br>.
Hi there,
Where can I find info on loadbalancing tomcat/apache ? we need
"statefull" load balancing (we have user session). Is this possible without
posting the user ID in links and forms ?
Tony.
RE: 3.2.1 Dies
Posted by Devon Ziegler <de...@usa.net>.
I can't say I know FOR SURE that the threads were dead, but after a
considerable length of time they never accumulated any more CPU usage
time (as seen via top or ps) whereas some of the others did. I know for
sure I had far more threads than the max_threads setting (12 for me)
because I'd see in excess of 80 threads for tomcat on the system after
several days uptime. I might try upping the max_threads to see if that
helps. We were trying to limit concurrent usage of the box (via tomcat)
because other apps use the same box and we didn't want to use too many
resources. That said, we'd rather have a stable tomcat and overload the
box a bit on occasion :) than an unstable/really slow tomcat. -Devon
----------------
Devon C. Ziegler
devonz@usa.net
2504 Preston Lake Dr.
Tucker, GA 30084
770-908-9462
-----Original Message-----
From: Thom Park [mailto:tpark@borland.com]
Sent: Monday, May 21, 2001 7:25 PM
To: tomcat-user@jakarta.apache.org
Subject: Re: 3.2.1 Dies
I wasn't able to check for dead threads - My symptom was that the tomcat
process
would 'hang' and no longer process connections.
I never got it to crash per se, but it sure stopped on me real good...
Increasing the pool size resolved the problem, specifically the
following
parameters (neatly copied from the tomcat users guide)
<Parameter
name="max_threads"
value="30"/>
<Parameter
name="max_spare_threads"
value="20"/>
<Parameter
name="min_spare_threads"
value="5" />
The values were arbitrary - I stuck mine up to max_threads = 200,
max_spare_threads=20, min_spare_threads=20
and lo - the hang problem went away.
It seems to me that when tomcat (3.2.1) get's pressed for thread
resources, it
doesn't do a very good job of recovering from this.
I would have thought that having a min_spare_threads set to some
non-zero value,
that there would still be some spare threads for clean up
work to be performed but, as far as I can tell, when you max out your
threads,
your stuck pending a reboot.
-Thom
Hunter Hillegas wrote:
> How can I check to see if I have a lot of "dead threads"?
>
> Hunter
>
> > From: "Devon Ziegler" <de...@channelogics.com>
> > Reply-To: tomcat-user@jakarta.apache.org
> > Date: Mon, 21 May 2001 16:43:04 -0400
> > To: <to...@jakarta.apache.org>
> > Subject: RE: 3.2.1 Dies
> >
> > Interesting.....used up all free threads.....increase thread
pool.....
> > shouldn't users just have to wait for a free thread if they hit the
pool
> > limit? Increasing it should let more requests be handled
simultaneously,
> > but how would it help longevity? I'm not saying it won't. Just
skeptical.
> >
> > That said, I seem to "lose" threads with tomcat 3.2.1 on Linux
2.2.14 using
> > Sun's 1.3 JDK. What I mean is that "dead" threads pile up WELL in
excess of
> > the thread pool limit. These threads don't seem to handle requests
any
> > more. Tomcat slows down quite noticeably too. I'm not sure I have
ever
> > left it up long enough to know if it might just die after enough
threads
> > "pile" up(well, it definitely would when it hit the OS' thread
limit). Are
> > you seeing far more threads than you would expect (well in excess of
the
> > pool size)? Maybe we are suffering from the same malady.
--
http://www.borland.com/newsgroups
http://www.borland.com/devsupport/disclaim.html
Re: 3.2.1 Dies
Posted by Hunter Hillegas <li...@lastonepicked.com>.
Yeah, I'm still having problems in this area.
Is 3.2.2 improved? I've seen a few messages about thread dead-lock problems
lately on 3.2.2...
I'm now logging the output of the tomcat.sh script so I should see if it is
OutOfMemory or whatever when everything dies...
Hunter
> From: "Bill Graham" <bi...@spotlife.com>
> Reply-To: tomcat-user@jakarta.apache.org
> Date: Wed, 23 May 2001 09:14:09 -0700
> To: <to...@jakarta.apache.org>
> Subject: RE: 3.2.1 Dies
>
> Someone about a month ago mentioned that this was a bug in Tomcat
> where it chokes when the max threads is exceeded. I've had the same
> problem with load testing that I've done and increasing the max threads
> worked for the number of threads I had. Unfortunately, this fix in not
> scaleable...
>
> bill
RE: 3.2.1 Dies
Posted by Bill Graham <bi...@spotlife.com>.
Someone about a month ago mentioned that this was a bug in Tomcat
where it chokes when the max threads is exceeded. I've had the same
problem with load testing that I've done and increasing the max threads
worked for the number of threads I had. Unfortunately, this fix in not
scaleable...
bill
> -----Original Message-----
> From: Thom Park [mailto:tpark@borland.com]
> Sent: Monday, May 21, 2001 4:25 PM
> To: tomcat-user@jakarta.apache.org
> Subject: Re: 3.2.1 Dies
>
>
> I wasn't able to check for dead threads - My symptom was that
> the tomcat process
> would 'hang' and no longer process connections.
>
> I never got it to crash per se, but it sure stopped on me real good...
>
> Increasing the pool size resolved the problem, specifically
> the following
> parameters (neatly copied from the tomcat users guide)
>
> <Parameter
> name="max_threads"
> value="30"/>
> <Parameter
> name="max_spare_threads"
> value="20"/>
> <Parameter
> name="min_spare_threads"
> value="5" />
>
> The values were arbitrary - I stuck mine up to max_threads = 200,
> max_spare_threads=20, min_spare_threads=20
> and lo - the hang problem went away.
>
> It seems to me that when tomcat (3.2.1) get's pressed for
> thread resources, it
> doesn't do a very good job of recovering from this.
> I would have thought that having a min_spare_threads set to
> some non-zero value,
> that there would still be some spare threads for clean up
> work to be performed but, as far as I can tell, when you max
> out your threads,
> your stuck pending a reboot.
>
> -Thom
>
>
>
> Hunter Hillegas wrote:
>
> > How can I check to see if I have a lot of "dead threads"?
> >
> > Hunter
> >
> > > From: "Devon Ziegler" <de...@channelogics.com>
> > > Reply-To: tomcat-user@jakarta.apache.org
> > > Date: Mon, 21 May 2001 16:43:04 -0400
> > > To: <to...@jakarta.apache.org>
> > > Subject: RE: 3.2.1 Dies
> > >
> > > Interesting.....used up all free threads.....increase
> thread pool.....
> > > shouldn't users just have to wait for a free thread if
> they hit the pool
> > > limit? Increasing it should let more requests be handled
> simultaneously,
> > > but how would it help longevity? I'm not saying it
> won't. Just skeptical.
> > >
> > > That said, I seem to "lose" threads with tomcat 3.2.1 on
> Linux 2.2.14 using
> > > Sun's 1.3 JDK. What I mean is that "dead" threads pile
> up WELL in excess of
> > > the thread pool limit. These threads don't seem to
> handle requests any
> > > more. Tomcat slows down quite noticeably too. I'm not
> sure I have ever
> > > left it up long enough to know if it might just die after
> enough threads
> > > "pile" up(well, it definitely would when it hit the OS'
> thread limit). Are
> > > you seeing far more threads than you would expect (well
> in excess of the
> > > pool size)? Maybe we are suffering from the same malady.
>
> --
> http://www.borland.com/newsgroups
> http://www.borland.com/devsupport/disclaim.html
>
Re: 3.2.1 Dies
Posted by Thom Park <tp...@borland.com>.
I wasn't able to check for dead threads - My symptom was that the tomcat process
would 'hang' and no longer process connections.
I never got it to crash per se, but it sure stopped on me real good...
Increasing the pool size resolved the problem, specifically the following
parameters (neatly copied from the tomcat users guide)
<Parameter
name="max_threads"
value="30"/>
<Parameter
name="max_spare_threads"
value="20"/>
<Parameter
name="min_spare_threads"
value="5" />
The values were arbitrary - I stuck mine up to max_threads = 200,
max_spare_threads=20, min_spare_threads=20
and lo - the hang problem went away.
It seems to me that when tomcat (3.2.1) get's pressed for thread resources, it
doesn't do a very good job of recovering from this.
I would have thought that having a min_spare_threads set to some non-zero value,
that there would still be some spare threads for clean up
work to be performed but, as far as I can tell, when you max out your threads,
your stuck pending a reboot.
-Thom
Hunter Hillegas wrote:
> How can I check to see if I have a lot of "dead threads"?
>
> Hunter
>
> > From: "Devon Ziegler" <de...@channelogics.com>
> > Reply-To: tomcat-user@jakarta.apache.org
> > Date: Mon, 21 May 2001 16:43:04 -0400
> > To: <to...@jakarta.apache.org>
> > Subject: RE: 3.2.1 Dies
> >
> > Interesting.....used up all free threads.....increase thread pool.....
> > shouldn't users just have to wait for a free thread if they hit the pool
> > limit? Increasing it should let more requests be handled simultaneously,
> > but how would it help longevity? I'm not saying it won't. Just skeptical.
> >
> > That said, I seem to "lose" threads with tomcat 3.2.1 on Linux 2.2.14 using
> > Sun's 1.3 JDK. What I mean is that "dead" threads pile up WELL in excess of
> > the thread pool limit. These threads don't seem to handle requests any
> > more. Tomcat slows down quite noticeably too. I'm not sure I have ever
> > left it up long enough to know if it might just die after enough threads
> > "pile" up(well, it definitely would when it hit the OS' thread limit). Are
> > you seeing far more threads than you would expect (well in excess of the
> > pool size)? Maybe we are suffering from the same malady.
--
http://www.borland.com/newsgroups
http://www.borland.com/devsupport/disclaim.html
Re: 3.2.1 Dies
Posted by Hunter Hillegas <li...@lastonepicked.com>.
How can I check to see if I have a lot of "dead threads"?
Hunter
> From: "Devon Ziegler" <de...@channelogics.com>
> Reply-To: tomcat-user@jakarta.apache.org
> Date: Mon, 21 May 2001 16:43:04 -0400
> To: <to...@jakarta.apache.org>
> Subject: RE: 3.2.1 Dies
>
> Interesting.....used up all free threads.....increase thread pool.....
> shouldn't users just have to wait for a free thread if they hit the pool
> limit? Increasing it should let more requests be handled simultaneously,
> but how would it help longevity? I'm not saying it won't. Just skeptical.
>
> That said, I seem to "lose" threads with tomcat 3.2.1 on Linux 2.2.14 using
> Sun's 1.3 JDK. What I mean is that "dead" threads pile up WELL in excess of
> the thread pool limit. These threads don't seem to handle requests any
> more. Tomcat slows down quite noticeably too. I'm not sure I have ever
> left it up long enough to know if it might just die after enough threads
> "pile" up(well, it definitely would when it hit the OS' thread limit). Are
> you seeing far more threads than you would expect (well in excess of the
> pool size)? Maybe we are suffering from the same malady.
RE: 3.2.1 Dies
Posted by Devon Ziegler <de...@channelogics.com>.
Interesting.....used up all free threads.....increase thread pool.....
shouldn't users just have to wait for a free thread if they hit the pool
limit? Increasing it should let more requests be handled simultaneously,
but how would it help longevity? I'm not saying it won't. Just skeptical.
That said, I seem to "lose" threads with tomcat 3.2.1 on Linux 2.2.14 using
Sun's 1.3 JDK. What I mean is that "dead" threads pile up WELL in excess of
the thread pool limit. These threads don't seem to handle requests any
more. Tomcat slows down quite noticeably too. I'm not sure I have ever
left it up long enough to know if it might just die after enough threads
"pile" up(well, it definitely would when it hit the OS' thread limit). Are
you seeing far more threads than you would expect (well in excess of the
pool size)? Maybe we are suffering from the same malady.
-----Original Message-----
From: Thom Park [mailto:tpark@borland.com]
Sent: Monday, May 21, 2001 4:11 PM
To: tomcat-user@jakarta.apache.org
Subject: Re: 3.2.1 Dies
You may have used up all of your free threads.
Try increasing the thread pool for the Http connector - see if it improves
the
longevity of your
tomcat instance.
-Thom
Hunter Hillegas wrote:
> Well, I am using the scripts and my Tomcat just dies after a few days of
> heavy load...
>
> We're talking hundreds of thousands of hits...
>
> Any other ideas are appreciated. I'm tempted to try 3.2.2 but since this
is
> a production site, I'm a bit scared off by the beta status of the
software.
>
> Hunter
>
> > From: "Tim O'Neil" <ti...@xythos.com>
> > Reply-To: tomcat-user@jakarta.apache.org
> > Date: Mon, 21 May 2001 12:18:52 -0700
> > To: tomcat-user@jakarta.apache.org
> > Subject: Re: 3.2.1 Dies
> >
> >> What does running with nohup do for you?
> >>
> >> I usually start Tomcat using tomcat.sh start and then just log out...
> >
> > Me too. Nohup (no hangup) runs the command in an "ingnore
> > hangup" signal mode- not the same as a process fork. I think
> > the cmd runs as an orphaned process. If he runs it as "nohup
> > cmd &" that's pretty much what the tomcat startup script
> > does, minus the process env setup. Really he should use the
> > scripts. I'm guessing he's not using the "&" at the end of
> > the invocation, the os sees the process is orphaned after
> > a while, and kills it.
--
http://www.borland.com/newsgroups
http://www.borland.com/devsupport/disclaim.html
Re: 3.2.1 Dies
Posted by Hunter Hillegas <li...@lastonepicked.com>.
I'll try that.
I've not played with those settings. Any suggestions as to a sane amount to
set it to?
Others out there?
Hunter
> From: Thom Park <tp...@borland.com>
> Organization: Borland Software Company
> Reply-To: tomcat-user@jakarta.apache.org
> Date: Mon, 21 May 2001 13:10:54 -0700
> To: tomcat-user@jakarta.apache.org
> Subject: Re: 3.2.1 Dies
>
> You may have used up all of your free threads.
>
> Try increasing the thread pool for the Http connector - see if it improves the
> longevity of your
> tomcat instance.
>
> -Thom
Re: 3.2.1 Dies
Posted by Thom Park <tp...@borland.com>.
You may have used up all of your free threads.
Try increasing the thread pool for the Http connector - see if it improves the
longevity of your
tomcat instance.
-Thom
Hunter Hillegas wrote:
> Well, I am using the scripts and my Tomcat just dies after a few days of
> heavy load...
>
> We're talking hundreds of thousands of hits...
>
> Any other ideas are appreciated. I'm tempted to try 3.2.2 but since this is
> a production site, I'm a bit scared off by the beta status of the software.
>
> Hunter
>
> > From: "Tim O'Neil" <ti...@xythos.com>
> > Reply-To: tomcat-user@jakarta.apache.org
> > Date: Mon, 21 May 2001 12:18:52 -0700
> > To: tomcat-user@jakarta.apache.org
> > Subject: Re: 3.2.1 Dies
> >
> >> What does running with nohup do for you?
> >>
> >> I usually start Tomcat using tomcat.sh start and then just log out...
> >
> > Me too. Nohup (no hangup) runs the command in an "ingnore
> > hangup" signal mode- not the same as a process fork. I think
> > the cmd runs as an orphaned process. If he runs it as "nohup
> > cmd &" that's pretty much what the tomcat startup script
> > does, minus the process env setup. Really he should use the
> > scripts. I'm guessing he's not using the "&" at the end of
> > the invocation, the os sees the process is orphaned after
> > a while, and kills it.
--
http://www.borland.com/newsgroups
http://www.borland.com/devsupport/disclaim.html
Re: 3.2.1 Dies
Posted by Hunter Hillegas <li...@lastonepicked.com>.
Well, I am using the scripts and my Tomcat just dies after a few days of
heavy load...
We're talking hundreds of thousands of hits...
Any other ideas are appreciated. I'm tempted to try 3.2.2 but since this is
a production site, I'm a bit scared off by the beta status of the software.
Hunter
> From: "Tim O'Neil" <ti...@xythos.com>
> Reply-To: tomcat-user@jakarta.apache.org
> Date: Mon, 21 May 2001 12:18:52 -0700
> To: tomcat-user@jakarta.apache.org
> Subject: Re: 3.2.1 Dies
>
>> What does running with nohup do for you?
>>
>> I usually start Tomcat using tomcat.sh start and then just log out...
>
> Me too. Nohup (no hangup) runs the command in an "ingnore
> hangup" signal mode- not the same as a process fork. I think
> the cmd runs as an orphaned process. If he runs it as "nohup
> cmd &" that's pretty much what the tomcat startup script
> does, minus the process env setup. Really he should use the
> scripts. I'm guessing he's not using the "&" at the end of
> the invocation, the os sees the process is orphaned after
> a while, and kills it.
Re: 3.2.1 Dies
Posted by Tim O'Neil <ti...@xythos.com>.
At 10:54 AM 5/21/2001 -0700, you wrote:
>What does running with nohup do for you?
>
>I usually start Tomcat using tomcat.sh start and then just log out...
Me too. Nohup (no hangup) runs the command in an "ingnore
hangup" signal mode- not the same as a process fork. I think
the cmd runs as an orphaned process. If he runs it as "nohup
cmd &" that's pretty much what the tomcat startup script
does, minus the process env setup. Really he should use the
scripts. I'm guessing he's not using the "&" at the end of
the invocation, the os sees the process is orphaned after
a while, and kills it.
Re: 3.2.1 Dies
Posted by Hunter Hillegas <li...@lastonepicked.com>.
What does running with nohup do for you?
I usually start Tomcat using tomcat.sh start and then just log out...
Hunter
> From: Srinadh Karumuri <sk...@bbn.com>
> Date: Mon, 21 May 2001 13:43:53 -0400
> To: Hunter Hillegas <li...@lastonepicked.com>, Tomcat User List
> <to...@jakarta.apache.org>
> Subject: Re: 3.2.1 Dies
>
> - I am running the tomcat using 'nohup' (UNIX command) on Solaris. My
> OutOfMemory errors were logged in nohup.log
Re: 3.2.1 Dies
Posted by Srinadh Karumuri <sk...@bbn.com>.
- I am running the tomcat using 'nohup' (UNIX command) on Solaris. My
OutOfMemory errors were logged in nohup.log
- Also, I remember seeing that NullPointer error when the old Tomcat didn't
shutdown properly. I manually killed the old hanging java process using
'kill -9' and the new one is all set (no errors).
Hope this helps,
-Sri
At 10:20 AM 05/21/2001 -0700, Hunter Hillegas wrote:
>Which log file would I look in?
>
>servlet.log shows nothing, neither does jasper.log...
>
>Hunter
>
>> From: Srinadh Karumuri <sk...@bbn.com>
>> Reply-To: tomcat-user@jakarta.apache.org
>> Date: Mon, 21 May 2001 12:03:45 -0400
>> To: tomcat-user@jakarta.apache.org, Tomcat User List
>> <to...@jakarta.apache.org>
>> Subject: Re: 3.2.1 Dies
>>
>> Did you check the log files?
>> This sounds like a OutOfMemoryError.
>>
>> You can increase the heap size while starting the tomcat.
>>
>> -Sri
>>
>> At 08:16 AM 05/21/2001 -0700, Hunter Hillegas wrote:
>>> I am running a high traffic site with Tomcat 3.2.1/Sun JVM 1.3/Linux
>>> 2.4.4/Apache 1.3.19. After a few days of serving up the hits very well,
>>> Tomcat just dies. It won't respond to shutdown requests and stops serving
>>> pages, giving out only null pointer exceptions. Killing the java processes
>>> manually and restarting Tomcat clears it up.
>>>
>>> Anyone else seen this? Any idea how to work around it?
>>>
>>> Hunter
>
________________________________________________________
Srinadh Karumuri
Senior Programmer/Analyst
Business Apps.
BBN Technologies (Verizon)
Ph:(617)873-2841
Re: 3.2.1 Dies
Posted by Hunter Hillegas <li...@lastonepicked.com>.
Which log file would I look in?
servlet.log shows nothing, neither does jasper.log...
Hunter
> From: Srinadh Karumuri <sk...@bbn.com>
> Reply-To: tomcat-user@jakarta.apache.org
> Date: Mon, 21 May 2001 12:03:45 -0400
> To: tomcat-user@jakarta.apache.org, Tomcat User List
> <to...@jakarta.apache.org>
> Subject: Re: 3.2.1 Dies
>
> Did you check the log files?
> This sounds like a OutOfMemoryError.
>
> You can increase the heap size while starting the tomcat.
>
> -Sri
>
> At 08:16 AM 05/21/2001 -0700, Hunter Hillegas wrote:
>> I am running a high traffic site with Tomcat 3.2.1/Sun JVM 1.3/Linux
>> 2.4.4/Apache 1.3.19. After a few days of serving up the hits very well,
>> Tomcat just dies. It won't respond to shutdown requests and stops serving
>> pages, giving out only null pointer exceptions. Killing the java processes
>> manually and restarting Tomcat clears it up.
>>
>> Anyone else seen this? Any idea how to work around it?
>>
>> Hunter
Re: 3.2.1 Dies
Posted by Srinadh Karumuri <sk...@bbn.com>.
Did you check the log files?
This sounds like a OutOfMemoryError.
You can increase the heap size while starting the tomcat.
-Sri
At 08:16 AM 05/21/2001 -0700, Hunter Hillegas wrote:
>I am running a high traffic site with Tomcat 3.2.1/Sun JVM 1.3/Linux
>2.4.4/Apache 1.3.19. After a few days of serving up the hits very well,
>Tomcat just dies. It won't respond to shutdown requests and stops serving
>pages, giving out only null pointer exceptions. Killing the java processes
>manually and restarting Tomcat clears it up.
>
>Anyone else seen this? Any idea how to work around it?
>
>Hunter
>
>
________________________________________________________
Srinadh Karumuri
Senior Programmer/Analyst
Business Apps.
BBN Technologies (Verizon)
Ph:(617)873-2841