You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Sean Carnes <sc...@gmail.com> on 2007/12/06 16:30:32 UTC
Tomcat 4.1.31 crashing with memory errors, crashing with no errors and shutting down cleanly without manual intervention
*We are having an issue with the tomcat service crashing version 4.1.31,
sometimes with these memory errors and sometimes not. We have a backup but
once the load moves to that server the backup crashes also almost
immediately after the load balancer switches. This has happened multiple
times to us over the past few days with no changes made to the server, just
a slight increase in load. We have no option to upgrade right now since
this is included in an enterprise application. We have the heap space set
to the maximum that it will allow us to. The vendor recommended 1536 but
this setting caused tomcat to not start at all, ~1200 was the highest I was
able to get it to go. I watched it crash yesterday and it did a clean
shutdown with no errors which was differrent. The vendor doesn't have a fix
yet and I wanted to see what you all think. There are other large logs, if
there is something specific that we would need to look at then let me know
and I will try and get it out.
Thanks,
Sean
Dec 5, 2007 4:11:05 PM - Error creating event for domain ss-ls01p:
java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.<init>(ArrayList.java:113)
at java.util.ArrayList.<init>(ArrayList.java:120)
at
com.aprisma.spectrum.app.event.web.model.BackEndEventDataModel.makeEventObject
(BackEndEventDataModel.java:566)
at
com.aprisma.spectrum.app.event.web.model.BackEndEventDataModel.applyEventUpdate
(BackEndEventDataModel.java:748)
at
com.aprisma.spectrum.app.event.web.model.BackEndEventDataModel$QueuedEventUpdate.run
(BackEndEventDataModel.java:116)
at com.aprisma.util.thread.JobQueue.runJobThread(JobQueue.java:232)
at com.aprisma.util.thread.JobQueue.access$000(JobQueue.java:38)
at com.aprisma.util.thread.JobQueue$JobRunnable.run(JobQueue.java:47)
at java.lang.Thread.run(Thread.java:595)
java.lang.OutOfMemoryError: Java heap space
at com.aprisma.spectrum.core.idl.CsCAttribute.CsCAttrValueHelper.read(
CsCAttrValueHelper.java:53)
at com.aprisma.spectrum.core.idl.CsCAttribute.CsCAttrValListHelper.read(
CsCAttrValListHelper.java:56)
at
com.aprisma.spectrum.core.idl.CsCEventDomainPackage.CsCEventHelper.read(
CsCEventHelper.java:64)
at
com.aprisma.spectrum.core.idl.CsCEventDomainPackage.CsCEventListHelper.read(
CsCEventListHelper.java:63)
at com.aprisma.spectrum.core.idl.CsCEventWatchCBPOA._invoke(
CsCEventWatchCBPOA.java:66)
at com.aprisma.spectrum.core.idl.CsCEventWatchCBPOA._invoke(
CsCEventWatchCBPOA.java:51)
at com.inprise.vbroker.poa.POAImpl.invoke(POAImpl.java:2822)
at com.inprise.vbroker.poa.ActivationRecord.invoke(ActivationRecord.java
:186)
at com.inprise.vbroker.GIOP.GiopProtocolAdapter.doRequest(
GiopProtocolAdapter.java:838)
at com.inprise.vbroker.GIOP.GiopProtocolAdapter.dispatchMessage(
GiopProtocolAdapter.java:1120)
at com.inprise.vbroker.orb.TPDispatcherImpl$TPDispatcher.run(
TPDispatcherImpl.java:100)
at com.inprise.vbroker.orb.ThreadPool$PoolWorker.run(ThreadPool.java:76)
java.lang.OutOfMemoryError: Java heap space
Dec 5, 2007 4:12:21 PM - EventFormatHelper: Error parsing event table enum
file nmiEventType: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "HostConfig[localhost]" java.lang.OutOfMemoryError: Java
heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "CORBAMonitorPool-243: IDLE" java.lang.OutOfMemoryError:
Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "VBJ ThreadPool Worker" java.lang.OutOfMemoryError: Java
heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "VBJ ThreadPool Worker" java.lang.OutOfMemoryError: Java
heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "StandardManager[/examples]" java.lang.OutOfMemoryError:
Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "CORBAMonitorPool-247: IDLE" Exception in thread "VBJ
ThreadPool Worker" java.lang.NullPointerException
at org.apache.coyote.tomcat4.OutputBuffer.realWriteChars(
OutputBuffer.java:563)
at org.apache.tomcat.util.buf.CharChunk.flushBuffer(CharChunk.java:435)
at org.apache.coyote.tomcat4.OutputBuffer.close(OutputBuffer.java:268)
at org.apache.coyote.tomcat4.CoyoteResponse.finishResponse(
CoyoteResponse.java:438)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java
:153)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection
(Http11Protocol.java:705)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java
:577)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:595)
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.NullPointerException
at org.apache.coyote.tomcat4.OutputBuffer.realWriteChars(
OutputBuffer.java:563)
at org.apache.tomcat.util.buf.CharChunk.flushBuffer(CharChunk.java:435)
at org.apache.coyote.tomcat4.OutputBuffer.close(OutputBuffer.java:268)
at org.apache.coyote.tomcat4.CoyoteResponse.finishResponse(
CoyoteResponse.java:438)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java
:153)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection
(Http11Protocol.java:705)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java
:577)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:595)
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
- Error unregistering mbean
java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-80-Processor519" java.lang.OutOfMemoryError: Java
heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Dec 5, 2007 4:14:19 PM (Thread-8) (MDL) - Administrator: Callback
notifications were missed from landscape ss-ls02p - resynching cached
alarms.
Dec 5, 2007 4:14:24 PM (Thread-8) (MDL) - Administrator: Callback
notifications were missed from landscape ss-ls08p - resynching cached
alarms.
Exception in thread "StandardManager[/customimages]"
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Dec 5, 2007 4:15:07 PM (Thread-8) (MDL) - Administrator: Callback
notifications were missed from landscape ss-ls06p - resynching cached
alarms.
Dec 5, 2007 4:15:18 PM - AlarmUpdates: Error invoking job (
com.aprisma.spectrum.app.alarm.web.model.BackEndAlarmDataModel$QueuedAlarmUpdate@afaf8f)
: java.lang.OutOfMemoryError: Java heap space
Exception in thread "CORBAMonitorPool-218: IDLE" java.lang.OutOfMemoryError:
Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-80-Processor304" java.lang.OutOfMemoryError: Java
heap space
java.lang.OutOfMemoryError: Java heap space
- Error unregistering mbean
java.lang.OutOfMemoryError: Java heap space
Exception in thread "CORBAMonitorPool-253: IDLE" Exception in thread
"StandardManager[/admin]" Exception in thread "CORBAMonitorPool-252: IDLE"
java.lang.OutOfMemoryError: Java heap space
Dec 5, 2007 4:17:05 PM (Thread-8) (MDL) - Administrator: Callback
notifications were missed from landscape ss-ls01p - resynching cached
alarms.
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "DatagramThread" java.lang.OutOfMemoryError: Java heap
space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Dec 5, 2007 4:18:17 PM - Error creating event for domain ss-ls01p:
java.lang.OutOfMemoryError: Java heap space
Dec 5, 2007 4:18:29 PM - EventUpdates@GlobalEventBackEnd: Error invoking job
(
com.aprisma.spectrum.app.event.web.model.BackEndEventDataModel$QueuedEventUpdate@1e0a91c)
: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Dec 5, 2007 4:18:37 PM - The Location Server on ss-ls01p is not available.
Dec 5, 2007 4:18:37 PM - Could not reach location server at ss-ls01p
Dec 5, 2007 4:18:37 PM (Thread-8) (MDL) - Administrator: Callback
notifications were missed from landscape ss-ls01p - resynching cached
alarms.
Dec 5, 2007 4:18:37 PM - Error restarting association watch on domain:
ss-ls03p for models:
{0x4,0x12d23,0xc487,0x2ee89,0x2b67b,0x3a90a,0x3655,0x3c677,0x2f866,0x338bd,0x11b2a,0x3c4ca,0x3a0a4,0x33742,0x5085,0x8f92,0x199bb,0x5385,0xb088,0x2dee5,0x199ba,0x117d5,0x3b48,0x3a07a,0x5496,0x13324,0x3a918,0x117c8,0x4a7c,0x33613,0x2d92b,0x3c675,0x2f9e9,0x3a0a2,0x199bc,0x12d72,0x33740,0x33739,0x0,0x4d87,0x37f4,0xc33a,0xc486,0x12f8f,0x3a909,0x3c673,0x341a6,0xc377,0x1e66e,0x38ffb,0x2c4b6,0xc4c2,0x3a07e,0x3378a,0x3373e,0x39734,0x1da1f,0x2f48f,0x8f93,0x2fa6c,0x4f26,0x3c019,0x3915e,0x5735,0x3c679,0x34999,0xcfd1,0x3373c,0x3014b,0xcfd2,0x3a0a6}
with error: {0}java.lang.OutOfMemoryError: Java heap space
Dec 5, 2007 4:18:57 PM (Thread-8) (MDL) - Administrator: Callback
notifications were missed from landscape ss-ls02p - resynching cached
alarms.
Dec 5, 2007 4:20:08 PM (Thread-8) (MDL) - Administrator: Callback
notifications were missed from landscape ss-ls01p - resynching cached
alarms.
Dec 5, 2007 4:20:49 PM (Thread-8) (MDL) - Administrator: Callback
notifications were missed from landscape ss-ls02p - resynching cached
alarms.
Stopping service Tomcat-Standalone
- Stopping Coyote HTTP/1.1 on http-80
Dec 5, 2007 4:21:58 PM (Thread-8) (MDL) - Administrator: Callback
notifications were missed from landscape ss-ls08p - resynching cached
alarms.
*
--
"Any sufficiently advanced technology is indistinguishable from magic."
- Arthur C. Clarke
Re: Tomcat 4.1.31 crashing with memory errors, crashing with no errors and shutting down cleanly without manual intervention
Posted by Andrew Miehs <an...@2sheds.de>.
On 06/12/2007, at 10:34 PM, Sean Carnes wrote:
> The highest that we could set the heap was to 1200. I tried higher
> and it
> would not start. It also seemed somewhat unstable above 1024 which
> was the
> previous setting, slowness updating the client and other things. The
> company that develops the enterprise s/w that uses tomcat said that
> settings
> over 1024 were unstable so my feeling was confirmed by them. We use
> an snmp
> agent to our nms to get system statistics. There was nothing out of
> the
> ordinary, other than tomcat using about 1298M which is the most that
> we have
> seen it use. We have everything up and running now and we are load
> balancing which is how it should have been set up in the first
> place. The
> memory usage of tomcat has dropped ~40% since we made that change.
> It was
> normally using between 600M & 800M now its down to about 4-500M give
> or
> take.
Hi Sean
It seems as if it sort of works at the moment by the sounds of this...
Things you can try when you are board and have time:
- Does Windows JVM 1.42 have jstat ?
- Try upgrading to JVM 1.5 - (linux if not available on windows)
- run jstat every minute and you should be able to get a good look
at
users vs. memory to see if this is really the problem.
And definitely - upgrade to the 64 bit JVM as soon as possible - RAM
is cheap
Andrew
RE: Tomcat 4.1.31 crashing with memory errors, crashing with no
errors and shutting down cleanly without manual intervention
Posted by Peter Crowther <Pe...@melandra.com>.
> From: Sean Carnes [mailto:scarnes@gmail.com]
> The highest that we could set the heap was to 1200.
That feels a little low, even on Windows. I wonder what's fragmenting the address space. I can get to about 1500 on x86 Windows 2003 Server Standard before the VM fails to start.
> We have everything up and running now and we are load
> balancing which is how it should have been set up in the
> first place. The
> memory usage of tomcat has dropped ~40% since we made that
> change. It was
> normally using between 600M & 800M now its down to about
> 4-500M give or take.
That's good news. Hope some of the worry is now fading!
- Peter
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 4.1.31 crashing with memory errors, crashing with no errors and shutting down cleanly without manual intervention
Posted by Sean Carnes <sc...@gmail.com>.
Andrew,
Believe me I wish this was on Linux soooo many things would be so much
easier for someone who is used to unix, but unfortunately we are on windows
server 2k3 the decision was made before I got here. We are moving to either
linux or Solaris in the future but that will take a while.
The highest that we could set the heap was to 1200. I tried higher and it
would not start. It also seemed somewhat unstable above 1024 which was the
previous setting, slowness updating the client and other things. The
company that develops the enterprise s/w that uses tomcat said that settings
over 1024 were unstable so my feeling was confirmed by them. We use an snmp
agent to our nms to get system statistics. There was nothing out of the
ordinary, other than tomcat using about 1298M which is the most that we have
seen it use. We have everything up and running now and we are load
balancing which is how it should have been set up in the first place. The
memory usage of tomcat has dropped ~40% since we made that change. It was
normally using between 600M & 800M now its down to about 4-500M give or
take.
I can't run ps on Windows unfortunately. As far as the JVM settings are
concerned I will have to see where they are set. Since this is not just
tomcat things are not all where they normally are but I will check. As far
as it dumping core we have 2 .hprof files in our directory which is what the
vendor had us send in so that they can look through them each one of those
came when it had full thread dump in the log files. The other time we saw
no .hprof but the message in the stdout.log that says tomcat is shutting
down.
Regards,
Sean
On Dec 6, 2007 12:42 PM, Andrew Miehs <an...@2sheds.de> wrote:
> Do you also have performance data for the front end machines?
>
> What OS are you running?
>
> Would definitely recommending installing sar (or sysstat package) if
> you are running linux.
> If Linux, which kernel?
>
> If it really is heap, have a look at:
>
> http://hausheer.osola.com/docs/5 for a simple description on how to
> fix this...
>
> Below is the google link which shows how this was found....
> http://www.google.com/search?rls=en-us&q=java.lang.OutOfMemoryError:+Java+heap+space&ie=UTF-8&oe=UTF-8
>
>
> > The thing I am wondering also is why it shuts down cleanly
> > sometimes. Is
> > there something internally that triggers that with this version of
> > Tomcat,
> > would an out of heap memory situation cause that. Maybe its in one
> > of the
> > logs but I am still going through all of them. There are multiple
> > megs of
> > them.
>
>
>
> Are you sure it shuts down cleanly - this seems strange - you may want
> to chnage `ulimit -c 1000` so that you can see whether it really core
> dumps or not....
>
> what does `ps auxH` report on your machine?
>
> How much memory is the thing using?
>
> How high have you configured your memory settings for the JVM?
>
>
>
> Cheers
>
> Andrew
--
"Any sufficiently advanced technology is indistinguishable from magic."
- Arthur C. Clarke
Re: Tomcat 4.1.31 crashing with memory errors, crashing with no errors and shutting down cleanly without manual intervention
Posted by Andrew Miehs <an...@2sheds.de>.
Do you also have performance data for the front end machines?
What OS are you running?
Would definitely recommending installing sar (or sysstat package) if
you are running linux.
If Linux, which kernel?
If it really is heap, have a look at:
http://hausheer.osola.com/docs/5 for a simple description on how to
fix this...
Below is the google link which shows how this was found....
http://www.google.com/search?rls=en-us&q=java.lang.OutOfMemoryError:+Java+heap+space&ie=UTF-8&oe=UTF-8
> The thing I am wondering also is why it shuts down cleanly
> sometimes. Is
> there something internally that triggers that with this version of
> Tomcat,
> would an out of heap memory situation cause that. Maybe its in one
> of the
> logs but I am still going through all of them. There are multiple
> megs of
> them.
Are you sure it shuts down cleanly - this seems strange - you may want
to chnage `ulimit -c 1000` so that you can see whether it really core
dumps or not....
what does `ps auxH` report on your machine?
How much memory is the thing using?
How high have you configured your memory settings for the JVM?
Cheers
Andrew
Re: Tomcat 4.1.31 crashing with memory errors, crashing with no errors and shutting down cleanly without manual intervention
Posted by Sean Carnes <sc...@gmail.com>.
Andrew,
The performance data that we have is for the backend servers. We are
monitoring the normal things cpu, memory, load swap, context switches,
threads etc etc on the system but nothing specific to the tomcat service,
just the overall health of the system. It is not easy to put a lot of
profiling on a production server, it may be a good thing to do to find out
the issue but not something that anyone wants on in production.
Cataline.out from one of the crashes is below :
2007-12-05 13:42:50 CoyoteAdapter An exception or error occurred in the
container during the request processing
java.lang.OutOfMemoryError: Java heap space
at org.apache.tomcat.util.buf.ByteChunk.toString(ByteChunk.java:457)
at org.apache.tomcat.util.buf.MessageBytes.toString(MessageBytes.java
:192)
at org.apache.tomcat.util.http.MimeHeaders.getHeader(MimeHeaders.java
:293)
at org.apache.coyote.Request.getHeader(Request.java:345)
at org.apache.coyote.tomcat4.CoyoteAdapter.postParseRequest(
CoyoteAdapter.java:201)
at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java
:150)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java
:799)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection
(Http11Protocol.java:705)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java
:577)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(
ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:595)
2007-12-05 13:43:55 CoyoteAdapter An exception or error occurred in the
container during the request processing
java.lang.OutOfMemoryError: Java heap space
2007-12-05 13:44:08 CoyoteAdapter An exception or error occurred in the
container during the request processing
java.lang.OutOfMemoryError: Java heap space
2007-12-05 13:44:21 CoyoteAdapter An exception or error occurred in the
container during the request processing
java.lang.OutOfMemoryError: Java heap space
2007-12-05 13:44:21 CoyoteAdapter An exception or error occurred in the
container during the request processing
java.lang.OutOfMemoryError: Java heap space
2007-12-05 13:44:34 CoyoteAdapter An exception or error occurred in the
container during the request processing
java.lang.OutOfMemoryError: Java heap space
2007-12-05 13:44:46 CoyoteAdapter An exception or error occurred in the
container during the request processing
java.lang.OutOfMemoryError: Java heap space
Sun 1.4.2 Jvm
A stack trace was not done yet.
backend stats for cpu/memory/disk look normal we collect them every 15
minutes.
The thing I am wondering also is why it shuts down cleanly sometimes. Is
there something internally that triggers that with this version of Tomcat,
would an out of heap memory situation cause that. Maybe its in one of the
logs but I am still going through all of them. There are multiple megs of
them.
Sean
On Dec 6, 2007 11:23 AM, Andrew Miehs <an...@2sheds.de> wrote:
>
> On 06/12/2007, at 5:12 PM, Peter Crowther wrote:
>
> >> From: Sean Carnes [mailto:scarnes@gmail.com]
> >> The back-end
> >> servers seem to be responding in a timely fashion right now. We have
> >> performance data from the time period and nothing seems
> >> abnormal.
>
> I unfortunately missed the first part of this thread ...
>
> If you are having problems and your performance data show nothing
> abnormal,
> then you either do not have enough data or you do not have a
> problem.. :-)
>
> What errors are you seeing? (What is in catalina.out)?
> Are you running out of threads? (you are probably runing JVM 1.42 based
> on the version of tomcat you are running - Sun and IBM JVM 1.42 used to
> respond with Out of memory when you were out of threads)...
>
> Have you done a stack trace on the tomcat?
>
> Do you have disk performance stats from the backend as well as CPU,
> and load?
>
> At the moment, it could be anything, but if you say you have 50% more
> users,
> this something could very easily be the effect of a little more load,
> which
> brings the whole thing to a standstill
>
>
> Cheers
>
> Andrew
>
>
>
--
"Any sufficiently advanced technology is indistinguishable from magic."
- Arthur C. Clarke
Re: Tomcat 4.1.31 crashing with memory errors, crashing with no errors and shutting down cleanly without manual intervention
Posted by Andrew Miehs <an...@2sheds.de>.
On 06/12/2007, at 5:12 PM, Peter Crowther wrote:
>> From: Sean Carnes [mailto:scarnes@gmail.com]
>> The back-end
>> servers seem to be responding in a timely fashion right now. We have
>> performance data from the time period and nothing seems
>> abnormal.
I unfortunately missed the first part of this thread ...
If you are having problems and your performance data show nothing
abnormal,
then you either do not have enough data or you do not have a
problem.. :-)
What errors are you seeing? (What is in catalina.out)?
Are you running out of threads? (you are probably runing JVM 1.42 based
on the version of tomcat you are running - Sun and IBM JVM 1.42 used to
respond with Out of memory when you were out of threads)...
Have you done a stack trace on the tomcat?
Do you have disk performance stats from the backend as well as CPU,
and load?
At the moment, it could be anything, but if you say you have 50% more
users,
this something could very easily be the effect of a little more load,
which
brings the whole thing to a standstill
Cheers
Andrew
RE: Tomcat 4.1.31 crashing with memory errors, crashing with no
errors and shutting down cleanly without manual intervention
Posted by Peter Crowther <Pe...@melandra.com>.
> From: Sean Carnes [mailto:scarnes@gmail.com]
> Thanks for the quick response. Your evaluation is similar to
> what I have
> been saying to my counterparts in regards to load balancing.
> The back-end
> servers seem to be responding in a timely fashion right now. We have
> performance data from the time period and nothing seems
> abnormal.
I can well believe it. What I'm pointing out is that *if* you could squeeze shorter response times out of these back-end servers, you might be able to ride out the increased load on the Tomcat servers. If your system's already as well-tuned as I suspect, however, there's probably nothing more to be gained there.
> I think
> this can be attributed mostly to the increased load, on that
> that day due to shift switchover we have about 50% more users.
Crikey. 50% more users is not a small amount!
Do you have any monitoring deployed that allows you to monitor free Java heap space and/or garbage collections? That would tell you how much headroom you had, and might be able to warn you of impending failure. I'd be cautious of adding too much monitoring, though, as it'll reduce your throughput and might tip a system over the edge sooner than it'd otherwise fail.
> Wednesday was the most that
> we have ever had. Thanks for the suggestions, you certainly helped me
> backup my theory to the powers that be.
All (mailing-list) advice free, and worth every penny! There are folks on this list who have much more experience with large systems than I have, and I'm rather hoping they'll chip in with their views.
- Peter
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 4.1.31 crashing with memory errors, crashing with no errors and shutting down cleanly without manual intervention
Posted by Sean Carnes <sc...@gmail.com>.
Pete,
Thanks for the quick response. Your evaluation is similar to what I have
been saying to my counterparts in regards to load balancing. The back-end
servers seem to be responding in a timely fashion right now. We have
performance data from the time period and nothing seems abnormal. I think
this can be attributed mostly to the increased load, on that that day due to
shift switchover we have about 50% more users. Wednesday was the most that
we have ever had. Thanks for the suggestions, you certainly helped me
backup my theory to the powers that be.
Regards,
Sean
On Dec 6, 2007 10:43 AM, Peter Crowther <Pe...@melandra.com> wrote:
> > From: Peter Crowther
> > Looks like that slight increase in load has tipped you over
> > from being just-about-alright to just-about-failing. If you
> > can't increase heap space, can't decrease load and can't
> > alter the application, your only remaining choice is to add
> > capacity: install another server and load-balance across N+1
> > servers rather than N servers. Also remember that if you
> > have plenty of spare RAM on the machines hosting Tomcat, and
> > it's just maximum heap size that's the issue, then you might
> > be able to run two Tomcat processes (on different ports) on
> > the same server, avoiding the need to deploy new hardware.
>
> Bad Netiquette to follow up my own answer, but I missed a point.
>
> Often, the amount of heap space required is related to the number of
> concurrent requests being processed. If this is the case for you, and
> you're using back-end services like relational databases, web services and
> the like, then you might be able to salvage the situation by speeding up one
> or more of those back-end services so that requests complete faster and
> hence there are fewer concurrent requests waiting on the Tomcat webapps.
> It's another option to consider if you have some control over these
> services but none over the Tomcat app.
>
> - Peter
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
--
"Any sufficiently advanced technology is indistinguishable from magic."
- Arthur C. Clarke
RE: Tomcat 4.1.31 crashing with memory errors, crashing with no
errors and shutting down cleanly without manual intervention
Posted by Peter Crowther <Pe...@melandra.com>.
> From: Peter Crowther
> Looks like that slight increase in load has tipped you over
> from being just-about-alright to just-about-failing. If you
> can't increase heap space, can't decrease load and can't
> alter the application, your only remaining choice is to add
> capacity: install another server and load-balance across N+1
> servers rather than N servers. Also remember that if you
> have plenty of spare RAM on the machines hosting Tomcat, and
> it's just maximum heap size that's the issue, then you might
> be able to run two Tomcat processes (on different ports) on
> the same server, avoiding the need to deploy new hardware.
Bad Netiquette to follow up my own answer, but I missed a point.
Often, the amount of heap space required is related to the number of concurrent requests being processed. If this is the case for you, and you're using back-end services like relational databases, web services and the like, then you might be able to salvage the situation by speeding up one or more of those back-end services so that requests complete faster and hence there are fewer concurrent requests waiting on the Tomcat webapps. It's another option to consider if you have some control over these services but none over the Tomcat app.
- Peter
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 4.1.31 crashing with memory errors, crashing with no
errors and shutting down cleanly without manual intervention
Posted by Peter Crowther <Pe...@melandra.com>.
> From: Sean Carnes [mailto:scarnes@gmail.com]
> *We are having an issue with the tomcat service crashing
> version 4.1.31,
> sometimes with these memory errors and sometimes not. We
> have a backup but
> once the load moves to that server the backup crashes also almost
> immediately after the load balancer switches. This has
> happened multiple
> times to us over the past few days with no changes made to
> the server, just a slight increase in load.
Looks like that slight increase in load has tipped you over from being just-about-alright to just-about-failing. If you can't increase heap space, can't decrease load and can't alter the application, your only remaining choice is to add capacity: install another server and load-balance across N+1 servers rather than N servers. Also remember that if you have plenty of spare RAM on the machines hosting Tomcat, and it's just maximum heap size that's the issue, then you might be able to run two Tomcat processes (on different ports) on the same server, avoiding the need to deploy new hardware.
- Peter
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org