You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mark Walker <sl...@hotmail.com> on 2002/12/05 17:36:46 UTC
Re: Thread count growth, /manager/sessions, and persistence
First, thanks to Glenn for your input. It has helped!
I have 3 areas of concern. I posted all in one e-mail to avoid duplicating
info and because they might be related (sorry if that was wrong). I had to
switch to my Tomcat server Monday because the WebSphere one got hung up (a
very rare occurrence). I was up on Tomcat for 2 hours. I had increased
maxProcessors to 10,000 since that is how many my other iSeries uses
(WebSphere:270 + IBM HTTP Server:10,005). The Tomcat server ran out of
memory well before that point (I think the thread count was between 3500 and
4500), which is hard to believe since I start the jvm with these memory
options: -Xms1024m and -Xmx3072m (the machine has 4GB of RAM). Here's where
I'm at now in each area:
#1 Threads: I ran a dump of the jvm (iSeries equivalent of the unix "quit",
I think) about an hour after taking the Tomcat server off of the internet.
It has 1773 threads in use. None of the stack traces reference any of my
classes. 12 of the threads appear to be used by Tomcat for the other
applications (examples, etc.). 86 of them are the open socket connections to
the various remote databases my application needs. The other 1675 break down
as follows:
-----------------------------------
63 of them are like this:
Thread priority: 5
Thread status: Waiting
Wait object: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable
Thread group: main
Runnable: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable
Stack:
org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run()V+0
(ThreadPool.java:507)
java/lang/Thread.run()V+11 (Thread.java:484)
Locks:
None
(any ideas here?)
-----------------------------------
1610 look like this:
Thread priority:5
Thread status: Java wait
Thread group: main
Runnable: org/apache/tomcat/util/threads/ThreadPool$ControlRunnable
Stack:
java/net/PlainSocketImpl.accept(Ljava/net/SocketImpl;)V+0
(PlainSocketImpl.java:413)
java/net/ServerSocket.implAccept(Ljava/net/Socket;)V+36
(ServerSocket.java:241)
com/sun/net/ssl/internal/ssl/SSLServerSocketImplŽ
.accept()Ljava/net/Socket;+0 (DashoA6275:0)
org/apache/tomcat/util/net/jsse/JSSESocketFactory.acceptSocket(Ljava/net/ServerSocket;)Ljava/net/Socket;+0
(JSSESocketFactory.java:238)
org/apache/tomcat/util/net/PoolTcpEndpoint.acceptSocket()Ljava/net/Socket;+0
(PoolTcpEndpoint.java:332)
org/apache/tomcat/util/net/TcpWorkerThread.runIt([Ljava/lang/Object;)V+0
(PoolTcpEndpoint.java:492)
org/apache/tomcat/util/threads/ThreadPool$ControlRunnable.run()V+0
(ThreadPool.java:507)
java/lang/Thread.run()V+11 (Thread.java:484)
Locks:
None
-----------------------------------
There are two other threads with identical call stacks to the 1610 group.
The difference is that they both show a thread status of "Running" . Also,
one of them lists the following lock information:
Locks:
Object id: E4777F625DC75890
Number of locks: 1
There is no reference to this object id anywhere else in any of the logs.
Does anyone have any ideas about these? Should I be concerned about them, or
are these just sockets the system is keeping available because it doesn't
"need" to garbage collect them?
#2 - Manager app session count:
Sometime in the first hour (probably at the 30 minute mark) of sitting idle
the manager app session count fell from 3671 to 1795. After three days it
still returns the following to a /sessions request: "... maximum session
inactive interval 30 - <40 minutes:1795 sessions"
- Is there a way to query Tomcat about these sessions via the manager
app or my web app?
- What would cause the manager app to not break down the list in 10
minute intervals as sessions near the timeout of 30 minutes (it has not
broken down the list in over 6 weeks of testing)?
- There can be a large number of objects references stored in a
session. I do call invalidate() as part of the logoff servlet, but I don't
null the references. Do I need to null them? If so, do I need to null each
reference inside a collection, or just the collection itself?
#3 - Persistent sessions:
I assume that my <Manager> group is correct to eliminate persistent
sessions(see below). If not, please let me know.
Thanks again for any help!
Mark Walker
>From: Glenn Nielsen <gl...@mail.more.net>
>Reply-To: "Tomcat Users List" <to...@jakarta.apache.org>
>To: Tomcat Users List <to...@jakarta.apache.org>
>Subject: Re: Thread count growth, /manager/sessions, and persistence
>Date: Tue, 26 Nov 2002 22:36:50 -0600
>
>Regarding sessions, by default a JSP page creates a session.
>All your JSP pages which don't require a session should have
>the attribute session="false" set in the page directive.
>
>The number of threads being used is not related to session management.
>
>To troubleshoot problems running out of Processors try telling the
>JVM to do a thread Stackdump when this happens. On unix you send
>the java process a QUIT signal. Analyze the thread Stackdumps to
>determine what the bottleneck is for your application running in
>Tomcat. i.e. What are all the threads doing when you run out of
>processors.
>
>BTW, 2500 processors seems excessive. Does your site really
>need to handle 2500 concurrent requests?
>
>Also try starting the JVM with the -verbose:gc arg using the
>CATALINA_OPTS environment variable. This will generate garbage
>collection data to stdout (catalina.out). Look for very long
>GC times in the 10's of seconds. You may need to tune your
>java min/max heap sizes and other GC related options to improve
>GC performance. When GC happens Tomcat freezes. This can cause
>requests to build up on the Coyote socket up to your acceptCount
>limit. And it can cause other failures for things which can timeout
>such as db connections.
>
>Regards,
>
>Glenn
>
>Mark Walker wrote:
>>I am trying to port an application over to Tomcat v 4.1.12 which has
>>worked fine for two years running in WebSphere on an iSeries. The JSPs
>>were at the .91 level. I have updated them to the 1.2 level. In addition,
>>I have installed the tomcat-util.jar from 4.1.14 to fix the erroneous ssl
>>debug messaging. From the user's perspective everything looks the same.
>>On the server side though, I seem to have a runaway thread problem. The
>>Tomcat manager app shows the number of active sessions to be anywhere
>>between 400 and 700 during the day, dwindling to a few dozen at night.
>>(Incidentally, the session count is always in one group: the 30-40 minute
>>one - I expected a breakdown in 10 minute intervals). But the thread count
>>just keeps rising. When the count reaches the maxProcessors limit a
>>message, "[INFO]ThreadPool - -All threads are busy, waiting. Please
>>increase maxThreads or check the servlet status" appears in the log. I
>>have tried "suggesting" garbage collection to the JVM at various times but
>>to no avail.
>>
>>Here are the <Connector> definitions in use:
>>
>> <Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
>> port="80" minProcessors="5" maxProcessors="2500"
>> enableLookups="true" redirectPort="443"
>> acceptCount="25" debug="0" connectionTimeout="20000"
>> useURIValidationHack="false" />
>>
>> <Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
>> port="443" minProcessors="25" maxProcessors="2500"
>> enableLookups="true"
>> acceptCount="25" debug="0" scheme="https" secure="true"
>> useURIValidationHack="false">
>> <Factory
>>className="org.apache.coyote.tomcat4.CoyoteServerSocketFactory"
>> clientAuth="false" keystoreFile="**filename**"
>> keystorePass="**password**" keystoreType="**type**"
>>protocol="TLS"/>
>> </Connector>
>>
>>(There is also the standard Coyote/JK2 AJP 1.3 Connector on 8009 but it
>>looks like I don't need that as I'm using Tomcat as a standalone server)
>>
>>
>>Here are my questions:
>>
>>Are there any known issues with Tomcat and excessive thread counts?
>>(Grasping at straws here I know. I have not found any other relevant
>>reference to Tomcat having thread issues like these, yet the same Java
>>code has served well for two years.)
>>
>>Could the lack of a breakdown list in the /manager/sessions output be a
>>symptom of a thread problem or is it just a newbie configuration error?
>>
>>Another item that might be an issue is persistent sessions. I don't want
>>them. I invalidate() the session when a user logs out. To that end I have
>>included the following within each <context> group:
>>
>> <Manager
>>className="org.apache.catalina.session.PersistentManager"
>> debug="0"
>> saveOnRestart="false"
>> maxActiveSessions="-1"
>> minIdleSwap="1800"
>> maxIdleSwap="-1"
>> maxIdleBackup="-1">
>> <Store
>>className="org.apache.catalina.session.FileStore"/>
>> </Manager>
>>
>>(I have also used -1 for the minIdleSwap. I changed it to 1800 to try and
>>keep the sessions in memory in case that was part of the problem.) Is this
>>the right approach to take? I'm concerned that something is wrong here
>>because if I end Tomcat and restart it the cpu usage goes from an average
>>of 6% to 40+%. A second restart puts it up to 99%. The system maintains
>>these rates until I switch to another server (At least the pages still
>>serve fine though).
>>
>>Lastly, in the event that WebSphere has just been covering my tracks well,
>>I would also welcome suggestions of Java discussion lists anyone has had
>>success with where I could post some particulars regarding our
>>multithreaded processing.
>>
>>
>>Thanks in advance!
>>
>>
>>
>>Mark Walker
>>
>>
>>_________________________________________________________________
>>Add photos to your e-mail with MSN 8. Get 2 months FREE*.
>>http://join.msn.com/?page=features/featuredemail
>>
>>
>>--
>>To unsubscribe, e-mail:
>><ma...@jakarta.apache.org>
>>For additional commands, e-mail:
>><ma...@jakarta.apache.org>
>
>
>
>
>--
>To unsubscribe, e-mail:
><ma...@jakarta.apache.org>
>For additional commands, e-mail:
><ma...@jakarta.apache.org>
_________________________________________________________________
The new MSN 8: advanced junk mail protection and 2 months FREE*
http://join.msn.com/?page=features/junkmail
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: How to set JVM memory size?
Posted by Ming Zhao <fl...@yahoo.com>.
Thank Jeremy and Brandon for you help.
Under windows 2k, I set an environment variable to
change the java heap size within JVM (just as Brandom
said). But it seems not work. So I'm not suer my
setting is correct or there is other problem.
At first I thought it's memory problem because I got
an error when I run a simulation java bean with longer
running time (real model time, not the bean running
time, I can set it on the website); but with short
running time, it can work well. The exception is:
type Exception report
message
description The server encountered an internal error
() that prevented it from fulfilling this request.
exception
org.apache.jasper.JasperException
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:248)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295)
at
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:431)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:386)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:537)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:533)
at java.lang.Thread.run(Thread.java:484)
root cause
javax.servlet.ServletException
at
org.apache.jasper.runtime.PageContextImpl.handlePageException(PageContextImpl.java:533)
at
org.apache.jsp.simul_jsp._jspService(simul_jsp.java:317)
at
org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:137)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:204)
at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:295)
at
org.apache.jasper.servlet.JspServlet.service(JspServlet.java:241)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:853)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:247)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:260)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardContext.invoke(StandardContext.java:2415)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.valves.ErrorDispatcherValve.invoke(ErrorDispatcherValve.java:170)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:172)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:641)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:174)
at
org.apache.catalina.core.StandardPipeline$StandardPipelineValveContext.invokeNext(StandardPipeline.java:643)
at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:480)
at
org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:995)
at
org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java:223)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:431)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:386)
at
org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:537)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:533)
at java.lang.Thread.run(Thread.java:484)
Do you have any suggestion?
Ming
--- Jeremy Joslin <je...@spotlife.com> wrote:
> It's important to realize that this is setting the
> size of the Java heap
> within the JVM (which sounds like what you're
> looking for) and not the
> actual size of the JVM itself. Also note that
> you're not always going
> to get the best performance out of your app by using
> a large heap size
> because while it decreases the frequency of GC's the
> time each GC takes
> is going to increase (more to collect). So, and I
> know I'm not the
> first to say this, play around with the different
> settings until you
> find the one that suits your app.
>
> Jeremy
>
> > -----Original Message-----
> > From: Brandon Cruz [mailto:bcruz@norvax.com]
> > Sent: Thursday, December 05, 2002 10:27 AM
> > To: Tomcat Users List
> > Subject: RE: How to set JVM memory size?
> >
> > Set an environment variable called JAVA_OPTS and
> set your max and min
> to
> > whatever you want like this...
> >
> > -Xmx512m -Xms256m
> >
> > Brandon
> >
> > -----Original Message-----
> > From: Ming Zhao [mailto:flying_minger@yahoo.com]
> > Sent: Thursday, December 05, 2002 11:49 AM
> > To: tomcat-user@jakarta.apache.org
> > Subject: How to set JVM memory size?
> >
> >
> > Hi,
> >
> > I have a question to seek for your help. How do I
> set
> > the JVM memory size? I use Apache Tomcat4.1.16 and
> > J2SDK1.3. Now it seems that I have a problem of
> not
> > enough memory to implement the java bean. I want
> to
> > set JVM memory with writing some commands to some
> > file. So Tomcat can activate J2SDK with set memory
> > size. How to do that?
> >
> > Thanks in advance,
> > Ming
> >
> > __________________________________________________
> > Do you Yahoo!?
> > New DSL Internet Access from SBC & Yahoo!
> > http://sbc.yahoo.com
> >
> > --
> > To unsubscribe, e-mail:
> >
> <ma...@jakarta.apache.org>
> > For additional commands, e-mail:
> > <ma...@jakarta.apache.org>
> >
> >
> >
> > --
> > To unsubscribe, e-mail: <mailto:tomcat-user-
> > unsubscribe@jakarta.apache.org>
> > For additional commands, e-mail:
> <mailto:tomcat-user-
> > help@jakarta.apache.org>
>
>
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: How to set JVM memory size?
Posted by Jeremy Joslin <je...@spotlife.com>.
It's important to realize that this is setting the size of the Java heap
within the JVM (which sounds like what you're looking for) and not the
actual size of the JVM itself. Also note that you're not always going
to get the best performance out of your app by using a large heap size
because while it decreases the frequency of GC's the time each GC takes
is going to increase (more to collect). So, and I know I'm not the
first to say this, play around with the different settings until you
find the one that suits your app.
Jeremy
> -----Original Message-----
> From: Brandon Cruz [mailto:bcruz@norvax.com]
> Sent: Thursday, December 05, 2002 10:27 AM
> To: Tomcat Users List
> Subject: RE: How to set JVM memory size?
>
> Set an environment variable called JAVA_OPTS and set your max and min
to
> whatever you want like this...
>
> -Xmx512m -Xms256m
>
> Brandon
>
> -----Original Message-----
> From: Ming Zhao [mailto:flying_minger@yahoo.com]
> Sent: Thursday, December 05, 2002 11:49 AM
> To: tomcat-user@jakarta.apache.org
> Subject: How to set JVM memory size?
>
>
> Hi,
>
> I have a question to seek for your help. How do I set
> the JVM memory size? I use Apache Tomcat4.1.16 and
> J2SDK1.3. Now it seems that I have a problem of not
> enough memory to implement the java bean. I want to
> set JVM memory with writing some commands to some
> file. So Tomcat can activate J2SDK with set memory
> size. How to do that?
>
> Thanks in advance,
> Ming
>
> __________________________________________________
> Do you Yahoo!?
> New DSL Internet Access from SBC & Yahoo!
> http://sbc.yahoo.com
>
> --
> To unsubscribe, e-mail:
> <ma...@jakarta.apache.org>
> For additional commands, e-mail:
> <ma...@jakarta.apache.org>
>
>
>
> --
> To unsubscribe, e-mail: <mailto:tomcat-user-
> unsubscribe@jakarta.apache.org>
> For additional commands, e-mail: <mailto:tomcat-user-
> help@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
RE: How to set JVM memory size?
Posted by Brandon Cruz <bc...@norvax.com>.
Set an environment variable called JAVA_OPTS and set your max and min to
whatever you want like this...
-Xmx512m -Xms256m
Brandon
-----Original Message-----
From: Ming Zhao [mailto:flying_minger@yahoo.com]
Sent: Thursday, December 05, 2002 11:49 AM
To: tomcat-user@jakarta.apache.org
Subject: How to set JVM memory size?
Hi,
I have a question to seek for your help. How do I set
the JVM memory size? I use Apache Tomcat4.1.16 and
J2SDK1.3. Now it seems that I have a problem of not
enough memory to implement the java bean. I want to
set JVM memory with writing some commands to some
file. So Tomcat can activate J2SDK with set memory
size. How to do that?
Thanks in advance,
Ming
__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com
--
To unsubscribe, e-mail:
<ma...@jakarta.apache.org>
For additional commands, e-mail:
<ma...@jakarta.apache.org>
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>
How to set JVM memory size?
Posted by Ming Zhao <fl...@yahoo.com>.
Hi,
I have a question to seek for your help. How do I set
the JVM memory size? I use Apache Tomcat4.1.16 and
J2SDK1.3. Now it seems that I have a problem of not
enough memory to implement the java bean. I want to
set JVM memory with writing some commands to some
file. So Tomcat can activate J2SDK with set memory
size. How to do that?
Thanks in advance,
Ming
__________________________________________________
Do you Yahoo!?
New DSL Internet Access from SBC & Yahoo!
http://sbc.yahoo.com
--
To unsubscribe, e-mail: <ma...@jakarta.apache.org>
For additional commands, e-mail: <ma...@jakarta.apache.org>