You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Breno Jacinto <br...@freeunix.com.br> on 2007/12/03 19:19:43 UTC
Hundreds of Instances
Hello,
I have an application running under Tomcat for a couple of months
now, and itś always been memory intensive (I run it in a combination
with Apache2 + JK + Berkeley DB XML in the backend). But from a few
days to today, it's been eating memory like crazy, and I can see that
when I have to restart it very often - 170 Tomcat instances running!
The site running is not that traffic intensive, so my first
supposition is that something is getting locked and making Tomcat
unable to release the memory when the job is done. So instances get
created but not destroyed. And memory progressively is filled, up to
50% of swap, then Apache2 stops responding, BDB XML gives
out-of-memory messages etc... which is pretty normal.
My question is if this is related to Tomcat directly - like a
memory leak, or if this is application / database specific. How could
I investigate this further? My setup:
Debian Linux 2.6.18-5-686 #1 SMP
Apache 2.0.54
Tomcat 5.5.15
JDK 1.5.0_06-b05
BDB XML 2.2.13
1.5 GB of RAM
regards,
--
--
:: Breno Jacinto ::
:: breno - at - gprt.ufpe.br ::
:: FingerPrint ::
2F15 8A61 F566 E442 8581
E3C0 EFF4 E202 74B7 7484
:: Persistir no difícil é a única maneira de torná-lo fácil algum dia. ::
Re: Hundreds of Instances
Posted by Breno Jacinto <br...@freeunix.com.br>.
Hi Christopher,
> Nope. You'll have to write it yourself. fortunately, It's fairly simple
> to write. Never written a Filter before? Just read the javadoc for the
> Filter interface and it should all be clear.
Thanks a lot. I'll check it out.
>
> - -chris
>
cheers,
--
--
:: Breno Jacinto ::
:: breno - at - gprt.ufpe.br ::
:: FingerPrint ::
2F15 8A61 F566 E442 8581
E3C0 EFF4 E202 74B7 7484
:: Persistir no difícil é a única maneira de torná-lo fácil algum dia. ::
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Breno,
Breno Jacinto wrote:
>> /foo/bar.jsp?id=1324 27ms
>> /foo/bar/jsp?id=1235 31ms
>> /foo/baz.jsp 897234ms
>
> Is this filter provided by Tomcat? Is there any documentation on
> this? The information would be very helpful, indeed.
Nope. You'll have to write it yourself. fortunately, It's fairly simple
to write. Never written a Filter before? Just read the javadoc for the
Filter interface and it should all be clear.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHVc6+9CaO5/Lv0PARAvB7AJ0aP0yhfoyzkFPLfnDrHXqbtEWPsgCguzIq
xBj2uWgYRfCNvrW6eAvd0kc=
=6FU+
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Breno Jacinto <br...@freeunix.com.br>.
Hi Christopher,
2007/12/4, Christopher Schultz <ch...@christopherschultz.net>:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Breno,
>
> Breno Jacinto wrote:
> > http://www.freeunix.com.br/tomcat/all-2007-12-04-after-restart.html
>
> It looks like your JSPs are doing a /lot/ of work:
>
> jsp [ *.jsp , *.jspx ]
> Processing time: 28.512 s Max time: 14843 ms Request count: 8 Error
> count: 0 Load time: 1 ms Classloading time: 1 ms
>
> 8 requests tool a total of 28 seconds to process? And one of those
> requests took 14 seconds? That's pretty slow. Do you know which request
> that was?
>
> In the [jk-8009] section of the "before restart", it looks like you've
> got a lot going on:
>
> Max processing time: 725764 ms
>
> If it's really taking more than 10 minutes to process a request, then
> something is definitely wrong. I wouldn't be surprised if you have
> long-running processes that continue after the client has disconnected.
> These processes tie up threads needlessly.
Yes, thats exactly the point. I'm sure there's something that
ties up the threads, I dont know yet what request(s).
>
> I would write a simple filter that logs the request URI + query string
> to a log file, then times the request processing itself and writes that
> out, too. Basically, you'll get a log file that says:
>
> /foo/bar.jsp?id=1324 27ms
> /foo/bar/jsp?id=1235 31ms
> /foo/baz.jsp 897234ms
Is this filter provided by Tomcat? Is there any documentation on
this? The information would be very helpful, indeed.
>
> then you can see which requests take forever and maybe find out why. The
> first step in debugging is reproducibility ;)
:) thanks for the advice.
>
> - -chris
best regards,
--
--
:: Breno Jacinto ::
:: breno - at - gprt.ufpe.br ::
:: FingerPrint ::
2F15 8A61 F566 E442 8581
E3C0 EFF4 E202 74B7 7484
:: Persistir no difícil é a única maneira de torná-lo fácil algum dia. ::
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Breno,
Breno Jacinto wrote:
> http://www.freeunix.com.br/tomcat/all-2007-12-04-after-restart.html
It looks like your JSPs are doing a /lot/ of work:
jsp [ *.jsp , *.jspx ]
Processing time: 28.512 s Max time: 14843 ms Request count: 8 Error
count: 0 Load time: 1 ms Classloading time: 1 ms
8 requests tool a total of 28 seconds to process? And one of those
requests took 14 seconds? That's pretty slow. Do you know which request
that was?
In the [jk-8009] section of the "before restart", it looks like you've
got a lot going on:
Max processing time: 725764 ms
If it's really taking more than 10 minutes to process a request, then
something is definitely wrong. I wouldn't be surprised if you have
long-running processes that continue after the client has disconnected.
These processes tie up threads needlessly.
I would write a simple filter that logs the request URI + query string
to a log file, then times the request processing itself and writes that
out, too. Basically, you'll get a log file that says:
/foo/bar.jsp?id=1324 27ms
/foo/bar/jsp?id=1235 31ms
/foo/baz.jsp 897234ms
then you can see which requests take forever and maybe find out why. The
first step in debugging is reproducibility ;)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHVX6W9CaO5/Lv0PARAll/AKCrGWjDPoLPEabVjwF6gnqX2v+aMACgwMYy
8iD5GOX/tUzhqKId9NDnHao=
=zvyZ
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Breno Jacinto <br...@freeunix.com.br>.
Hi Chuck,
2007/12/3, Caldarale, Charles R <Ch...@unisys.com>:
> > From: Breno Jacinto [mailto:breno@freeunix.com.br]
> > Subject: Re: Hundreds of Instances
> >
> > when does tomcat creates a new thread, and
> > when and how does it get deallocated?
>
> This is controlled by the maxThreads, maxSpareThreads, and
> minSpareThreads attributes of <Connector> elements:
> http://tomcat.apache.org/tomcat-5.5-doc/config/ajp.html
> http://tomcat.apache.org/tomcat-5.5-doc/config/http.html
>
> However, this may not be the "instances" Tomcat is talking about during
> termination. What is the exact message you're seeing? It may be
> related to the number of requests or servlets still active, which may
> indicate you have a webapp that gets stalled and won't finish
> processing. That could easily result in excessive memory consumption.
I hadn'd thought about it. But to get things right, here's is how
the server is behaving, right now.
SCJDS:/usr/local/tomcat/bin# free -m
total used free shared buffers cached
Mem: 1488 1456 31 0 6 18
-/+ buffers/cache: 1431 56
Swap: 2651 1041 1609
SCJDS:/usr/local/tomcat/bin#
I dont think it takes a long time until it everything starts
getting slow and unresponsive. More detailed information, I have the
complete status of the server, according to the Tomcat manager status
app: http://www.freeunix.com.br/tomcat/all-2007-12-04.html
There you can also see the JVM memory status. Even if Linux is
reporting lots of memory consumption, the JVM seems OK (71.54 MB
free, total 126.68 MB).
So, now, I stop the service:
SCJDS:~# /etc/init.d/tomcat stop
Here's what I get in the log:
INFO: Pausing Coyote HTTP/1.1 on http-8080
04/12/2007 12:48:48 org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
.BdbXmlListener: Fechando ambiente
.EnvironmentInit:fechaContainer:LogAdminAcesso.dbxml
.EnvironmentInit:fechaContainer:Usuarios.dbxml
.EnvironmentInit:fechaContainer:MaquinasAdmin.dbxml
.EnvironmentInit:fechaContainer:BOs.dbxml
.EnvironmentInit:fechaContainer:ContadorSite.dbxml
.EnvironmentInit:fechaContainer:LogAdminAcao.dbxml
.EnvironmentInit:fechaContainer:ResponsaveisDelegacia.dbxml
.EnvironmentInit:fechaContainer:LogVitimaAcesso.dbxml
.EnvironmentInit:cleanup
.BdbXmlListener: Ambiente fechado
04/12/2007 12:49:15 org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 160 instance(s) to be deallocated
04/12/2007 12:49:17 org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 160 instance(s) to be deallocated
04/12/2007 12:49:18 org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 160 instance(s) to be deallocated
04/12/2007 12:49:18 org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 1 instance(s) to be deallocated
04/12/2007 12:49:19 org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 1 instance(s) to be deallocated
04/12/2007 12:49:20 org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 1 instance(s) to be deallocated
04/12/2007 12:49:36 org.apache.coyote.http11.Http11BaseProtocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080
04/12/2007 12:49:36 org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: Failed shutdown of Apache Portable Runtime
See, many instances. Now, my memory status:
SCJDS:~# free -m
total used free shared buffers cached
Mem: 1488 136 1352 0 11 50
-/+ buffers/cache: 75 1413
Swap: 2651 54 2596
SCJDS:~#
More than 1 GB free. The server status, after the restart:
http://www.freeunix.com.br/tomcat/all-2007-12-04-after-restart.html
This behaviour is pretty consistent, tomorrow I'll get the same
consumption and have to restart the server again.
I'll take a look at the servlets you mentioned, there are plenty of them.
best regards,
>
> - Chuck
>
--
--
:: Breno Jacinto ::
:: breno - at - gprt.ufpe.br ::
:: FingerPrint ::
2F15 8A61 F566 E442 8581
E3C0 EFF4 E202 74B7 7484
:: Persistir no difícil é a única maneira de torná-lo fácil algum dia. ::
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Hundreds of Instances
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Breno Jacinto [mailto:breno@freeunix.com.br]
> Subject: Re: Hundreds of Instances
>
> when does tomcat creates a new thread, and
> when and how does it get deallocated?
This is controlled by the maxThreads, maxSpareThreads, and
minSpareThreads attributes of <Connector> elements:
http://tomcat.apache.org/tomcat-5.5-doc/config/ajp.html
http://tomcat.apache.org/tomcat-5.5-doc/config/http.html
However, this may not be the "instances" Tomcat is talking about during
termination. What is the exact message you're seeing? It may be
related to the number of requests or servlets still active, which may
indicate you have a webapp that gets stalled and won't finish
processing. That could easily result in excessive memory consumption.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Breno Jacinto <br...@freeunix.com.br>.
Hi,
Just one curiosity came to my mind: when does tomcat creates a new
thread, and when and how does it get deallocated? If this is not a
question for this list, please ignore it.
cheers,
--
:: Breno Jacinto ::
:: breno - at - gprt.ufpe.br ::
:: FingerPrint ::
2F15 8A61 F566 E442 8581
E3C0 EFF4 E202 74B7 7484
:: Persistir no difícil é a única maneira de torná-lo fácil algum dia. ::
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Hundreds of Instances
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Breno Jacinto [mailto:breno@freeunix.com.br]
> Subject: Re: Hundreds of Instances
>
> They get very, very slow because Linux starts to swap like
> crazy, after the Tomcat process eats almost 100% of RAM.
If you're encountering significant swapping with your current heap and
RAM sizes, you might want to consider running with a *smaller* heap
(assuming you can't add RAM, of course). The impact of paging on
allocation and garbage collection times can be rather significant. That
doesn't mean you should stop researching your memory usage, of course.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
MATERIAL and is thus for use only by the intended recipient. If you
received this in error, please contact the sender and delete the e-mail
and its attachments from all computers.
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Delian,
Delian Krustev wrote:
>>> Really? From your other message, you make it look like Java is taking
>>> something like 1GB of memory. Sure, the JVM adds some overhead to the
>>> heap size you configured, but it shouldn't be more than 5% or 10%...
>>> nothing like 200%.
>
> It could be more that 200% depending on the heap size and other parameters.
True. If you have a 5MB heap then maybe the JVM takes more than twice as
much space. :p
> Furthermore if the swapping is a problem you'd better limit the memory
> available to the JVM. IMHO this could not be done via VM parameters.
> Use ulimit instead.
Use JVM parameters instead of ulimit.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHWV9s9CaO5/Lv0PARAks6AJ0SwEG2hUmE4og6/pCUHGQQrhMfAwCfUwN1
4nhreRkgngACvVSwYIeQ18Q=
=lvPN
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Delian Krustev <to...@krustev.net>.
> > Really? From your other message, you make it look like Java is taking
> > something like 1GB of memory. Sure, the JVM adds some overhead to the
> > heap size you configured, but it shouldn't be more than 5% or 10%...
> > nothing like 200%.
It could be more that 200% depending on the heap size and other parameters.
> I tried JProfiler sometime ago, and it worked fine. Too bad it's
> not free software. I'll look for others.
I'd recommend forgetting about the profiler, enabling JMX, attach to the JVM
and see which type of memory is excessively used.
Furthermore if the swapping is a problem you'd better limit the memory
available to the JVM. IMHO this could not be done via VM parameters.
Use ulimit instead.
This way out of memory errors will start to appear(instead of swapping) with
the needed memory type shown in the exception.
Cheers
--
Delian
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Breno Jacinto <br...@freeunix.com.br>.
Hi Christopher,
> You could turn on the AccessLogValve and then simply look at the log
> files. Another option would be to write your own Valve or Filter and
> record the accesses yourself, but the AccessLogValve is already there. I
> think it can easily be configured to emit "standard" httpd-style logs so
> you can use tools webalizer to graph the usage over time and stuff.
I found a post in the past with a tutorial on how to do this
second option. I'll try it out, thanks!
>
> > JAVA_OPTS="-server -Xms128m -Xmx384m
> > -Djava.library.path=/usr/src/dbxml-2.2.13/install/lib/"
>
> Really? From your other message, you make it look like Java is taking
> something like 1GB of memory. Sure, the JVM adds some overhead to the
> heap size you configured, but it shouldn't be more than 5% or 10%...
> nothing like 200%.
No, I meant Tomcat is taking it! Over time, as I showed in the
"free -m" output. Also, when I look at "top", the java process is the
first one in memory consumption.
> If you really are only serving 10 users per minute, attaching a memory
> profiler isn't going to significantly impact your performance (unless
> you're running with an old VM which requires disabling the JIT in order
> to instrument the VM). You said you're running 1.5.0, which should allow
> this type of instrumentation.
I tried JProfiler sometime ago, and it worked fine. Too bad it's
not free software. I'll look for others.
cheers,
> - -chris
--
--
:: Breno Jacinto ::
:: breno - at - gprt.ufpe.br ::
:: FingerPrint ::
2F15 8A61 F566 E442 8581
E3C0 EFF4 E202 74B7 7484
:: Persistir no difícil é a única maneira de torná-lo fácil algum dia. ::
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Breno,
Breno Jacinto wrote:
> Hi Christopher,
>
>
>>> Not really. I dont have the exact numbers, but I think we'd get at
>>> most 50 users every 10 minutes = 5 users / min. I believe that´s
>>> pretty low.
>> Is that estimated, or measured?
>
> It's just estimation. BTW, is there any tool that could help me
> with effectively measuring this, get some statistics etc?
You could turn on the AccessLogValve and then simply look at the log
files. Another option would be to write your own Valve or Filter and
record the accesses yourself, but the AccessLogValve is already there. I
think it can easily be configured to emit "standard" httpd-style logs so
you can use tools webalizer to graph the usage over time and stuff.
> JAVA_OPTS="-server -Xms128m -Xmx384m
> -Djava.library.path=/usr/src/dbxml-2.2.13/install/lib/"
Really? From your other message, you make it look like Java is taking
something like 1GB of memory. Sure, the JVM adds some overhead to the
heap size you configured, but it shouldn't be more than 5% or 10%...
nothing like 200%.
> OK. The only problem is that this site is on production, but I
> believe it'll possible to run Tomcat under the profiler and get some
> significant results. An alternative is to replicate the system and try
> simulating some load and common operations (which I found out JMeter
> is the tool for this).
If you really are only serving 10 users per minute, attaching a memory
profiler isn't going to significantly impact your performance (unless
you're running with an old VM which requires disabling the JIT in order
to instrument the VM). You said you're running 1.5.0, which should allow
this type of instrumentation.
> :)). I had this experience once - it's hard to believe char[] are
> so evil! :).
Not evil... just plentiful. All those class and method names gotta go
somewhere!
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHVXy89CaO5/Lv0PARAny+AKCiGeKbj2mdkcA/VMs5Ov9eZeO4ygCeMqrG
w08hluEeYuVtcl0Piszaas4=
=YG+c
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Breno Jacinto <br...@freeunix.com.br>.
Hi Christopher,
> > Not really. I dont have the exact numbers, but I think we'd get at
> > most 50 users every 10 minutes = 5 users / min. I believe that´s
> > pretty low.
>
> Is that estimated, or measured?
It's just estimation. BTW, is there any tool that could help me
with effectively measuring this, get some statistics etc?
> > I assume the garbage collector cleans them all.
>
> As long as you are putting those search results only into request
> attributes or something like that for temporary storage, you should be
> okay. Don't forget that a single *huge* result set can still cause an
> OOME if you don't have enough memory allocated.
> Gotcha. How much memory do you have configured for the JVM, and how much
> memory is actually free? IF you're swapping a lot, maybe you need to a)
> buy more memory or b) lower the memory allocation to your app. Of
> course, lowering your memory allocation could lead to OOME, but both
> problems are effectively the same.
This is how my JAVA_OPTS goes:
JAVA_OPTS="-server -Xms128m -Xmx384m
-Djava.library.path=/usr/src/dbxml-2.2.13/install/lib/"
I had changed the default configuration after reading a High
Performance paper on Tomcat, which is on the site.
I'll post more details (the server status) in the reply to chuck,
please take a look.
> The memory profiler is your best bet. Look at, say, 24 hours worth of GC
> activity (if your server stays up that long). You should have a GC curve
> that looks like this: /\/\/\/\/\/\/\/\/\/. If if goes up and up and up,
> you obviously have a memory leak. If it only goes up at intervals (say,
> once an hour), then maybe you have a rare event or timed event that
> leaks, rather than something that happens all the time (say, with every
> request).
> That will give you some idea of what you are looking for.
OK. The only problem is that this site is on production, but I
believe it'll possible to run Tomcat under the profiler and get some
significant results. An alternative is to replicate the system and try
simulating some load and common operations (which I found out JMeter
is the tool for this).
>
> Oh, and don't freak out when you find out how much memory is consumed by
> char[] objects ;)
:)). I had this experience once - it's hard to believe char[] are
so evil! :).
>
> - -chris
best regards,
--
:: Breno Jacinto ::
:: breno - at - gprt.ufpe.br ::
:: FingerPrint ::
2F15 8A61 F566 E442 8581
E3C0 EFF4 E202 74B7 7484
:: Persistir no difícil é a única maneira de torná-lo fácil algum dia. ::
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Breno,
Breno Jacinto wrote:
>> Tomcat itself is extremely stable. This is most likely an application
>> problem. Perhaps your application does well with little traffic but does
>> not scale well. You got a burst of traffic and everything fell apart. ??
>
> Not really. I dont have the exact numbers, but I think we'd get at
> most 50 users every 10 minutes = 5 users / min. I believe that´s
> pretty low.
Is that estimated, or measured?
> I assume the garbage collector cleans them all.
As long as you are putting those search results only into request
attributes or something like that for temporary storage, you should be
okay. Don't forget that a single *huge* result set can still cause an
OOME if you don't have enough memory allocated.
> Yeah, its not crashing. Nor the JVM either. They get very, very
> slow because Linux starts to swap like crazy, after the Tomcat process
> eats almost 100% of RAM.
Gotcha. How much memory do you have configured for the JVM, and how much
memory is actually free? IF you're swapping a lot, maybe you need to a)
buy more memory or b) lower the memory allocation to your app. Of
course, lowering your memory allocation could lead to OOME, but both
problems are effectively the same.
> I'm pretty sure it´s not a burst, but
> something that progressively gets allocated and not deallocated. I'm
> not sure why, but thanks for the idea of the profiler. I'll dig for
> one.
The memory profiler is your best bet. Look at, say, 24 hours worth of GC
activity (if your server stays up that long). You should have a GC curve
that looks like this: /\/\/\/\/\/\/\/\/\/. If if goes up and up and up,
you obviously have a memory leak. If it only goes up at intervals (say,
once an hour), then maybe you have a rare event or timed event that
leaks, rather than something that happens all the time (say, with every
request).
That will give you some idea of what you are looking for.
Oh, and don't freak out when you find out how much memory is consumed by
char[] objects ;)
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHVWqK9CaO5/Lv0PARArG7AJ9yk+QetNqRsJ5ua91WQZqE0b6JEQCfUVdP
ahQyWiVH6KdVwawPThik9yA=
=aYhz
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Breno Jacinto <br...@freeunix.com.br>.
Hi Christopher,
2007/12/3, Christopher Schultz <ch...@christopherschultz.net>:
> You are probably seeing 170 threads, not 170 Tomcat instances.
Yes, you're right. I used instances because thats what tomcat says
when it's shutting down.
> Tomcat itself is extremely stable. This is most likely an application
> problem. Perhaps your application does well with little traffic but does
> not scale well. You got a burst of traffic and everything fell apart. ??
Not really. I dont have the exact numbers, but I think we'd get at
most 50 users every 10 minutes = 5 users / min. I believe that´s
pretty low.
I already had several memory problems with BDB XML - thought I had
hunted them down, but now I start to have the same problems. I don't
think it's related to the fact that searches are returned to
ArrayLists, which are pretty common operations and we get bigger
results with time - I assume the garbage collector cleans them all.
But BDB XML works with bindings for Java, and so there's a lot of
memory allocation issues.
> If Tomcat isn't crashing, then everything should be fine. If you are
> getting OOMEs, then you need to find out what's causing you to run out
> of memory. Several memory profilers are mentioned on this list all the
> time. It's also not a bad idea to have the JVM record a thread dump to
> stdout whenever an OOME occurs. You can find all of this information in
> the list archives or google.
Yeah, its not crashing. Nor the JVM either. They get very, very
slow because Linux starts to swap like crazy, after the Tomcat process
eats almost 100% of RAM. I'm pretty sure it´s not a burst, but
something that progressively gets allocated and not deallocated. I'm
not sure why, but thanks for the idea of the profiler. I'll dig for
one.
> - -chris
best regards,
--
--
:: Breno Jacinto ::
:: breno - at - gprt.ufpe.br ::
:: FingerPrint ::
2F15 8A61 F566 E442 8581
E3C0 EFF4 E202 74B7 7484
:: Persistir no difícil é a única maneira de torná-lo fácil algum dia. ::
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Hundreds of Instances
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Breno,
Breno Jacinto wrote:
> I have an application running under Tomcat for a couple of months
> now, and itś always been memory intensive (I run it in a combination
> with Apache2 + JK + Berkeley DB XML in the backend). But from a few
> days to today, it's been eating memory like crazy, and I can see that
> when I have to restart it very often - 170 Tomcat instances running!
You are probably seeing 170 threads, not 170 Tomcat instances.
> My question is if this is related to Tomcat directly - like a
> memory leak, or if this is application / database specific.
Tomcat itself is extremely stable. This is most likely an application
problem. Perhaps your application does well with little traffic but does
not scale well. You got a burst of traffic and everything fell apart. ??
> How could
> I investigate this further? My setup:
>
> Debian Linux 2.6.18-5-686 #1 SMP
> Apache 2.0.54
> Tomcat 5.5.15
> JDK 1.5.0_06-b05
> BDB XML 2.2.13
> 1.5 GB of RAM
If Tomcat isn't crashing, then everything should be fine. If you are
getting OOMEs, then you need to find out what's causing you to run out
of memory. Several memory profilers are mentioned on this list all the
time. It's also not a bad idea to have the JVM record a thread dump to
stdout whenever an OOME occurs. You can find all of this information in
the list archives or google.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHVFv69CaO5/Lv0PARAusIAKCsGGlEintDCawwXQygze3mpqmLdwCgjsYC
5DYtYAd92gr6nJ5DREoKJvI=
=88yd
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org