You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Anurag Kapur <an...@gmail.com> on 2010/10/08 21:15:47 UTC
Tomcat 5.5.25 | Memory leak in Web Application
*Problem Statement*
Memory leak in web application running on Tomcat
*System Information*
Tomcat 5.5.27
Apache HTTPD 2.2.9
Java HotSpot(TM) Server VM (build 1.5.0_14-b03, mixed mode)
Mod_proxy_http connector
Solaris 10 x86
Tomcat JVM heap settings: min:2GB and max:2GB
*Observations and Investigation Done*
Under load, the web application slowly runs out of heap space. The heap
utilization graph suggests a memory leak type pattern.
Heap dumps were gathered to determine the root cause and observations are as
below from the dumps:
1. HashMap entries from the below object reference tree seem to consume
over 80% of the used tenured generation space.
org/apache/jasper/compiler/JspRuntimeContext
↓
Java/util/Collections$SynchronizedMap
↓
Java/util/HashMap
↓
Java/util/HashMap$Entry
1. Heap dumps gathered during different times and after a Full GC always
indicate 100 entries
2. The number of objects referenced by each HashMap entry vary between
2-3
1. These are either a String and JspServletWrapper object
or
1. A String, JspServletWrapper and *another HashMap Entry*. This call
reference tree of HashMap entries referenced by another HashMap entry can
repeat to a depth of approximately 8-10 nodes
The above is indicated in an object reference tree obtained after analyzing
the Heap dump as shown below
1. The maximum percentage of the memory occupied by the HashMap entry
object is by a character array that seems like the JSP servlet response.
HTML response (tags with content) can be seen in the character array
*Questions*
1. I am stuck at this point and have run out of ideas on how to get to
the root cause of this issue. Do you have any ideas/suggestions to help
identify the root cause?
2. I google and found the following interesting links
1. http://www.mail-archive.com/dev@tomcat.apache.org/msg03395.html Is
there any know issues, configuration that causes a memory leak
by caching of
servlet responses in the container and not flushing the cached objects?
2.
http://mail-archives.apache.org/mod_mbox/tomcat-users/200303.mbox/%3C000c01c2ee2c$41c079a0$03b696c0@garethdqw0t9if%3EWe
use a lot of JSTL in our web application. Are there any known issues
around memory issues as indicated on the link?
Any inputs to help identify the root cause of this problem would be highly
appreciated.
Kind Regards
Anurag
-------------------------------------------------------------------
Anurag Kapur
Senior Associate - Technology,
Sapient Corporation.
http://www.linkedin.com/in/anuragkapur
http://www.google.com/profiles/anuragkapur
-------------------------------------------------------------------
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Pid <pi...@pidster.com>.
On 11/10/2010 17:30, Anurag Kapur wrote:
> The objects holding references to the character arrays that ultimately
> consume all the memory are of type
> org.apache.jasper.runtime.BodyContentImpl as indicated in the object
> reference tree below:
>
> There was a bug reported in Tomcat 5.5.9 which says
>
> The problem is that this huge array never gets reset due to the object pooling
> implementation in Jasper (JspFactoryImpl maintains a pool of PageContextImpl
> objects. Each PageContextImpl object maintains an array of BodyContentImpl
> objects), so the memory it consumed is never returned to the heap.
>
> https://issues.apache.org/bugzilla/show_bug.cgi?id=37793
>
> We do not use the suggest JVM argument in our Tomcat 5.5.27 JVM
> -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
>
> Could adding this argument to the Tomcat JVM resolve the problem?
I suspect it would only work if the app has JSPs which were pre-compiled
against 5.5.9, rather than a newer version.
p
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Pid <pi...@pidster.com>.
On 19/10/2010 12:36, Anurag Kapur wrote:
> 1. I am not quite sure if I have the correct answer to your question but I
>> think the most probable reason is that we use tags provided by the CMS to
>> cache the HTML response (JSP caches). The body of these tags can hold large
>> chunks of HTML response. Can this be a suitable explanation to your
>> question?
How are you keying the content in the cache?
Is it possible that old content isn't being removed and that this is the
cause of the memory increase over time?
p
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by André Warnier <aw...@ice-sa.com>.
Anurag,
Anurag Kapur wrote:
...
>>
>> 3. What does the JVM argument actually
>> do? -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
>> I understand it does not turn off tag pooling and instead limits the size
>> of the buffer. Can you please elaborate what this means? What happens when a
>> body tag which exceeds a certain threshold of size?
>>
I don't understand the matter at hand, but when I search Google for
org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER
I am getting a page which seems to describe this.
Maybe it is the same for your other questions ?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Anurag Kapur <an...@gmail.com>.
Hello all,
Can one of you please help answer the last set of queries I have on this
please?
Thanks
Anurag
On Mon, Oct 18, 2010 at 3:02 PM, Anurag Kapur <an...@gmail.com> wrote:
> Thanks Chris/Mark/Charles/Pid for your help with this. The issue has been
> fixed after using the JVM argument :-
>
> -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
>
> I get a healthy heap utilization graph on the same web application under
> similar load test conditions as indicated in the graph in the link below
>
> http://3.bp.blogspot.com/_Oq9OlP-8ap0/TLxLM-dsf_I/AAAAAAAACPk/SolVOyYpAuI/s1600/heap_utilization_post_fix.jpg
>
> The application sustains the load over a 2 hour period without showing any
> signs of degradation as it did earlier.
>
> Before closing this chapter I wanted better clarity on a couple of
> questions:-
>
> >> Also, yes, ours being a content management application, we have large
> >> body tags. We use several tag libraries that come with the CMS
> >> implementation to fetch content from the CMS.
> >
> > Usually CMS applications /manage/ content, rather than hard-coding it.
> > What kind of huge body tags do you have?
>
> 1. I am not quite sure if I have the correct answer to your question but I
> think the most probable reason is that we use tags provided by the CMS to
> cache the HTML response (JSP caches). The body of these tags can hold large
> chunks of HTML response. Can this be a suitable explanation to your
> question?
>
> 2. The large character arrays I saw in the heap dumps had a lot of empty
> elements. For example, the first 500 or so elements in the array were empty
> and then the HTML response was seen in some elements again followed by empty
> elements and then actual HTML content again. This pattern was spread across
> the entire character array. Can these correspond to white spaces? Can this
> be because I do not have *trimSpaces *option enabled as mentioned here:- http://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html#Production
> Configuration<http://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html#Production+Configuration>
> ?
>
> 3. What does the JVM argument actually
> do? -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
> I understand it does not turn off tag pooling and instead limits the size
> of the buffer. Can you please elaborate what this means? What happens when a
> body tag which exceeds a certain threshold of size?
>
> Regards
> Anurag
>
> On Wed, Oct 13, 2010 at 7:16 PM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Anurag,
> >
> > On 10/12/2010 5:47 PM, Anurag Kapur wrote:
> >> I have probably attached an incomplete snapshot of the memory
> >> utilization graph.
> >
> > I'm only looking at what is on your blog. That graph looks good. Perhaps
> > you could update your blog with a graph that illustrates your conclusion.
> >
> >> I have done a few tests with the JVM argument suggested in the bug
> >> report. The initial tests look good.
> >
> > Good.
> >
> >> Also, yes, ours being a content management application, we have large
> >> body tags. We use several tag libraries that come with the CMS
> >> implementation to fetch content from the CMS.
> >
> > Usually CMS applications /manage/ content, rather than hard-coding it.
> > What kind of huge body tags do you have?
> >
> > Good luck,
> > - -chris
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v1.4.10 (MingW32)
> > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> >
> > iEYEARECAAYFAky193MACgkQ9CaO5/Lv0PC8ZwCeOCRg4pGeK7QjTVkeV7tNJUcD
> > GBgAnRO63f4ed/ZRewJcgLC7FwJwLg20
> > =ZC7S
> > -----END PGP SIGNATURE-----
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
> >
>
>
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Anurag Kapur <an...@gmail.com>.
Thanks Chris/Mark/Charles/Pid for your help with this. The issue has been
fixed after using the JVM argument :-
-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
I get a healthy heap utilization graph on the same web application under
similar load test conditions as indicated in the graph in the link below
http://3.bp.blogspot.com/_Oq9OlP-8ap0/TLxLM-dsf_I/AAAAAAAACPk/SolVOyYpAuI/s1600/heap_utilization_post_fix.jpg
The application sustains the load over a 2 hour period without showing any
signs of degradation as it did earlier.
Before closing this chapter I wanted better clarity on a couple of
questions:-
>> Also, yes, ours being a content management application, we have large
>> body tags. We use several tag libraries that come with the CMS
>> implementation to fetch content from the CMS.
>
> Usually CMS applications /manage/ content, rather than hard-coding it.
> What kind of huge body tags do you have?
1. I am not quite sure if I have the correct answer to your question but I
think the most probable reason is that we use tags provided by the CMS to
cache the HTML response (JSP caches). The body of these tags can hold large
chunks of HTML response. Can this be a suitable explanation to your
question?
2. The large character arrays I saw in the heap dumps had a lot of empty
elements. For example, the first 500 or so elements in the array were empty
and then the HTML response was seen in some elements again followed by empty
elements and then actual HTML content again. This pattern was spread across
the entire character array. Can these correspond to white spaces? Can this
be because I do not have *trimSpaces *option enabled as mentioned
here:- http://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html#Production
Configuration?
3. What does the JVM argument actually
do? -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
I understand it does not turn off tag pooling and instead limits the size of
the buffer. Can you please elaborate what this means? What happens when a
body tag which exceeds a certain threshold of size?
Regards
Anurag
On Wed, Oct 13, 2010 at 7:16 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Anurag,
>
> On 10/12/2010 5:47 PM, Anurag Kapur wrote:
>> I have probably attached an incomplete snapshot of the memory
>> utilization graph.
>
> I'm only looking at what is on your blog. That graph looks good. Perhaps
> you could update your blog with a graph that illustrates your conclusion.
>
>> I have done a few tests with the JVM argument suggested in the bug
>> report. The initial tests look good.
>
> Good.
>
>> Also, yes, ours being a content management application, we have large
>> body tags. We use several tag libraries that come with the CMS
>> implementation to fetch content from the CMS.
>
> Usually CMS applications /manage/ content, rather than hard-coding it.
> What kind of huge body tags do you have?
>
> Good luck,
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAky193MACgkQ9CaO5/Lv0PC8ZwCeOCRg4pGeK7QjTVkeV7tNJUcD
> GBgAnRO63f4ed/ZRewJcgLC7FwJwLg20
> =ZC7S
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Anurag,
On 10/12/2010 5:47 PM, Anurag Kapur wrote:
> I have probably attached an incomplete snapshot of the memory
> utilization graph.
I'm only looking at what is on your blog. That graph looks good. Perhaps
you could update your blog with a graph that illustrates your conclusion.
> I have done a few tests with the JVM argument suggested in the bug
> report. The initial tests look good.
Good.
> Also, yes, ours being a content management application, we have large
> body tags. We use several tag libraries that come with the CMS
> implementation to fetch content from the CMS.
Usually CMS applications /manage/ content, rather than hard-coding it.
What kind of huge body tags do you have?
Good luck,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAky193MACgkQ9CaO5/Lv0PC8ZwCeOCRg4pGeK7QjTVkeV7tNJUcD
GBgAnRO63f4ed/ZRewJcgLC7FwJwLg20
=ZC7S
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Anurag Kapur <an...@gmail.com>.
Thanks for your inputs.
I have probably attached an incomplete snapshot of the memory
utilization graph. What happens as more time progresses is that the
utilization keeps increasing and the amount of heap that gets
collected keeps decreasing. After some time the system starts doing
Full GCs every second and the application stops responding. The graph
I had previously attached was from an instance where I simulated the
behaviour in our test env but did not wait for the system to fail
over. I stopped the test since I was seeing the same pattern which
ultimately falls over.
I have done a few tests with the JVM argument suggested in the bug
report. The initial tests look good. The memory utilization seems
healthy. The heap, unlike the snapshot I sent previously, seems to get
collected by pretty much the same amount after each Full GC cycle. So
there is no upward trend in the utilization as seen in the previous
snapshots.
Also, yes, ours being a content management application, we have large
body tags. We use several tag libraries that come with the CMS
implementation to fetch content from the CMS.
I will be doing some more thorough tests tomorrow before reaching a conclusion.
Thank you for all your inputs so far. I will share my final findings
in-case they can be useful to someone in the future with a similar
issue.
On Tue, Oct 12, 2010 at 10:28 PM, Mark Thomas <ma...@apache.org> wrote:
> On 12/10/2010 19:45, Christopher Schultz wrote:
>> markt marked this bug as FIXED, but I see no indication of a resolution.
>> Perhaps that was meant to be WONTFIX?
>
> Nope. I meant FIXED. As in "There is now an option you can use to
> disable this behaviour if you don't like it".
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Mark Thomas <ma...@apache.org>.
On 13/10/2010 19:14, Christopher Schultz wrote:
> Mark,
>
> On 10/12/2010 5:28 PM, Mark Thomas wrote:
>> On 12/10/2010 19:45, Christopher Schultz wrote:
>>> markt marked this bug as FIXED, but I see no indication of a resolution.
>>> Perhaps that was meant to be WONTFIX?
>
>> Nope. I meant FIXED. As in "There is now an option you can use to
>> disable this behaviour if you don't like it".
>
> Gotcha. It wasn't clear from the comment that the fix was somewhat of a
> workaround.
>
> Any reason those buffers never release their memory?
I believe performance.
> Was the complexity
> of the proposed fix deemed too high for the rare cases where this
> problem occurs (that is, shoddy JSPs)?
I believe so but to be honest I haven't looked at the proposed fix for
quite some time.
> Finally, is this same technique
> used in Tomcat 6.x and is the same workaround available?
6.0.x and 7.0.x. Same workaround.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 10/12/2010 5:28 PM, Mark Thomas wrote:
> On 12/10/2010 19:45, Christopher Schultz wrote:
>> markt marked this bug as FIXED, but I see no indication of a resolution.
>> Perhaps that was meant to be WONTFIX?
>
> Nope. I meant FIXED. As in "There is now an option you can use to
> disable this behaviour if you don't like it".
Gotcha. It wasn't clear from the comment that the fix was somewhat of a
workaround.
Any reason those buffers never release their memory? Was the complexity
of the proposed fix deemed too high for the rare cases where this
problem occurs (that is, shoddy JSPs)? Finally, is this same technique
used in Tomcat 6.x and is the same workaround available?
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAky19ugACgkQ9CaO5/Lv0PBRFACgistXA3G55lUJ9hE4Yp43Op/9
CJAAn03E1e/3lxhu6sz3d+/Cj1H+hiP0
=3ZzW
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Mark Thomas <ma...@apache.org>.
On 12/10/2010 19:45, Christopher Schultz wrote:
> markt marked this bug as FIXED, but I see no indication of a resolution.
> Perhaps that was meant to be WONTFIX?
Nope. I meant FIXED. As in "There is now an option you can use to
disable this behaviour if you don't like it".
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Anurag,
On 10/11/2010 12:30 PM, Anurag Kapur wrote:
> I have added my problem statement with Images to by blog here:
> http://anuragkapur-techbytes.blogspot.com/2010/10/tomcat-5527-memory-leak-in-escenic-cms.html
The memory profile you show there appears to have a good GC curve:
tenured generation stabilizes over time, and eden space sees regular
activity that is eventually collected. The process repeats itself as GCs
take place.
I see no problem, here, other than your OOME. I disagree with your
analysis that this heap utilization graph "suggests a memory leak type
pattern".
> The objects holding references to the character arrays that ultimately
> consume all the memory are of type
> org.apache.jasper.runtime.BodyContentImpl as indicated in the object
> reference tree below:
>
> There was a bug reported in Tomcat 5.5.9 which says
>
> The problem is that this huge array never gets reset due to the object pooling
> implementation in Jasper (JspFactoryImpl maintains a pool of PageContextImpl
> objects. Each PageContextImpl object maintains an array of BodyContentImpl
> objects), so the memory it consumed is never returned to the heap.
>
> https://issues.apache.org/bugzilla/show_bug.cgi?id=37793
>
> We do not use the suggest JVM argument in our Tomcat 5.5.27 JVM
> -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
>
> Could adding this argument to the Tomcat JVM resolve the problem?
Certainly you could try it. It appears from this bug report that this
should only be a problem with JSPs where there are large body tags. Are
you using such body tags? If so, then this may help. Otherwise, it
probably will not help.
markt marked this bug as FIXED, but I see no indication of a resolution.
Perhaps that was meant to be WONTFIX?
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAky0rLIACgkQ9CaO5/Lv0PAGfwCeONuTmfs0mew6DLeYCgod5MKu
JcAAn14OemGYLUtcpXWucCszxqVuQfvC
=2G2E
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Anurag Kapur <an...@gmail.com>.
The objects holding references to the character arrays that ultimately
consume all the memory are of type
org.apache.jasper.runtime.BodyContentImpl as indicated in the object
reference tree below:
There was a bug reported in Tomcat 5.5.9 which says
The problem is that this huge array never gets reset due to the object pooling
implementation in Jasper (JspFactoryImpl maintains a pool of PageContextImpl
objects. Each PageContextImpl object maintains an array of BodyContentImpl
objects), so the memory it consumed is never returned to the heap.
https://issues.apache.org/bugzilla/show_bug.cgi?id=37793
We do not use the suggest JVM argument in our Tomcat 5.5.27 JVM
-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true
Could adding this argument to the Tomcat JVM resolve the problem?
-Anurag
On Mon, Oct 11, 2010 at 1:55 PM, Anurag Kapur <an...@gmail.com> wrote:
>
> Thanks Chris and Chuck for your help so far.
> I have added my problem statement with Images to by blog here: http://anuragkapur-techbytes.blogspot.com/2010/10/tomcat-5527-memory-leak-in-escenic-cms.html
>
>
> On Sat, Oct 9, 2010 at 12:13 AM, Caldarale, Charles R <Ch...@unisys.com> wrote
>>
>> > > Attachments are stripped: there is no image to view, here.
>> >
>> > I have attached the heap usage graph as a file this time (heap_usage.jpg)
>>
>> Please read Chris' statement again: attachments are stripped. If you want us to look at the pictures, you'll have to post them in some public location on the web.
>
> Heap utilization is like this:
> http://3.bp.blogspot.com/_Oq9OlP-8ap0/TLMAno5Z2lI/AAAAAAAACPA/pUr70osY3HU/s1600/heap_usage.jpg
> An object call reference tree snapshot that I wanted to show you is here:
> http://2.bp.blogspot.com/_Oq9OlP-8ap0/TLMBRLiUzTI/AAAAAAAACPE/ogO2HI2uYig/s1600/object_reference_tree.jpg
>
>>
>> > Or can the address of the object change in the
>> > lifetime of the tomcat jvm?
>>
>> An object can move on each GC.
>>
> So how should I determine if an object (HashMap entry or character array) that I saw in the first heap dump exists in the next hump dump I obtained after the 4th Full GC?
> Struggling to determine the root cause. Any further pointers would be much appreciated.
> Thanks
> Anurag
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Anurag Kapur <an...@gmail.com>.
Thanks Chris and Chuck for your help so far.
I have added my problem statement with Images to by blog here:
http://anuragkapur-techbytes.blogspot.com/2010/10/tomcat-5527-memory-leak-in-escenic-cms.html
<http://anuragkapur-techbytes.blogspot.com/2010/10/tomcat-5527-memory-leak-in-escenic-cms.html>
On Sat, Oct 9, 2010 at 12:13 AM, Caldarale, Charles R <
Chuck.Caldarale@unisys.com> wrote
>
> > > Attachments are stripped: there is no image to view, here.
> >
> > I have attached the heap usage graph as a file this time (heap_usage.jpg)
>
> Please read Chris' statement again: attachments are stripped. If you want
> us to look at the pictures, you'll have to post them in some public location
> on the web.
Heap utilization is like this:
http://3.bp.blogspot.com/_Oq9OlP-8ap0/TLMAno5Z2lI/AAAAAAAACPA/pUr70osY3HU/s1600/heap_usage.jpg
An object call reference tree snapshot that I wanted to show you is here:
http://2.bp.blogspot.com/_Oq9OlP-8ap0/TLMBRLiUzTI/AAAAAAAACPE/ogO2HI2uYig/s1600/object_reference_tree.jpg
> > Or can the address of the object change in the
> > lifetime of the tomcat jvm?
>
> An object can move on each GC.
>
> So how should I determine if an object (HashMap entry or character array)
that I saw in the first heap dump exists in the next hump dump I obtained
after the 4th Full GC?
Struggling to determine the root cause. Any further pointers would be much
appreciated.
Thanks
Anurag
RE: Tomcat 5.5.25 | Memory leak in Web Application
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Anurag Kapur [mailto:anuragkapur@gmail.com]
> Subject: Re: Tomcat 5.5.25 | Memory leak in Web Application
> > Attachments are stripped: there is no image to view, here.
>
> I have attached the heap usage graph as a file this time (heap_usage.jpg)
Please read Chris' statement again: attachments are stripped. If you want us to look at the pictures, you'll have to post them in some public location on the web.
> Or can the address of the object change in the
> lifetime of the tomcat jvm?
An object can move on each GC.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Anurag Kapur <an...@gmail.com>.
On Fri, Oct 8, 2010 at 11:04 PM, Anurag Kapur <an...@gmail.com> wrote:
> Thanks for your response Chris.
>
> On Fri, Oct 8, 2010 at 10:07 PM, Christopher Schultz <
> chris@christopherschultz.net> wrote:
>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Anurag,
>>
>> On 10/8/2010 3:15 PM, Anurag Kapur wrote:
>> > _Problem Statement_
>> > Memory leak in web application running on Tomcat
>>
>> _Solution Statement_
>>
>> Fix your memory leak
>>
>
> That's the plan :)
>
>
>> > _System Information_
>> >
>> > Tomcat 5.5.27
>>
>> This statement plus the subject line are confusing me.
>>
>
> Apologies. I have made a typo in the subject line. I am using Tomcat 5.5.27
>
> > _Observations and Investigation Done_
>> >
>> > Under load, the web application slowly runs out of heap space. The heap
>> > utilization graph suggests a memory leak type pattern.
>>
>> Attachments are stripped: there is no image to view, here.
>
>
> I have attached the heap usage graph as a file this time (heap_usage.jpg)
>
>
>
> > Heap dumps were gathered to determine the root cause and observations
>> > are as below from the dumps:
>> >
>> > 1. HashMap entries from the below object reference tree seem to
>> > consume over 80% of the used tenured generation space.
>> >
>> > org/apache/jasper/compiler/JspRuntimeContext
>> > Java/util/Collections$SynchronizedMap
>> > Java/util/HashMap
>> > Java/util/HashMap$Entry
>>
>> If this doesn't grow, then it doesn't sound like a memory leak.
>>
>
> The number of entries under the HashMap referenced by JspRuntimeContent ->
> Collections$SynchronizedMap does not increase. However the size of these
> 100 entries increases with time. In my last test, after the first full GC
> the size of this HashMap was around 350MB. With time and under load, this
> increased. Next heap dump gathered after the 4th Full GC showed that the
> size of this hash map had doubled to around 730MB. The number of entries
> immediately under this HashMap however remained constant.
>
>
>>
>> > 2. Heap dumps gathered during different times and after a Full GC
>> > always indicate 100 entries
>>
>> This indicates zero growth, so I would guess no memory leak.
>>
>
> Yes, the number of immediate entries in the Hashmap remained constant
> however the size these occupy in the heap increased.
>
>
>>
>> > 3. The number of objects referenced by each HashMap entry vary
>> > between 2-3
>> > 1. These are either a String and JspServletWrapper object
>> >
>> > or
>> >
>> > 2. A String, JspServletWrapper and _another HashMap Entry_.
>> > This call reference tree of HashMap entries referenced by
>> > another HashMap entry can repeat to a depth of approximately
>> > 8-10 nodes
>>
>> That sounds weird.
>>
>
> To make this a little clear, I have attached a screen shot of the object
> reference tree from IBM Heap analyzer I used on the heap dumps obtained.
> (object_reference_tree.jpg)
>
>
>>
>> > The above is indicated in an object reference tree obtained after
>> > analyzing the Heap dump as shown below
>> >
>> > 4. The maximum percentage of the memory occupied by the HashMap entry
>> > object is by a character array that seems like the JSP servlet
>> > response. HTML response (tags with content) can be seen in the
>> > character array
>>
>> Does this response text stay in memory indefinitely? When you say it's
>> the response, is it the actual response to a particular client, or is it
>> just static text from the JSP source? If the latter, this is completely
>> expected: the JSP compiles-in all static text
>
>
> I am having difficulties at this stage determining this with 100%
> confidence. I am mostly seeing static HTML tags in the character array that
> would be coming from the static JSP source. However I have also seen some
> references to dynamic content that is sent back to the client in these
> charcter arrays. There is atleast 1 such character array in each of these
> 100 has map entries. Sometime more in cases where there are more hash map
> entries within an entry. I have not seen the size of all these charcter
> arrays but a few are as big as 7MB.
>
> .
>>
>> > _Questions_
>> >
>> > 1. I am stuck at this point and have run out of ideas on how to get
>> > to the root cause of this issue. Do you have any ideas/suggestions
>> > to help identify the root cause?
>>
>> Finding the roots to those references should help. For instance, what
>> page is holding on to all those entries? It's possible that the
>> JSPRuntimeContext requires these data even after the compiler has
>> compiled the code.
>>
>
> I can also see the name of the custom JSP files in the object call
> reference trees under most of these 100 hash map entries. However some of
> them are relatively simple and I am unable to think of a reason why such a
> JSP would hold reference in the heap causing the leak. As you suggested, I
> am now trying to search for objects I see in the heap dump obtained after
> the first Full GC in the dump obtained after the 4th Full GC.
>
>
I did a quick check on the heap dumps. I noted the address of the character
array referenced by the largest entry in the hash map from the first dump. I
then searched for this address in the second dump. However could not find
it. I used the IBM Heap Analyzer for running this search. Hence chances of a
manual error are ruled it. Is this sufficient evidence to conclude that the
character array I saw in the first heap dump was garbage collected and hence
its address was not found in the second dump? Or can the address of the
object change in the lifetime of the tomcat jvm?
> Thanks for your inputs so far. Any other inputs while I do the
> investigation highlighted above?
>
> > 2. I google and found the following interesting links
>> > 1.
>> http://www.mail-archive.com/dev@tomcat.apache.org/msg03395.html
>> > Is there any know issues, configuration that causes a memory
>> > leak by caching of servlet responses in the container and
>> > not flushing the cached objects?
>>
>> The above seems to be a complaint about how JSPs work: they are
>> translated into .java files, then compiled and loaded. These loaded
>> class files contain lots of static strings. There's nothing to be done
>> about this.
>>
>> > 2.
>> http://mail-archives.apache.org/mod_mbox/tomcat-users/200303.mbox/%3C000c01c2ee2c$41c079a0$03b696c0@garethdqw0t9if%3E
>> > We use a lot of JSTL in our web application. Are there any
>> > known issues around memory issues as indicated on the link?
>>
>> This one is seriously old: 2003 was a long time ago (even older than
>> your version of Tomcat). I don't think that post is relevant anymore.
>>
>> You should consider upgrading. Check the changelog
>> (http://tomcat.apache.org/tomcat-5.5-doc/changelog.html) for changes to
>> Jasper between your version (5.5.25 or 5.5.27) and the latest (5.5.30).
>> You also might consider upgrading to Tomcat 6.0.
>>
>> - -chris
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.10 (MingW32)
>> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>>
>> iEYEARECAAYFAkyviC8ACgkQ9CaO5/Lv0PCHQQCeKH/X4Ee4nN1Z3x19wCNkWW3Q
>> pf0An2e+6trJtZezARIhx3GXifLUVpWc
>> =Id6K
>> -----END PGP SIGNATURE-----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
>>
>>
>
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Anurag Kapur <an...@gmail.com>.
Thanks for your response Chris.
On Fri, Oct 8, 2010 at 10:07 PM, Christopher Schultz <
chris@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Anurag,
>
> On 10/8/2010 3:15 PM, Anurag Kapur wrote:
> > _Problem Statement_
> > Memory leak in web application running on Tomcat
>
> _Solution Statement_
>
> Fix your memory leak
>
That's the plan :)
> > _System Information_
> >
> > Tomcat 5.5.27
>
> This statement plus the subject line are confusing me.
>
Apologies. I have made a typo in the subject line. I am using Tomcat 5.5.27
> _Observations and Investigation Done_
> >
> > Under load, the web application slowly runs out of heap space. The heap
> > utilization graph suggests a memory leak type pattern.
>
> Attachments are stripped: there is no image to view, here.
I have attached the heap usage graph as a file this time (heap_usage.jpg)
> Heap dumps were gathered to determine the root cause and observations
> > are as below from the dumps:
> >
> > 1. HashMap entries from the below object reference tree seem to
> > consume over 80% of the used tenured generation space.
> >
> > org/apache/jasper/compiler/JspRuntimeContext
> > Java/util/Collections$SynchronizedMap
> > Java/util/HashMap
> > Java/util/HashMap$Entry
>
> If this doesn't grow, then it doesn't sound like a memory leak.
>
The number of entries under the HashMap referenced by JspRuntimeContent ->
Collections$SynchronizedMap does not increase. However the size of these
100 entries increases with time. In my last test, after the first full GC
the size of this HashMap was around 350MB. With time and under load, this
increased. Next heap dump gathered after the 4th Full GC showed that the
size of this hash map had doubled to around 730MB. The number of entries
immediately under this HashMap however remained constant.
>
> > 2. Heap dumps gathered during different times and after a Full GC
> > always indicate 100 entries
>
> This indicates zero growth, so I would guess no memory leak.
>
Yes, the number of immediate entries in the Hashmap remained constant
however the size these occupy in the heap increased.
>
> > 3. The number of objects referenced by each HashMap entry vary
> > between 2-3
> > 1. These are either a String and JspServletWrapper object
> >
> > or
> >
> > 2. A String, JspServletWrapper and _another HashMap Entry_.
> > This call reference tree of HashMap entries referenced by
> > another HashMap entry can repeat to a depth of approximately
> > 8-10 nodes
>
> That sounds weird.
>
To make this a little clear, I have attached a screen shot of the object
reference tree from IBM Heap analyzer I used on the heap dumps obtained.
(object_reference_tree.jpg)
>
> > The above is indicated in an object reference tree obtained after
> > analyzing the Heap dump as shown below
> >
> > 4. The maximum percentage of the memory occupied by the HashMap entry
> > object is by a character array that seems like the JSP servlet
> > response. HTML response (tags with content) can be seen in the
> > character array
>
> Does this response text stay in memory indefinitely? When you say it's
> the response, is it the actual response to a particular client, or is it
> just static text from the JSP source? If the latter, this is completely
> expected: the JSP compiles-in all static text
I am having difficulties at this stage determining this with 100%
confidence. I am mostly seeing static HTML tags in the character array that
would be coming from the static JSP source. However I have also seen some
references to dynamic content that is sent back to the client in these
charcter arrays. There is atleast 1 such character array in each of these
100 has map entries. Sometime more in cases where there are more hash map
entries within an entry. I have not seen the size of all these charcter
arrays but a few are as big as 7MB.
.
>
> > _Questions_
> >
> > 1. I am stuck at this point and have run out of ideas on how to get
> > to the root cause of this issue. Do you have any ideas/suggestions
> > to help identify the root cause?
>
> Finding the roots to those references should help. For instance, what
> page is holding on to all those entries? It's possible that the
> JSPRuntimeContext requires these data even after the compiler has
> compiled the code.
>
I can also see the name of the custom JSP files in the object call reference
trees under most of these 100 hash map entries. However some of them are
relatively simple and I am unable to think of a reason why such a JSP would
hold reference in the heap causing the leak. As you suggested, I am now
trying to search for objects I see in the heap dump obtained after the first
Full GC in the dump obtained after the 4th Full GC.
Thanks for your inputs so far. Any other inputs while I do the investigation
highlighted above?
> 2. I google and found the following interesting links
> > 1.
> http://www.mail-archive.com/dev@tomcat.apache.org/msg03395.html
> > Is there any know issues, configuration that causes a memory
> > leak by caching of servlet responses in the container and
> > not flushing the cached objects?
>
> The above seems to be a complaint about how JSPs work: they are
> translated into .java files, then compiled and loaded. These loaded
> class files contain lots of static strings. There's nothing to be done
> about this.
>
> > 2.
> http://mail-archives.apache.org/mod_mbox/tomcat-users/200303.mbox/%3C000c01c2ee2c$41c079a0$03b696c0@garethdqw0t9if%3E
> > We use a lot of JSTL in our web application. Are there any
> > known issues around memory issues as indicated on the link?
>
> This one is seriously old: 2003 was a long time ago (even older than
> your version of Tomcat). I don't think that post is relevant anymore.
>
> You should consider upgrading. Check the changelog
> (http://tomcat.apache.org/tomcat-5.5-doc/changelog.html) for changes to
> Jasper between your version (5.5.25 or 5.5.27) and the latest (5.5.30).
> You also might consider upgrading to Tomcat 6.0.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAkyviC8ACgkQ9CaO5/Lv0PCHQQCeKH/X4Ee4nN1Z3x19wCNkWW3Q
> pf0An2e+6trJtZezARIhx3GXifLUVpWc
> =Id6K
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat 5.5.25 | Memory leak in Web Application
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Anurag,
On 10/8/2010 3:15 PM, Anurag Kapur wrote:
> _Problem Statement_
> Memory leak in web application running on Tomcat
_Solution Statement_
Fix your memory leak
> _System Information_
>
> Tomcat 5.5.27
This statement plus the subject line are confusing me.
> _Observations and Investigation Done_
>
> Under load, the web application slowly runs out of heap space. The heap
> utilization graph suggests a memory leak type pattern.
Attachments are stripped: there is no image to view, here.
> Heap dumps were gathered to determine the root cause and observations
> are as below from the dumps:
>
> 1. HashMap entries from the below object reference tree seem to
> consume over 80% of the used tenured generation space.
>
> org/apache/jasper/compiler/JspRuntimeContext
> Java/util/Collections$SynchronizedMap
> Java/util/HashMap
> Java/util/HashMap$Entry
If this doesn't grow, then it doesn't sound like a memory leak.
> 2. Heap dumps gathered during different times and after a Full GC
> always indicate 100 entries
This indicates zero growth, so I would guess no memory leak.
> 3. The number of objects referenced by each HashMap entry vary
> between 2-3
> 1. These are either a String and JspServletWrapper object
>
> or
>
> 2. A String, JspServletWrapper and _another HashMap Entry_.
> This call reference tree of HashMap entries referenced by
> another HashMap entry can repeat to a depth of approximately
> 8-10 nodes
That sounds weird.
> The above is indicated in an object reference tree obtained after
> analyzing the Heap dump as shown below
>
> 4. The maximum percentage of the memory occupied by the HashMap entry
> object is by a character array that seems like the JSP servlet
> response. HTML response (tags with content) can be seen in the
> character array
Does this response text stay in memory indefinitely? When you say it's
the response, is it the actual response to a particular client, or is it
just static text from the JSP source? If the latter, this is completely
expected: the JSP compiles-in all static text.
> _Questions_
>
> 1. I am stuck at this point and have run out of ideas on how to get
> to the root cause of this issue. Do you have any ideas/suggestions
> to help identify the root cause?
Finding the roots to those references should help. For instance, what
page is holding on to all those entries? It's possible that the
JSPRuntimeContext requires these data even after the compiler has
compiled the code.
> 2. I google and found the following interesting links
> 1. http://www.mail-archive.com/dev@tomcat.apache.org/msg03395.html
> Is there any know issues, configuration that causes a memory
> leak by caching of servlet responses in the container and
> not flushing the cached objects?
The above seems to be a complaint about how JSPs work: they are
translated into .java files, then compiled and loaded. These loaded
class files contain lots of static strings. There's nothing to be done
about this.
> 2. http://mail-archives.apache.org/mod_mbox/tomcat-users/200303.mbox/%3C000c01c2ee2c$41c079a0$03b696c0@garethdqw0t9if%3E
> We use a lot of JSTL in our web application. Are there any
> known issues around memory issues as indicated on the link?
This one is seriously old: 2003 was a long time ago (even older than
your version of Tomcat). I don't think that post is relevant anymore.
You should consider upgrading. Check the changelog
(http://tomcat.apache.org/tomcat-5.5-doc/changelog.html) for changes to
Jasper between your version (5.5.25 or 5.5.27) and the latest (5.5.30).
You also might consider upgrading to Tomcat 6.0.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkyviC8ACgkQ9CaO5/Lv0PCHQQCeKH/X4Ee4nN1Z3x19wCNkWW3Q
pf0An2e+6trJtZezARIhx3GXifLUVpWc
=Id6K
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org