You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Andreas Knecht <an...@atlassian.com> on 2007/04/10 02:43:40 UTC
OutOfMemory problem on Tomcat 5.5.20 and Java 1.5
Hi all,
I'm one of the developers working on JIRA
<http://www.atlassian.com/software/jira>. We've currently got a
problem, where Tomcat seems to be running out of heap space, when
running JIRAs functional tests. Here are all the relevant system
properties:
*JDK: *java version "1.5.0_09"
Java(TM) 2 Runtime Environment, Standard Edition (build
1.5.0_09-b01)
Java HotSpot(TM) Server VM (build 1.5.0_09-b01, mixed mode)
*Tomcat:* v.5.5.20
*JVM args:* -Xmx256m -XX:+HeapDumpOnOutOfMemoryError
The functional tests simply run a number of HTTP requests to the
web-server, simulating form submissions, button clicks, URL requests,
etc (and obviously assert that the correct text is shown in the response).
About a week ago, this test started failing with OutOfMemory errors. I
realize that increasing the maximum memory will probably get rid of this
problem, but we've never had problems running the func tests with 256MB
in the past, and we want to make sure we haven't introduced a memory
leak. We examined the heapdump produced, and it looks like a large
amount of the memory is retained by the
org.apache.jasper.compiler.JspRuntimeContext (138 MB to be exact).
I've attached a screenshot of the JspRuntimeContext in YourKit. It seems
that the 'jsps' SynchronizedMap is taking up all the space. The problem
does NOT occur on Java 1.6.
Any ideas, if it is normal behavior for the JspRuntimeContext to take up
this much memory? I'm happy to make more information available if
needed. The heap dump is about 65MB in size when zipped.
Cheers,
Andreas
Re: OutOfMemory problem on Tomcat 5.5.20 and Java 1.5
Posted by Mark Thomas <ma...@apache.org>.
Mark Thomas wrote:
> Andreas Knecht wrote:
>> Hi all,
>>
>> I'm one of the developers working on JIRA
>
> This really belongs on the users list.
Sorry got confused in my mail client. My bad.
Mark
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: OutOfMemory problem on Tomcat 5.5.20 and Java 1.5
Posted by Mark Thomas <ma...@apache.org>.
Andreas Knecht wrote:
> Hi all,
>
> I'm one of the developers working on JIRA
This really belongs on the users list. Also, attachments tend to get
stripped out so you might want to consider providing a url to the
screenshot.
Mark
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: OutOfMemory problem on Tomcat 5.5.20 and Java 1.5
Posted by Andreas Knecht <an...@atlassian.com>.
Hi Mark,
Thanks for your input. Tomcat is started up via cargo with all its
default options. This is how we've always run our functional tests.
It appears that by adding new functionality (read: more JSPs, etc) we've
simply reached a threshold now where we need to run the functional tests
with more memory.
I'm also a little baffled why this doesn't occur on JVM 1.6 however. I
verified that we also use -Xmx256m on the 1.6 JVM. (all the settings
are exactly the same and the server is also started up by Cargo. The
only difference is the version of the JVM)
I suppose the good thing about this is that, while analyzing these
heapdumps we did find some memory inefficiencies in JIRA however.
Especially to do with small leaks of the VelocityEngine (about 90Mb are
taken up by the velocity engine) by JIRA code. Initially we thought
this was normal, as velocity also caches the rendered velocity
templates, however it looks like in very rare circumstance we're leaking
the entire VelocityEngine object. We're not 100% sure yet how this
happens, but we're investigating it at the moment.
We had a look at http://issues.apache.org/bugzilla/show_bug.cgi?id=37793
with Jeff Turner, but it doesn't seem that this is the major cause of
the problem.
Thanks again for your help.
Cheers,
Andreas
Mark Thomas wrote:
> Andreas Knecht wrote:
>
>> Just a quick addition. The memory dump is now available here:
>> http://repository.atlassian.com/temp/java_pid27558.hprof.gz
>>
>> Cheers,
>> Andreas
>>
>>
> You appear to be running Tomcat in development mode.
> (http://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html)
>
> This means that source of every JSP is held in memory to provide
> detailed messages in the event of an error. If you have large JSPs
> this will hurt. It looks like this accounts for 50MB+
>
> The compiled JSPs account for 80MB+. I am guess there is a lot of text
> in some of these JSPs.
>
> This doesn't explain why you don't see the error on a 1.6 jvm. Are you
> sure you are still using 256MB of RAM with the 1.6 jvm?
>
> 27% of your memory is used by char[]. Have you read
> http://issues.apache.org/bugzilla/show_bug.cgi?id=37793 ? It might
> offer some insight.
>
> HTH,
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: OutOfMemory problem on Tomcat 5.5.20 and Java 1.5
Posted by Mark Thomas <ma...@apache.org>.
Andreas Knecht wrote:
> Just a quick addition. The memory dump is now available here:
> http://repository.atlassian.com/temp/java_pid27558.hprof.gz
>
> Cheers,
> Andreas
>
You appear to be running Tomcat in development mode.
(http://tomcat.apache.org/tomcat-5.5-doc/jasper-howto.html)
This means that source of every JSP is held in memory to provide
detailed messages in the event of an error. If you have large JSPs
this will hurt. It looks like this accounts for 50MB+
The compiled JSPs account for 80MB+. I am guess there is a lot of text
in some of these JSPs.
This doesn't explain why you don't see the error on a 1.6 jvm. Are you
sure you are still using 256MB of RAM with the 1.6 jvm?
27% of your memory is used by char[]. Have you read
http://issues.apache.org/bugzilla/show_bug.cgi?id=37793 ? It might
offer some insight.
HTH,
Mark
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: OutOfMemory problem on Tomcat 5.5.20 and Java 1.5
Posted by Andreas Knecht <an...@atlassian.com>.
Just a quick addition. The memory dump is now available here:
http://repository.atlassian.com/temp/java_pid27558.hprof.gz
Cheers,
Andreas
Mark Thomas wrote:
> Andreas Knecht wrote:
>
>> About a week ago, this test started failing with OutOfMemory errors.
>>
>
> Obvious question. What changed before it started failing?
>
>
>> I've attached a screenshot of the JspRuntimeContext in YourKit.
>>
>
> The attachement didn't come through. Can you make it availalbe via a URL?
>
>
>> The problem does NOT occur on Java 1.6.
>>
>
> Hmm. That points more to a jvm issue rather than a Tomcat one although
> there isn't enough info to rule out Tomcat entirely at this point. Can
> you try with the latest 1.5 jvm?
>
>
>> Any ideas, if it is normal behavior for the JspRuntimeContext to take up
>> this much memory? I'm happy to make more information available if
>> needed. The heap dump is about 65MB in size when zipped.
>>
>
> 138MB seems rather high. The screenshot would help here.
>
> Mark
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: OutOfMemory problem on Tomcat 5.5.20 and Java 1.5
Posted by Andreas Knecht <an...@atlassian.com>.
Hi Mark,
Sorry about the attachment, I wasn't aware that they get stripped out
when e-mailing this list. I've opened an issue on
http://jira.atlassian.com now for this:
http://jira.atlassian.com/browse/JRA-12524
Mark Thomas wrote:
> Andreas Knecht wrote:
>
>> About a week ago, this test started failing with OutOfMemory errors.
>>
>
> Obvious question. What changed before it started failing?
>
Difficult to say, as this is happening on our continuous integration and
the version being built is the 'bleeding edge' of JIRA (i.e. the latest
version from the source repository). However, we've had a look through
the memory held by JIRA classes in the heapdump, and there doesn't seem
to be anything out of the ordinary. I will however go through most of
the commits since this issue started occurring and try to see if we
introduced anything problematic.
>
>> I've attached a screenshot of the JspRuntimeContext in YourKit.
>>
>
> The attachement didn't come through. Can you make it availalbe via a URL?
>
Done. It's availabel here: http://jira.atlassian.com/browse/JRA-12524
>
>> The problem does NOT occur on Java 1.6.
>>
>
> Hmm. That points more to a jvm issue rather than a Tomcat one although
> there isn't enough info to rule out Tomcat entirely at this point. Can
> you try with the latest 1.5 jvm?
>
I'll give that a go.
>
>> Any ideas, if it is normal behavior for the JspRuntimeContext to take up
>> this much memory? I'm happy to make more information available if
>> needed. The heap dump is about 65MB in size when zipped.
>>
>
> 138MB seems rather high. The screenshot would help here.
>
> Mark
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@tomcat.apache.org
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
>
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: OutOfMemory problem on Tomcat 5.5.20 and Java 1.5
Posted by Mark Thomas <ma...@apache.org>.
Andreas Knecht wrote:
> About a week ago, this test started failing with OutOfMemory errors.
Obvious question. What changed before it started failing?
> I've attached a screenshot of the JspRuntimeContext in YourKit.
The attachement didn't come through. Can you make it availalbe via a URL?
> The problem does NOT occur on Java 1.6.
Hmm. That points more to a jvm issue rather than a Tomcat one although
there isn't enough info to rule out Tomcat entirely at this point. Can
you try with the latest 1.5 jvm?
> Any ideas, if it is normal behavior for the JspRuntimeContext to take up
> this much memory? I'm happy to make more information available if
> needed. The heap dump is about 65MB in size when zipped.
138MB seems rather high. The screenshot would help here.
Mark
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org