You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Monsieur fsfu <mo...@gmail.com> on 2011/07/20 01:32:45 UTC
Tomcat 7 parallel deployment and PermGen Heap Space
Hi,
I was checking out the parallel deployment feature of tomcat 7 and
encountered an issue with PermGen space of JVM. After the initial
deployment (xyz##001.war), PermGen space reaches ~120mb of space and
after parallel deployment (xyz##002.war) it almost doubles (~205mb)
which make sense. Now if i remove the old war file (xyz##001.war) by
removing xyz##001.xml
($catalina_home/conf/Catalina/locahost/xyz##001.xml), PermGen space
doesn't come down. I have waited upto 4 hrs but no luck. Once i
restart the JVM then only it comes down to ~120 range. Is anybody else
faced this issue? I would hate to restart the JVM after deployment
which totally defeats the purpose of parallel deployment.
Setup details:
OS --> RHEL - 5.5 (64 Bit)
Tomcat --> 7.0.14
Java --> jre1.6.0_25 (64 bit)
Tomcat APR --> apr-1.4.5 (64 bit)
java service wrapper --> 3.5.9 (64 bit)
Attached are server.xml and wrapper.conf files.
Thanks in advance
Kapil
Re: Tomcat 7 parallel deployment and PermGen Heap Space
Posted by Konstantin Kolinko <kn...@gmail.com>.
2011/7/20 Monsieur fsfu <mo...@gmail.com>:
> I was checking out the parallel deployment feature of tomcat 7 and
> encountered an issue with PermGen space of JVM. After the initial
> deployment (xyz##001.war), PermGen space reaches ~120mb of space and
> after parallel deployment (xyz##002.war) it almost doubles (~205mb)
> which make sense. Now if i remove the old war file (xyz##001.war) by
> removing xyz##001.xml
> ($catalina_home/conf/Catalina/locahost/xyz##001.xml), PermGen space
> doesn't come down. I have waited upto 4 hrs but no luck.
PermGen usage is caused by classloader of the old webapp that is still
present in memory. There can be many causes of that. You would have to
take a heap dump and analyze it using a tool like MAT from Eclipse
IDE.
You may read
http://wiki.apache.org/tomcat/MemoryLeakProtection
Best regards,
Konstantin Kolinko
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] Tomcat 7 parallel deployment and PermGen Heap Space
Posted by Sylvain Laurent <sy...@m4x.org>.
My personal advice with the Oracle driver is to put it at the server level, not at the webapp level.
For instance if you use the query timeouts, oracle JDBC spawns a Thread to handle the timeouts, and there is no way to properly stop this thread so tomcat will continue complaining about leaking a classloader...
With the driver at the server level, the warnings will disappear and your webapp won't leak (or at least not because of the JDBC driver).
Sylvain
On 20 juil. 2011, at 22:59, Mark Thomas wrote:
> On 20/07/2011 21:46, Christopher Schultz wrote:
>> Mark,
>>
>> On 7/20/2011 4:25 AM, Mark Thomas wrote:
>>> You need to de-register the driver you registered. However, that
>>> won't be causing the leak since Tomcat already did the clean up for
>>> you. Time to get the profiler out.
>>
>> Quick question: I get these JDBC Driver messages from Tomcat when I
>> undeploy, but my code does not register the driver itself...
>
> The code might not, but registration is triggered by the application
> including the JAR. Just putting a JAR with a JDBC driver and the
> appropriate content in META-INF/services is enough to trigger
> registration with the JVM. Tomcat has nothing to do with this.
>
>> Tomcat does
>> via a <Resource> defined in context.xml. Is that driver registered with
>> the WebappClassLoader and therefore considered "registered" by the webapp?
>
> No. It is registered with the DriverManager. It is loaded by the
> WebappClassLoader which triggers the potential memory leak.
>
>> I've been irritated by this behavior for a while because I feel like the
>> container is performing the registration of the driver, but expecting
>> the webapp to perform the de-registration.
>
> The META-INF/services mechanism does not provide automatic
> de-registration so the app has to take care of it.
>
> Mark
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] Tomcat 7 parallel deployment and PermGen Heap Space
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 7/20/2011 4:59 PM, Mark Thomas wrote:
> On 20/07/2011 21:46, Christopher Schultz wrote:
>> Mark,
>>
>> On 7/20/2011 4:25 AM, Mark Thomas wrote:
>>> You need to de-register the driver you registered. However, that
>>> won't be causing the leak since Tomcat already did the clean up
>>> for you. Time to get the profiler out.
>>
>> Quick question: I get these JDBC Driver messages from Tomcat when
>> I undeploy, but my code does not register the driver itself...
>
> The code might not, but registration is triggered by the application
> including the JAR. Just putting a JAR with a JDBC driver and the
> appropriate content in META-INF/services is enough to trigger
> registration with the JVM. Tomcat has nothing to do with this.
I was about to boldly state that I didn't have a JDBC driver in my
webapp's lib directory. Fortunately for me, I triple-checked first to
make sure that wasn't the case and, for some reason, the driver had
snuck into that directory. In my webapp's source, the JAR file was
sitting there participating in the build process, yet not under source
control. I must have done something stupid at some point. Fortunately,
this is dev and not prod :)
So, I have saved myself the embarrassment of stating that my webapp
/absolutely does not/ have a JDBC driver in it, opting for the equally
embarrassing "I should have checked my setup before publicly complaining
about Tomcat's behavior" :)
>> Tomcat does via a <Resource> defined in context.xml. Is that driver
>> registered with the WebappClassLoader and therefore considered
>> "registered" by the webapp?
>
> No. It is registered with the DriverManager. It is loaded by the
> WebappClassLoader which triggers the potential memory leak.
Er, that's what I meant.
>> I've been irritated by this behavior for a while because I feel
>> like the container is performing the registration of the driver,
>> but expecting the webapp to perform the de-registration.
>
> The META-INF/services mechanism does not provide automatic
> de-registration so the app has to take care of it.
I don't use META-INF/services but it's obvious what the problem is, here.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk4toxYACgkQ9CaO5/Lv0PDMpwCgpHgHx1c9APsMLNirnetR7/p+
KEYAnilC8kwpxrJ74ffWtJvlysP+J3C/
=5pO+
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] Tomcat 7 parallel deployment and PermGen Heap Space
Posted by Mark Thomas <ma...@apache.org>.
On 20/07/2011 21:46, Christopher Schultz wrote:
> Mark,
>
> On 7/20/2011 4:25 AM, Mark Thomas wrote:
>> You need to de-register the driver you registered. However, that
>> won't be causing the leak since Tomcat already did the clean up for
>> you. Time to get the profiler out.
>
> Quick question: I get these JDBC Driver messages from Tomcat when I
> undeploy, but my code does not register the driver itself...
The code might not, but registration is triggered by the application
including the JAR. Just putting a JAR with a JDBC driver and the
appropriate content in META-INF/services is enough to trigger
registration with the JVM. Tomcat has nothing to do with this.
> Tomcat does
> via a <Resource> defined in context.xml. Is that driver registered with
> the WebappClassLoader and therefore considered "registered" by the webapp?
No. It is registered with the DriverManager. It is loaded by the
WebappClassLoader which triggers the potential memory leak.
> I've been irritated by this behavior for a while because I feel like the
> container is performing the registration of the driver, but expecting
> the webapp to perform the de-registration.
The META-INF/services mechanism does not provide automatic
de-registration so the app has to take care of it.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [OT] Tomcat 7 parallel deployment and PermGen Heap Space
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Mark,
On 7/20/2011 4:25 AM, Mark Thomas wrote:
> You need to de-register the driver you registered. However, that
> won't be causing the leak since Tomcat already did the clean up for
> you. Time to get the profiler out.
Quick question: I get these JDBC Driver messages from Tomcat when I
undeploy, but my code does not register the driver itself... Tomcat does
via a <Resource> defined in context.xml. Is that driver registered with
the WebappClassLoader and therefore considered "registered" by the webapp?
I've been irritated by this behavior for a while because I feel like the
container is performing the registration of the driver, but expecting
the webapp to perform the de-registration.
- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk4nPr8ACgkQ9CaO5/Lv0PCtawCglncNAXh36PhsbX7RumBEY7UK
8OgAni40dlStbgDSEtskmROivDqPvO0b
=9j/G
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 7 parallel deployment and PermGen Heap Space
Posted by Mark Thomas <ma...@apache.org>.
On 20/07/2011 05:26, Monsieur fsfu wrote:
> @Chuck -- The moment I remove context xml (xyz##001.xml) file, tomcat
> automagically removes the corresponding dir from webapps. So that's
> not an issue.
>
> In my log file i see the following message
>
> INFO | jvm 1 | 2011/07/19 16:11:07 | Jul 19, 2011 4:11:07 PM
> org.apache.catalina.startup.HostConfig checkResources
> INFO | jvm 1 | 2011/07/19 16:11:07 | INFO: Undeploying context [##001]
> INFO | jvm 1 | 2011/07/19 16:11:07 | Jul 19, 2011 4:11:07 PM
> org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
> INFO | jvm 1 | 2011/07/19 16:11:07 | SEVERE: The web application
> [##001] registered the JDBC driver [oracle.jdbc.driver.OracleDriver]
> but failed to unregister it when the web application was stopped. To
> prevent a memory leak, the JDBC Driver has been forcibly unregistered.
>
> Currently i am using commons-dbcp connection pooling. I will switch to
> Tomcat-JDBC connection pool and see if memory leak message goes away
> or not. I will update this thread with my findings.
It won't. You need to de-register the driver you registered. However,
that won't be causing the leak since Tomcat already did the clean up for
you. Time to get the profiler out.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 7 parallel deployment and PermGen Heap Space
Posted by Monsieur fsfu <mo...@gmail.com>.
@Chuck -- The moment I remove context xml (xyz##001.xml) file, tomcat
automagically removes the corresponding dir from webapps. So that's
not an issue.
In my log file i see the following message
INFO | jvm 1 | 2011/07/19 16:11:07 | Jul 19, 2011 4:11:07 PM
org.apache.catalina.startup.HostConfig checkResources
INFO | jvm 1 | 2011/07/19 16:11:07 | INFO: Undeploying context [##001]
INFO | jvm 1 | 2011/07/19 16:11:07 | Jul 19, 2011 4:11:07 PM
org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
INFO | jvm 1 | 2011/07/19 16:11:07 | SEVERE: The web application
[##001] registered the JDBC driver [oracle.jdbc.driver.OracleDriver]
but failed to unregister it when the web application was stopped. To
prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Currently i am using commons-dbcp connection pooling. I will switch to
Tomcat-JDBC connection pool and see if memory leak message goes away
or not. I will update this thread with my findings.
Thanks all you guys with your suggestion...
On Tue, Jul 19, 2011 at 5:33 PM, Caldarale, Charles R
<Ch...@unisys.com> wrote:
>> From: Monsieur fsfu [mailto:monsieurfsfu@gmail.com]
>> Subject: Tomcat 7 parallel deployment and PermGen Heap Space
>
>> Now if i remove the old war file (xyz##001.war) by removing
>> xyz##001.xml ($catalina_home/conf/Catalina/locahost/xyz##001.xml),
>> PermGen space doesn't come down.
>
> First, verify that the first instance really has been undeployed.
>
> Second, unless you insure that at least two full GCs happen after undeploying the first .war file, class unloading will not occur. Even then, it's a bit of guess whether or not the JVM will choose to unload classes in the absence of pressure on PermGen.
>
> Regardless, as others suggested, running some heap analysis tool will find out if you've got a leak built into your webapp, or if the JVM has simply decided not to bother with class unloading yet.
>
> - Chuck
>
>
> THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: Tomcat 7 parallel deployment and PermGen Heap Space
Posted by "Caldarale, Charles R" <Ch...@unisys.com>.
> From: Monsieur fsfu [mailto:monsieurfsfu@gmail.com]
> Subject: Tomcat 7 parallel deployment and PermGen Heap Space
> Now if i remove the old war file (xyz##001.war) by removing
> xyz##001.xml ($catalina_home/conf/Catalina/locahost/xyz##001.xml),
> PermGen space doesn't come down.
First, verify that the first instance really has been undeployed.
Second, unless you insure that at least two full GCs happen after undeploying the first .war file, class unloading will not occur. Even then, it's a bit of guess whether or not the JVM will choose to unload classes in the absence of pressure on PermGen.
Regardless, as others suggested, running some heap analysis tool will find out if you've got a leak built into your webapp, or if the JVM has simply decided not to bother with class unloading yet.
- Chuck
THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 7 parallel deployment and PermGen Heap Space
Posted by Rainer Jung <ra...@kippdata.de>.
On 20.07.2011 01:32, Monsieur fsfu wrote:
> Hi,
>
> I was checking out the parallel deployment feature of tomcat 7 and
> encountered an issue with PermGen space of JVM. After the initial
> deployment (xyz##001.war), PermGen space reaches ~120mb of space and
> after parallel deployment (xyz##002.war) it almost doubles (~205mb)
> which make sense. Now if i remove the old war file (xyz##001.war) by
> removing xyz##001.xml
> ($catalina_home/conf/Catalina/locahost/xyz##001.xml), PermGen space
> doesn't come down. I have waited upto 4 hrs but no luck. Once i
> restart the JVM then only it comes down to ~120 range. Is anybody else
> faced this issue? I would hate to restart the JVM after deployment
> which totally defeats the purpose of parallel deployment.
>
> Setup details:
>
> OS --> RHEL - 5.5 (64 Bit)
> Tomcat --> 7.0.14
> Java --> jre1.6.0_25 (64 bit)
> Tomcat APR --> apr-1.4.5 (64 bit)
> java service wrapper --> 3.5.9 (64 bit)
>
> Attached are server.xml and wrapper.conf files.
Look for the word "leak" in the Tomcat log files after undeploying one
version. Those log lines give you hints, why Tomcat could not unload the
classes.
Unfortunately the solutions can be tricky, so for complex web
applications it is not uncommon, that you migt have to live with those
leaks, in other words provide more perm gen space and still restart
every now and then.
Regards,
Rainer
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org