You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-user@db.apache.org by Alan Burlison <Al...@sun.com> on 2007/09/24 14:40:28 UTC
Derby causes PermGen leaks in Tomcat
I have a webapp running under Tomcat with Derby embedded inside it, and
after a number of redeploys Tomcat falls over with the dreaded PermGen
space error. I'm running Derby in server mode as I want to be able to
connect to it externally whilst the webapp is running. The webapp has
startup/shutdown hooks to start and stop Derby, but the 'stop' part
obviously isn't working. I'm doing a DriverManager.getConnection() with
shutdown=true, followed by NetworkServerControl.shutdown(). After this
I can't connect to the database (as expected), but jmap shows a load of
Derby objects are still loaded, and jstack shows several instances of this:
"derby.antiGC" daemon prio=1 tid=0x088e7000 nid=0xd in Object.wait()
[0xc64ce000..0xc64ceb60]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0xcaff88f0> (a
org.apache.derby.impl.services.monitor.AntiGC)
at java.lang.Object.wait(Object.java:485)
at org.apache.derby.impl.services.monitor.AntiGC.run(Unknown
Source)
- locked <0xcaff88f0> (a
org.apache.derby.impl.services.monitor.AntiGC)
at java.lang.Thread.run(Thread.java:619)
So it's clear that although the Derby server is stopped, it's still
zombified somewhere inside the Tomcat JVM. How do I cleanly and
completely remove Derby from a running JVM?
--
Alan Burlison
--
Re: Derby causes PermGen leaks in Tomcat
Posted by Alan Burlison <Al...@sun.com>.
Alan Burlison wrote:
>> However, when I do:
>> - load embedded driver (with newInstance())
>> - start the network server using the NetworkServerControl API
>> - shutdown Derby system (example URL: "jdbc:derby:;shutdown=true")
>> - shutdown server (netServerControl.shutdown())
>>
>> I don't see the AntiGC thread (using jstack).
>
> Ahah! I'm doing method 1 above, not method 2. I think you've hit the
> nail on the head, I need to shut down the system, not just the
> individual database before shutting down the server.
I've added the code you suggested and now antiGC dies, but undeploying
my app still doesn't clean it out of memory, so something else is
obviously still wrong.
--
Alan Burlison
--
Re: Derby causes PermGen leaks in Tomcat
Posted by John Embretsen <Jo...@Sun.COM>.
Alan Burlison wrote:
> John Embretsen wrote:
>
>> When I run a small test case (outside of Tomcat) I see the AntiGC
>> thread hanging around after doing:
>> - load embedded driver (with newInstance())
>
> Which class has the newInstance() method?
In my simple test case, I load the driver by doing
Class.forName("org.apache.derby.jdbc.EmbeddedDriver").newInstance();
(I'm not able to reload the driver after system shutdown without using
newInstance())
--
John
Re: Derby causes PermGen leaks in Tomcat
Posted by Alan Burlison <Al...@sun.com>.
John Embretsen wrote:
> I haven't seen any details of how you unload the embedded driver,
DriverManager.deregisterDriver(driver) where driver is obtained from
Class.forName.
> but
> have you tried shutting down the Derby system explicitly as well as (or
> instead of) shutting down the particular database?
>
> When I run a small test case (outside of Tomcat) I see the AntiGC thread
> hanging around after doing:
> - load embedded driver (with newInstance())
Which class has the newInstance() method?
> - start the network server using the NetworkServerControl API
> - shutdown database (example URL: "jdbc:derby:leakDB;shutdown=true")
> - shutdown server (netServerControl.shutdown())
>
> However, when I do:
> - load embedded driver (with newInstance())
> - start the network server using the NetworkServerControl API
> - shutdown Derby system (example URL: "jdbc:derby:;shutdown=true")
> - shutdown server (netServerControl.shutdown())
>
> I don't see the AntiGC thread (using jstack).
Ahah! I'm doing method 1 above, not method 2. I think you've hit the
nail on the head, I need to shut down the system, not just the
individual database before shutting down the server.
> PermGen space usage seems to be fairly stable at around 11-12 MB even
> after several hundred iterations of either of the scenarios mentioned
> above, so I'm not able to reproduce your case outside Tomcat (please
> post if you are!). I suspect Tomcat may be a part of the problem here...
I suspect you are right - as far as I know Tomcat uses a custom
classloader for each webapp, and PermGen leaks in Tomcat are a
well-known 'feature'.
--
Alan Burlison
--
Re: Derby causes PermGen leaks in Tomcat
Posted by John Embretsen <Jo...@Sun.COM>.
Alan Burlison wrote:
>> Derby objects are still loaded, and jstack shows several instances of
>> this:
>>
>> "derby.antiGC" daemon prio=1 tid=0x088e7000 nid=0xd in Object.wait()
>> So it's clear that although the Derby server is stopped, it's still
>> zombified somewhere inside the Tomcat JVM. How do I cleanly and
>> completely remove Derby from a running JVM?
>
> I found this thread that looked relevant:
> http://www.nabble.com/RE%3A-AntiGC-Not-Shutting-Down-p1217286.html
>
> "Shutting down the databases before shutting down the server has solved
> all of my problems."
>
> I've modified my code to do the same, and I've even tried explicitly
> unloading the Derby JDBC driver, but I still get an instance of AntiGC
> hanging around, although all the other Derby threads do terminate.
I haven't seen any details of how you unload the embedded driver, but have you
tried shutting down the Derby system explicitly as well as (or instead of)
shutting down the particular database?
When I run a small test case (outside of Tomcat) I see the AntiGC thread hanging
around after doing:
- load embedded driver (with newInstance())
- start the network server using the NetworkServerControl API
- shutdown database (example URL: "jdbc:derby:leakDB;shutdown=true")
- shutdown server (netServerControl.shutdown())
However, when I do:
- load embedded driver (with newInstance())
- start the network server using the NetworkServerControl API
- shutdown Derby system (example URL: "jdbc:derby:;shutdown=true")
- shutdown server (netServerControl.shutdown())
I don't see the AntiGC thread (using jstack).
PermGen space usage seems to be fairly stable at around 11-12 MB even after
several hundred iterations of either of the scenarios mentioned above, so I'm
not able to reproduce your case outside Tomcat (please post if you are!). I
suspect Tomcat may be a part of the problem here...
--
John
Re: Derby causes PermGen leaks in Tomcat
Posted by Alan Burlison <Al...@sun.com>.
> Derby objects are still loaded, and jstack shows several instances of this:
>
> "derby.antiGC" daemon prio=1 tid=0x088e7000 nid=0xd in Object.wait()
> [0xc64ce000..0xc64ceb60]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0xcaff88f0> (a
> org.apache.derby.impl.services.monitor.AntiGC)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.derby.impl.services.monitor.AntiGC.run(Unknown
> Source)
> - locked <0xcaff88f0> (a
> org.apache.derby.impl.services.monitor.AntiGC)
> at java.lang.Thread.run(Thread.java:619)
>
> So it's clear that although the Derby server is stopped, it's still
> zombified somewhere inside the Tomcat JVM. How do I cleanly and
> completely remove Derby from a running JVM?
I found this thread that looked relevant:
http://www.nabble.com/RE%3A-AntiGC-Not-Shutting-Down-p1217286.html
"Shutting down the databases before shutting down the server has solved
all of my problems."
I've modified my code to do the same, and I've even tried explicitly
unloading the Derby JDBC driver, but I still get an instance of AntiGC
hanging around, although all the other Derby threads do terminate.
I'm using the embedded driver to access the database from other threads
within the JVM that Derby is running in, and running the network server
to provide external access to the DB.
--
Alan Burlison
--
Re: Derby causes PermGen leaks in Tomcat
Posted by Alan Burlison <Al...@sun.com>.
Kathey Marsden wrote:
>> So it's clear that although the Derby server is stopped, it's still
>> zombified somewhere inside the Tomcat JVM. How do I cleanly and
>> completely remove Derby from a running JVM?
>>
> Is it possible you are seeing DERBY-2905? If so please try out the patch
> for that issue which is currently posted for review.
> https://issues.apache.org/jira/browse/DERBY-2905
Although I haven't rebuilt Derby with this patch included, I'm 95%
certain that DERBY-2905 is indeed the cause. Although I'd solved the
AntiGC thread problem, I still couldn't redeploy more than 15-16 times.
I added code to the servlet so that when it was shut down it manually
unloaded all the JDBC drivers (including the AutoloadedDriver), and now
I can redeploy 50 times without a problem. That's basically the same os
the proposed fix for DERBY-2905.
--
Alan Burlison
--
Re: Derby causes PermGen leaks in Tomcat
Posted by Alan Burlison <Al...@sun.com>.
Alan Burlison wrote:
> Also, I've put code in that manually unregisters the EmbeddedDriver, but
> I'm a bit puzzled because it doesn't seem to be loaded in the first
> place - I've added code to list the loaded drivers befor and after the
> unload, and here is what I see:
>
> Unloading org.apache.derby.jdbc.EmbeddedDriver@1e91259
> Before
> sun.jdbc.odbc.JdbcOdbcDriver@35e6e3
> org.apache.derby.jdbc.AutoloadedDriver@1a2760f
> org.apache.derby.jdbc.ClientDriver@1cd2b82
> After
> sun.jdbc.odbc.JdbcOdbcDriver@35e6e3
> org.apache.derby.jdbc.AutoloadedDriver@1a2760f
> org.apache.derby.jdbc.ClientDriver@1cd2b82
>
> It does seem to point to the AutoloadedDriver as being the problem
> though...
I've added code to deregister all the Derby drivers on shutdown, I
iterate over the loaded drivers and call DriverManager.deregisterDriver
on any drivers that are in the org.apache.derby.jdbc package:
Before
sun.jdbc.odbc.JdbcOdbcDriver@185572a
org.apache.derby.jdbc.ClientDriver@5aebd9
org.apache.derby.jdbc.AutoloadedDriver@113981b
After
sun.jdbc.odbc.JdbcOdbcDriver@185572a
However, jstack *still* shows an instance of derby.antiGC is running, so
although DERBY-2905 may be a factor, I think there is another problem
as well.
--
Alan Burlison
--
Re: Derby causes PermGen leaks in Tomcat
Posted by Alan Burlison <Al...@sun.com>.
Kathey Marsden wrote:
> Information on how to build Derby is at:
>
> http://db.apache.org/derby/dev/derby_source.html
>
> It is not too hard to setup.
>
> I use Eclipse to apply patches and have had less luck with the patch
> utility.
Thanks, I'll take a look.
--
Alan Burlison
--
Re: Derby causes PermGen leaks in Tomcat
Posted by Kathey Marsden <km...@sbcglobal.net>.
Alan Burlison wrote:
>
> How do I test the patch? I've no idea how to build Derby, I'm just
> using the standard 10.3.1.4 release Jars.
>
Information on how to build Derby is at:
http://db.apache.org/derby/dev/derby_source.html
It is not too hard to setup.
I use Eclipse to apply patches and have had less luck with the patch
utility.
Kathey
Re: Derby causes PermGen leaks in Tomcat
Posted by Alan Burlison <Al...@sun.com>.
Kathey Marsden wrote:
> Alan Burlison wrote:
>>
>> So it's clear that although the Derby server is stopped, it's still
>> zombified somewhere inside the Tomcat JVM. How do I cleanly and
>> completely remove Derby from a running JVM?
>>
> Is it possible you are seeing DERBY-2905? If so please try out the patch
> for that issue which is currently posted for review.
> https://issues.apache.org/jira/browse/DERBY-2905
It's entirely possible, in fact I'd say it is very likely. Poking
around a heap dump with jhat shows that the AutoloadedDriver is still
resident after undeploying the application.
How do I test the patch? I've no idea how to build Derby, I'm just
using the standard 10.3.1.4 release Jars.
Also, I've put code in that manually unregisters the EmbeddedDriver, but
I'm a bit puzzled because it doesn't seem to be loaded in the first
place - I've added code to list the loaded drivers befor and after the
unload, and here is what I see:
Unloading org.apache.derby.jdbc.EmbeddedDriver@1e91259
Before
sun.jdbc.odbc.JdbcOdbcDriver@35e6e3
org.apache.derby.jdbc.AutoloadedDriver@1a2760f
org.apache.derby.jdbc.ClientDriver@1cd2b82
After
sun.jdbc.odbc.JdbcOdbcDriver@35e6e3
org.apache.derby.jdbc.AutoloadedDriver@1a2760f
org.apache.derby.jdbc.ClientDriver@1cd2b82
It does seem to point to the AutoloadedDriver as being the problem though...
--
Alan Burlison
--
Re: Derby causes PermGen leaks in Tomcat
Posted by Kathey Marsden <km...@sbcglobal.net>.
Alan Burlison wrote:
>
> So it's clear that although the Derby server is stopped, it's still
> zombified somewhere inside the Tomcat JVM. How do I cleanly and
> completely remove Derby from a running JVM?
>
Is it possible you are seeing DERBY-2905? If so please try out the patch
for that issue which is currently posted for review.
https://issues.apache.org/jira/browse/DERBY-2905
Thanks
Kathey