help/advice needed on reference chasing

[Daniel John Debrunner <dj...@apache.org>]

For DERBY-538 I have code to use a class loader, for the application
jars installed in a database, based upon the standard Java
URLClassLoader. The advantages to this are the standard security
features of URLClassLoader are picked up such as signature checking and
Jar sealing.

However, I hit a couple of test failures where a restore from backup
fails because the old database cannot be removed because an open file
cannot be deleted on Windows.

Basically it is a reflection of/related to this Java bug where
URLClassLoader keeps the jar files that it is using open.

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4950148

The code that fails is from backupRestore.sql
-----------------------------------------------------------------------
call sqlj.install_jar('extin/brtestjar.jar', 'aggjar', 0);
call SYSCS_UTIL.SYSCS_SET_DATABASE_PROPERTY('derby.database.classpath',
'APP.aggjar');
create function dv(P1 INT) RETURNS INT NO SQL external name
'dbytesting.CodeInAJar.doubleMe' language java parameter style java;
select cast (dv(x) as dec(5,2)) from x;

-- empty statrement cache
-- call ec();

----take a backup.
call SYSCS_UTIL.SYSCS_BACKUP_DATABASE('extinout/mybackup');
disconnect;
connect 'wombat;shutdown=true';

---restore a databases
connect 'wombat;restoreFrom=extinout/mybackup/wombat';
-----------------------------------------------------------------------

The last connect to restore the database is failing.

I can do either of these two things to make the restore work:

  1) don't execute the select with the user's function that is using
code from the aggjar jar file

  2) Uncomment the call to the ec() procedure that empties the statement
cache.

Thus the compiled plan is obviously holding onto a reference to the
generated class that in turn has a reference to the
dbytesting.CodeInAJar class that was loaded by the URLClassLoader. This
URLClassLoader has the open file that is the aggjar jar file.

But I can't figure out what is holding onto the statement cache and/or
the compiled plan when the test is failing, I have so far done all of:

    - null out the reference to the statement cache in the language
factory at database shutdown
    - null out references to the URLCLassLoader in the ClassFactory  at
database shutdown
    - checked ij is performing a close on the connection
    - checked the background raw store daemon thread is going away  at
database shutdown
    - executed extra simple queries after the select to ensure ij is not
hanging onto the last ResultSet obtained
    - Looped 100 times on database shutdown, calling System.gc and
System.runFinalization with a 40ms sleep on each iteration

I do see the URLClassLoader being finalized eventually (too late,
several statements later in the script) and with the empty the statement
cache, the URLClassLoader is finalized at database shutdown as one would
expect.

Is there any tooling builtin to the JVM to show reference chains, so I
can see at shutdown who is holding onto the generated plan?

Thanks!
Dan.

Re: help/advice needed on reference chasing

[John Embretsen <Jo...@Sun.COM>]

Bryan Pendleton wrote:

>> Is there any tooling builtin to the JVM to show reference chains, so I
>> can see at shutdown who is holding onto the generated plan?
> 
> Perhaps this can help? https://hat.dev.java.net/doc/README.html
> 
> bryan
> 

It depends on which jvm you are using. The tool referenced above, HAT (Heap 
Analysis Tool), is now obsolete, although you might be able to get the 
information you need from it. You can create a binary heap dump using hprof 
(included in JDK1.5, see 
http://java.sun.com/developer/technicalArticles/Programming/HPROF.html) and 
browse it using HAT or jhat (included with Mustang/JDK1.6).

Mustang also includes numerous other great improvements in this regard. You 
may want to check out the OQL (Object Query Language):

http://blogs.sun.com/roller/page/sundararajan?entry=querying_java_heap_with_oql

Snmipped example:

Show referents that are not referred by another object. i.e., the referent is 
reachable only by that soft reference:

     select f.referent from java.lang.ref.SoftReference f
     where f.referent != null && referrers(f.referent).length == 1


Hope it helps...


-- 
John

Re: help/advice needed on reference chasing

[Bryan Pendleton <bp...@amberpoint.com>]

> Is there any tooling builtin to the JVM to show reference chains, so I
> can see at shutdown who is holding onto the generated plan?

Perhaps this can help? https://hat.dev.java.net/doc/README.html

bryan