Project Jigsaw, Apache Tomcat and RMI related memory leaks

[Mark Thomas <ma...@apache.org>]

Hi,

The Apache Tomcat community was asked by Rory O'Donnell for feedback on
JDK 9 + Project Jigsaw. Having provided that feedback we were directed
here so I have reproduced that feedback below.


I've started testing Tomcat trunk with JDK 9 + Project Jigsaw and it
looks like we are going to hit a bunch of problems related to Tomcat's
memory leak protection.

The short version is there are lots of JDK calls that can result in a
reference being retained to the current class loader. If that class
loader is the web application class loader it often ends up being pinned
in memory. This triggers a memory leak when the web application is
reloaded since the web application class loader is not eligible for GC.

Tomcat generally uses reflection to find these problems. It then does
one of two things:
- If the JRE provides an API the application developer should have used
to clean up the reference, Tomcat does this for them and then logs a
very loud error message telling the developer they need to fix their app.
- If there is nothing the developer could have done to avoid the
problem, Tomcat cleans it up. Usually this is via reflection again.

Reporting this second class of issues as JRE bugs has been on my TODO
list for a long time. It looks like Java 9 is going to bump this to the
top of the list.

The first problem we have hit is related to RMI. The memory leak is
triggered by a call to:
java.rmi.registry.Registry.bind() or
java.rmi.registry.Registry.rebind()

The problem occurs when an object of a class loaded by the web
application class loader is bound to the RMI registry.

There is no standard API (that I have found) that completely removes all
references. In particular
java.rmi.registry.Registry.unbind()
doesn't help.

The code Tomcat uses to clean this up is at [1]. Essentially, we need to
remove any reference to the web application's class loader from the RMI
caches.

With JDK 9 this fails with:
java.lang.reflect.InaccessibleObjectException: Unable to make member of
class sun.rmi.transport.Target accessible:  module java.rmi does not
export sun.rmi.transport to unnamed module ...

I took a look at the JDK 9 API but I could not find a way to bypass
this.

Possible solutions include:
1. Some way that allows us to continue to use reflection as per the
current code.

2. A new method somewhere in the RMI API that clears all references
associated with a given class loader from the cache.

3. Modify Registry.unbind() so it removes all references.

4. Something else...

I do have a concern with 3 on its own that, while that would allow
applications to clear their own references, it would mean Tomcat would
have no way to check for the problem.

Ideally, I'd like to see the API extended so a) applications are able to
clean up after themselves and b) Tomcat can check for leaked references
and generate error messages for the ones found.

Any and all suggestions gratefully received.

Thanks,

Mark

[1]
http://svn.apache.org/viewvc/tomcat/trunk/java/org/apache/catalina/loader/WebappClassLoaderBase.java?view=annotate#l2214


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org

Re: Project Jigsaw, Apache Tomcat and RMI related memory leaks

[Mark Thomas <ma...@apache.org>]

On 18/04/2016 12:14, Alan Bateman wrote:
> On 18/04/2016 10:18, Mark Thomas wrote:
>> Hi,
>>
>> The Apache Tomcat community was asked by Rory O'Donnell for feedback on
>> JDK 9 + Project Jigsaw. Having provided that feedback we were directed
>> here so I have reproduced that feedback below.
>>
>>
>> I've started testing Tomcat trunk with JDK 9 + Project Jigsaw and it
>> looks like we are going to hit a bunch of problems related to Tomcat's
>> memory leak protection.
>>
>> The short version is there are lots of JDK calls that can result in a
>> reference being retained to the current class loader. If that class
>> loader is the web application class loader it often ends up being pinned
>> in memory. This triggers a memory leak when the web application is
>> reloaded since the web application class loader is not eligible for GC.
>>
>> Tomcat generally uses reflection to find these problems. It then does
>> one of two things:
>> - If the JRE provides an API the application developer should have used
>> to clean up the reference, Tomcat does this for them and then logs a
>> very loud error message telling the developer they need to fix their app.
>> - If there is nothing the developer could have done to avoid the
>> problem, Tomcat cleans it up. Usually this is via reflection again.
>>
>> Reporting this second class of issues as JRE bugs has been on my TODO
>> list for a long time. It looks like Java 9 is going to bump this to the
>> top of the list.
> I remember you brought this up on core-libs-dev [1] and the suggestion
> at the time was to create bugs for any of the issues that are still
> relevant. So yes, bumping the priority on that would be good.
> 
> On the RMI issue then it would be good to start a thread specifically on
> this topic on core-libs-dev.

OK. I'll restart this thread there. I'm also reviewing the various other
leaks and I'll start new threads as each one is confirmed.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@tomcat.apache.org
For additional commands, e-mail: dev-help@tomcat.apache.org