You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@river.apache.org by Peter Firmstone <ji...@zeus.net.au> on 2010/04/11 03:24:58 UTC
ClassLoader and Class Garbage Collection issues with Serialization.
I have stumbled across a troubling problem with Serialization relating
to Garbage Collection of Classes and ClassLoaders and was hoping someone
might be able to shed some light on the issue.
Is it really true that the more objects you distribute, the greater your
memory consumption because Class files and ClassLoaders cannot be
garbage collected?
Regards,
Peter.
The issue can be found here:
http://www.ibm.com/developerworks/java/library/j-dclp3/index.html
And here's the relevant information, pasted from the link:
Problems related to garbage collection and serialization
The garbage collector interacts closely with the class loader. Among
other things, the collector examines the class loader data structures to
determine which classes are /live/ -- that is, are not garbage
collectable. This can often lead to some unexpected problems.
Figure 2 illustrates a situation where serialization affects the garbage
collection (GC) of classes and a class loader in an unexpected way:
*Figure 2. Serialization example*
Serialization example
In this example, |SerializationTest| instantiates a |URLClassLoader|,
called |loader|. After loading |SerializationClass|, the class loader is
dereferenced. The expectation is that this will allow the classes loaded
by it to be garbage collected. The code for these classes is illustrated
in Listings 9 and 10:
*Listing 9. SerializationTest.java*
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
public class SerializationTest extends ClassLoader {
public static void main(String args[]) {
try {
URLClassLoader loader = new URLClassLoader(new URL[] { new URL(
"file://C:/CL_Article/Serialization/dir1/") });
System.out.println("Loading SerializationClass");
Class c = loader.loadClass("SerializationClass");
System.out.println("Creating an instance of SerializationClass");
c.newInstance();
System.out.println("Dereferencing the class loader");
c = null;
loader = null;
System.out.println("Running GC...");
System.gc();
System.out.println("Triggering a Javadump");
com.ibm.jvm.Dump.JavaDump();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (InstantiationException e) {
e.printStackTrace();
} catch (IllegalAccessException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
}
*Listing 10. SerializationClass.java*
import java.io.File;
import java.io.FileOutputStream;
import java.io.ObjectOutputStream;
import java.io.Serializable;
public class SerializationClass implements Serializable {
private static final long serialVersionUID = 5024741671582526226L;
public SerializationClass() {
try {
File file = new File("C:/CL_Article/Serialization/test.txt");
FileOutputStream fos = new FileOutputStream(file);
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(this);
oos.reset();
oos.close();
fos.close();
oos = null;
fos = null;
file = null;
} catch (Exception e) {
e.printStackTrace();
}
}
}
Using a Javadump, it is possible to discover whether the class loader
has been garbage collected. (See the first article in this series for
more on using Javadump.) If the following section appears in the list of
class loaders, then it has not been collected:
------a- Loader java/net/URLClassLoader(0x44DC6DE0), Shadow 0x00ADB6D8,
Parent sun/misc/Launcher$AppClassLoader(0x00ADB7B0)
Number of loaded classes 1
Number of cached classes 11
Allocation used for loaded classes 1
Package owner 0x00ADB6D8
Though dereferencing a user-defined class loader seems like a way to
ensure that the classes are garbage collected, this is not actually the
case. In the previous example, the problem stems from the use of
|java.io.ObjectOutputStream.writeObject(Object obj)| and its
implications on GC.
When |writeObject()| is invoked (to serialize |SerializationClass|), a
reference to this class object is passed internally to
|ObjectStreamClass| and stored in a lookup table (that is, in an
internal cache). This reference is kept to speed up future serialization
of the same class.
When the class loader is dereferenced, the classes that it loaded are
not garbage collectable. This is because there is a live reference to
the |SerializationClass| class from the |ObjectStreamClass| lookup
table. |ObjectStreamClass| is a primordial class and therefore is never
garbage collected. The lookup table is referenced from a static field in
|ObjectStreamClass| and is kept in the class itself rather than in an
instance of it. As a result, the reference to |SerializationClass|
exists for the lifetime of the JVM, and the class thus cannot be garbage
collected. Importantly, the |SerializationClass| class has a reference
to its defining class loader, and so it cannot be completely
dereferenced either.
To avoid this problem, any classes that are to be serialized should be
loaded by a class loader that does not need to be garbage collected --
by the system class loader, for example.
Re: ClassLoader and Class Garbage Collection issues with Serialization.
Posted by Peter Firmstone <ji...@zeus.net.au>.
Awesome, thank you very much Peter, you've saved me countless hours. &
Thanks for the test example.
So in reality if I implement a stream based lookup implementation that
returns results, ordered by their packages versions, such that
Classloaders are shared for common code 1. Which would control
ClassLoading of different proxies, such that those with commonality are
dealt with first and discarded by the client, leaving garbage collection
to clean up, clients might run out of memory? The goal is to enable
lookups to be performed over the internet proper, enabling massive
result sets to be returned as a stream for inspection.
Could DGC be relied upon for Remote Method Invocation to prevent used
classes from being garbage collected, if ObjectStreamClass was utilising
weak references?
Serialization alone, without Remote Method Invocation and DGC, would
have this problem of class files and their compute expensive descriptors
becoming garbage collected too often with weak references.
In my case, certainly on the CDC platform if I utilise standard Java
Serialization, the client risks running out of memory?
Are they any jvm options that can be used to increase the memory reserve
that triggers garbage collection of soft referents?
Regards,
Peter.
1. This has implications for trust, since typically Permission is
assigned to a PermissionDomain -> ClassLoader -> Codebase, if several
services shared the same Codebase, I would have to rely on Principals
and Codebase Signers for trust.
Trust would rely on:
* Permissions Granted based on codebase signer (Upper bound of trust
for the client).
* ProxyTrustVerifier - does the Service trust the proxy (Is this the
Services proxy instance)?
* Principal: the Client can authenticate itself with the Service node.
* Principal: the Service node can authenticate itself with the
client (How can a services Permissions be restricted based on
identity when several services share the same codebase at the client)?
Peter Jones wrote:
> [Everything I say below is about Sun's JDK/JRE implementation, although I believe that IBM's inherits the same serialization code.]
>
> Since the fix for 5056445/6232010, in JDK 6 and 5.0u7, the static class descriptor cache in java.io.ObjectStreamClass does not maintain strong references to serializable classes that have been used. A potential confusion, though, is that it does maintain "soft" reachability to them, which is stronger than "weak" (see the java.lang.ref package doc), and which means that such classes (and thus their defining class loaders) might only be garbage collected if there is sufficient pressure on the JVM heap that their storage is needed to satisfy other allocations. In other words, the collector merely noticing that the classes are not strongly reachable would not not sufficient for them to be collected.
>
> See below an edit of that SerializationTest.java, updated to use a reference queue to monitor the garbage collection of the class in a non-IBM-specific way. It should pass, with JDK 6 and 5.0u7, but note the invocation of flushSoftRefs necessary to force sufficient heap pressure to cause soft references to be cleared-- System.gc alone is not sufficient.
>
> Some background: The ObjectStreamClass implementation faces a dilemma about whether to maintain soft or weak references to the cached class descriptors (which in turn reference their corresponding classes). It could use weak references instead, but then cache entries would likely be cleared very soon after a given serialization activity has completed, going against the purpose of caching these expensive-to-compute descriptors. What it would really like to do is make its referencing of a class descriptor conditional on the reachability of the corresponding class itself, but that is not doable with the current java.lang.ref API. Doing that sort of thing is the subject of this RFE (which is probably my personal favorite open Java libraries RFE):
>
> http://bugs.sun.com/view_bug.do?bug_id=4630118
>
> Even before the 5056445/6232010 fix, though, the situation wasn't as bad as suggested by the article you quoted. The serializable classes were indeed strongly referenced by the static cache, as keys, but the corresponding class descriptor values were (again) softly referenced, and the cache would clear out entries whose soft references had been cleared. The problem was that this clearing did not happen in the background, rather it required some cache usage to occur, so garbage collection of unused serializable classes required ongoing serialization activity. Also, this clearing was not in a closed loop with the garbage collector, thus risking OutOfMemoryError in tight situations.
>
> -- Peter
>
>
> On Apr 10, 2010, at 9:24 PM, Peter Firmstone wrote:
>
>
>> I have stumbled across a troubling problem with Serialization relating to Garbage Collection of Classes and ClassLoaders and was hoping someone might be able to shed some light on the issue.
>>
>> Is it really true that the more objects you distribute, the greater your memory consumption because Class files and ClassLoaders cannot be garbage collected?
>>
>> Regards,
>>
>> Peter.
>>
>> The issue can be found here:
>> http://www.ibm.com/developerworks/java/library/j-dclp3/index.html
>>
>> And here's the relevant information, pasted from the link:
>>
>>
>> Problems related to garbage collection and serialization
>>
>> The garbage collector interacts closely with the class loader. Among other things, the collector examines the class loader data structures to determine which classes are /live/ -- that is, are not garbage collectable. This can often lead to some unexpected problems.
>>
>> Figure 2 illustrates a situation where serialization affects the garbage collection (GC) of classes and a class loader in an unexpected way:
>>
>>
>> *Figure 2. Serialization example*
>>
>> Serialization example
>>
>> In this example, |SerializationTest| instantiates a |URLClassLoader|, called |loader|. After loading |SerializationClass|, the class loader is dereferenced. The expectation is that this will allow the classes loaded by it to be garbage collected. The code for these classes is illustrated in Listings 9 and 10:
>>
>>
>> *Listing 9. SerializationTest.java*
>>
>> import java.net.MalformedURLException;
>> import java.net.URL;
>> import java.net.URLClassLoader;
>>
>> public class SerializationTest extends ClassLoader {
>>
>> public static void main(String args[]) {
>> try {
>> URLClassLoader loader = new URLClassLoader(new URL[] { new URL(
>> "file://C:/CL_Article/Serialization/dir1/") });
>> System.out.println("Loading SerializationClass");
>> Class c = loader.loadClass("SerializationClass");
>> System.out.println("Creating an instance of SerializationClass");
>> c.newInstance();
>> System.out.println("Dereferencing the class loader");
>> c = null;
>> loader = null;
>> System.out.println("Running GC...");
>> System.gc();
>> System.out.println("Triggering a Javadump");
>> com.ibm.jvm.Dump.JavaDump();
>> } catch (MalformedURLException e) {
>> e.printStackTrace();
>> } catch (InstantiationException e) {
>> e.printStackTrace();
>> } catch (IllegalAccessException e) {
>> e.printStackTrace();
>> } catch (ClassNotFoundException e) {
>> e.printStackTrace();
>> }
>> }
>> }
>>
>>
>>
>> *Listing 10. SerializationClass.java*
>>
>> import java.io.File;
>> import java.io.FileOutputStream;
>> import java.io.ObjectOutputStream;
>> import java.io.Serializable;
>>
>> public class SerializationClass implements Serializable {
>>
>> private static final long serialVersionUID = 5024741671582526226L;
>>
>> public SerializationClass() {
>> try {
>> File file = new File("C:/CL_Article/Serialization/test.txt");
>> FileOutputStream fos = new FileOutputStream(file);
>> ObjectOutputStream oos = new ObjectOutputStream(fos);
>> oos.writeObject(this);
>> oos.reset();
>> oos.close();
>> fos.close();
>> oos = null;
>> fos = null;
>> file = null;
>> } catch (Exception e) {
>> e.printStackTrace();
>> }
>> }
>> }
>>
>>
>> Using a Javadump, it is possible to discover whether the class loader has been garbage collected. (See the first article in this series for more on using Javadump.) If the following section appears in the list of class loaders, then it has not been collected:
>>
>> ------a- Loader java/net/URLClassLoader(0x44DC6DE0), Shadow 0x00ADB6D8,
>> Parent sun/misc/Launcher$AppClassLoader(0x00ADB7B0) Number of loaded classes 1 Number of cached classes 11 Allocation used for loaded classes 1 Package owner 0x00ADB6D8
>>
>>
>> Though dereferencing a user-defined class loader seems like a way to ensure that the classes are garbage collected, this is not actually the case. In the previous example, the problem stems from the use of |java.io.ObjectOutputStream.writeObject(Object obj)| and its implications on GC.
>>
>> When |writeObject()| is invoked (to serialize |SerializationClass|), a reference to this class object is passed internally to |ObjectStreamClass| and stored in a lookup table (that is, in an internal cache). This reference is kept to speed up future serialization of the same class.
>>
>> When the class loader is dereferenced, the classes that it loaded are not garbage collectable. This is because there is a live reference to the |SerializationClass| class from the |ObjectStreamClass| lookup table. |ObjectStreamClass| is a primordial class and therefore is never garbage collected. The lookup table is referenced from a static field in |ObjectStreamClass| and is kept in the class itself rather than in an instance of it. As a result, the reference to |SerializationClass| exists for the lifetime of the JVM, and the class thus cannot be garbage collected. Importantly, the |SerializationClass| class has a reference to its defining class loader, and so it cannot be completely dereferenced either.
>>
>> To avoid this problem, any classes that are to be serialized should be loaded by a class loader that does not need to be garbage collected -- by the system class loader, for example.
>>
>
>
> import java.lang.ref.Reference;
> import java.lang.ref.ReferenceQueue;
> import java.lang.ref.WeakReference;
> import java.net.MalformedURLException;
> import java.net.URL;
> import java.net.URLClassLoader;
> import java.util.ArrayList;
> import java.util.List;
>
> public class SerializationTest extends ClassLoader {
>
> public static void main(String args[]) throws Exception {
> URLClassLoader loader = new URLClassLoader(new URL[] {
> new URL("file:dir1/")
> });
> System.out.println("Loading SerializationClass");
> Class c = loader.loadClass("SerializationClass");
> System.out.println("Creating an instance of SerializationClass");
> c.newInstance();
>
> ReferenceQueue<Class<?>> queue = new ReferenceQueue<Class<?>>();
> Reference<Class<?>> ref = new WeakReference<Class<?>>(c, queue);
>
> System.out.println("Dereferencing the class loader");
> c = null;
> loader = null;
> System.out.println("Running GC...");
> System.gc();
> //System.out.println("Triggering a Javadump");
> //com.ibm.jvm.Dump.JavaDump();
> flushSoftRefs();
>
> Reference<? extends Class<?>> dequeued = queue.remove(1000);
> if (dequeued == ref) {
> System.out.println("SerializationClass garbage collected");
> } else {
> throw new Error("SerializationClass not garbage collected");
> }
> }
>
> private static void flushSoftRefs() {
> try {
> List<Object> l = new ArrayList<Object>();
> while (true) l.add(new byte[10000]);
> } catch (OutOfMemoryError e) {
> System.out.println("memory exhausted");
> }
> }
> }
>
>
>
Re: ClassLoader and Class Garbage Collection issues with Serialization.
Posted by Gregg Wonderly <gr...@wonderly.org>.
Peter Jones wrote:
> Doing that sort of thing is the subject of this RFE (which is probably
> my personal favorite open Java libraries RFE):
I know I've had a vote on 4630118 for quite some time, and I just moved another
open vote to that as well. Now it has 7 :-( Maybe we can excite some
additional interest in it, and perhaps even getting Doug Lea and others on the
concurrency interest list to get more interested would be helpful.
Gregg Wonderly
Re: ClassLoader and Class Garbage Collection issues with Serialization.
Posted by Peter Jones <pc...@roundroom.net>.
[Everything I say below is about Sun's JDK/JRE implementation, although I believe that IBM's inherits the same serialization code.]
Since the fix for 5056445/6232010, in JDK 6 and 5.0u7, the static class descriptor cache in java.io.ObjectStreamClass does not maintain strong references to serializable classes that have been used. A potential confusion, though, is that it does maintain "soft" reachability to them, which is stronger than "weak" (see the java.lang.ref package doc), and which means that such classes (and thus their defining class loaders) might only be garbage collected if there is sufficient pressure on the JVM heap that their storage is needed to satisfy other allocations. In other words, the collector merely noticing that the classes are not strongly reachable would not not sufficient for them to be collected.
See below an edit of that SerializationTest.java, updated to use a reference queue to monitor the garbage collection of the class in a non-IBM-specific way. It should pass, with JDK 6 and 5.0u7, but note the invocation of flushSoftRefs necessary to force sufficient heap pressure to cause soft references to be cleared-- System.gc alone is not sufficient.
Some background: The ObjectStreamClass implementation faces a dilemma about whether to maintain soft or weak references to the cached class descriptors (which in turn reference their corresponding classes). It could use weak references instead, but then cache entries would likely be cleared very soon after a given serialization activity has completed, going against the purpose of caching these expensive-to-compute descriptors. What it would really like to do is make its referencing of a class descriptor conditional on the reachability of the corresponding class itself, but that is not doable with the current java.lang.ref API. Doing that sort of thing is the subject of this RFE (which is probably my personal favorite open Java libraries RFE):
http://bugs.sun.com/view_bug.do?bug_id=4630118
Even before the 5056445/6232010 fix, though, the situation wasn't as bad as suggested by the article you quoted. The serializable classes were indeed strongly referenced by the static cache, as keys, but the corresponding class descriptor values were (again) softly referenced, and the cache would clear out entries whose soft references had been cleared. The problem was that this clearing did not happen in the background, rather it required some cache usage to occur, so garbage collection of unused serializable classes required ongoing serialization activity. Also, this clearing was not in a closed loop with the garbage collector, thus risking OutOfMemoryError in tight situations.
-- Peter
On Apr 10, 2010, at 9:24 PM, Peter Firmstone wrote:
> I have stumbled across a troubling problem with Serialization relating to Garbage Collection of Classes and ClassLoaders and was hoping someone might be able to shed some light on the issue.
>
> Is it really true that the more objects you distribute, the greater your memory consumption because Class files and ClassLoaders cannot be garbage collected?
>
> Regards,
>
> Peter.
>
> The issue can be found here:
> http://www.ibm.com/developerworks/java/library/j-dclp3/index.html
>
> And here's the relevant information, pasted from the link:
>
>
> Problems related to garbage collection and serialization
>
> The garbage collector interacts closely with the class loader. Among other things, the collector examines the class loader data structures to determine which classes are /live/ -- that is, are not garbage collectable. This can often lead to some unexpected problems.
>
> Figure 2 illustrates a situation where serialization affects the garbage collection (GC) of classes and a class loader in an unexpected way:
>
>
> *Figure 2. Serialization example*
>
> Serialization example
>
> In this example, |SerializationTest| instantiates a |URLClassLoader|, called |loader|. After loading |SerializationClass|, the class loader is dereferenced. The expectation is that this will allow the classes loaded by it to be garbage collected. The code for these classes is illustrated in Listings 9 and 10:
>
>
> *Listing 9. SerializationTest.java*
>
> import java.net.MalformedURLException;
> import java.net.URL;
> import java.net.URLClassLoader;
>
> public class SerializationTest extends ClassLoader {
>
> public static void main(String args[]) {
> try {
> URLClassLoader loader = new URLClassLoader(new URL[] { new URL(
> "file://C:/CL_Article/Serialization/dir1/") });
> System.out.println("Loading SerializationClass");
> Class c = loader.loadClass("SerializationClass");
> System.out.println("Creating an instance of SerializationClass");
> c.newInstance();
> System.out.println("Dereferencing the class loader");
> c = null;
> loader = null;
> System.out.println("Running GC...");
> System.gc();
> System.out.println("Triggering a Javadump");
> com.ibm.jvm.Dump.JavaDump();
> } catch (MalformedURLException e) {
> e.printStackTrace();
> } catch (InstantiationException e) {
> e.printStackTrace();
> } catch (IllegalAccessException e) {
> e.printStackTrace();
> } catch (ClassNotFoundException e) {
> e.printStackTrace();
> }
> }
> }
>
>
>
> *Listing 10. SerializationClass.java*
>
> import java.io.File;
> import java.io.FileOutputStream;
> import java.io.ObjectOutputStream;
> import java.io.Serializable;
>
> public class SerializationClass implements Serializable {
>
> private static final long serialVersionUID = 5024741671582526226L;
>
> public SerializationClass() {
> try {
> File file = new File("C:/CL_Article/Serialization/test.txt");
> FileOutputStream fos = new FileOutputStream(file);
> ObjectOutputStream oos = new ObjectOutputStream(fos);
> oos.writeObject(this);
> oos.reset();
> oos.close();
> fos.close();
> oos = null;
> fos = null;
> file = null;
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> }
>
>
> Using a Javadump, it is possible to discover whether the class loader has been garbage collected. (See the first article in this series for more on using Javadump.) If the following section appears in the list of class loaders, then it has not been collected:
>
> ------a- Loader java/net/URLClassLoader(0x44DC6DE0), Shadow 0x00ADB6D8,
> Parent sun/misc/Launcher$AppClassLoader(0x00ADB7B0) Number of loaded classes 1 Number of cached classes 11 Allocation used for loaded classes 1 Package owner 0x00ADB6D8
>
>
> Though dereferencing a user-defined class loader seems like a way to ensure that the classes are garbage collected, this is not actually the case. In the previous example, the problem stems from the use of |java.io.ObjectOutputStream.writeObject(Object obj)| and its implications on GC.
>
> When |writeObject()| is invoked (to serialize |SerializationClass|), a reference to this class object is passed internally to |ObjectStreamClass| and stored in a lookup table (that is, in an internal cache). This reference is kept to speed up future serialization of the same class.
>
> When the class loader is dereferenced, the classes that it loaded are not garbage collectable. This is because there is a live reference to the |SerializationClass| class from the |ObjectStreamClass| lookup table. |ObjectStreamClass| is a primordial class and therefore is never garbage collected. The lookup table is referenced from a static field in |ObjectStreamClass| and is kept in the class itself rather than in an instance of it. As a result, the reference to |SerializationClass| exists for the lifetime of the JVM, and the class thus cannot be garbage collected. Importantly, the |SerializationClass| class has a reference to its defining class loader, and so it cannot be completely dereferenced either.
>
> To avoid this problem, any classes that are to be serialized should be loaded by a class loader that does not need to be garbage collected -- by the system class loader, for example.
import java.lang.ref.Reference;
import java.lang.ref.ReferenceQueue;
import java.lang.ref.WeakReference;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.ArrayList;
import java.util.List;
public class SerializationTest extends ClassLoader {
public static void main(String args[]) throws Exception {
URLClassLoader loader = new URLClassLoader(new URL[] {
new URL("file:dir1/")
});
System.out.println("Loading SerializationClass");
Class c = loader.loadClass("SerializationClass");
System.out.println("Creating an instance of SerializationClass");
c.newInstance();
ReferenceQueue<Class<?>> queue = new ReferenceQueue<Class<?>>();
Reference<Class<?>> ref = new WeakReference<Class<?>>(c, queue);
System.out.println("Dereferencing the class loader");
c = null;
loader = null;
System.out.println("Running GC...");
System.gc();
//System.out.println("Triggering a Javadump");
//com.ibm.jvm.Dump.JavaDump();
flushSoftRefs();
Reference<? extends Class<?>> dequeued = queue.remove(1000);
if (dequeued == ref) {
System.out.println("SerializationClass garbage collected");
} else {
throw new Error("SerializationClass not garbage collected");
}
}
private static void flushSoftRefs() {
try {
List<Object> l = new ArrayList<Object>();
while (true) l.add(new byte[10000]);
} catch (OutOfMemoryError e) {
System.out.println("memory exhausted");
}
}
}
Re: ClassLoader and Class Garbage Collection issues with Serialization.
Posted by Peter Jones <pc...@roundroom.net>.
6232010 was one of the related bugs, although it was specifically about pinned stream subclasses, as opposed to serializable classes in the class descriptor cache-- it and the class descriptor cache were fixed at the same time, though, along with 5056445.
I highly doubt that those fixes have been backported to CDC, because they depended on java.util.concurrent.ConcurrentHashMap, which was introduced in Java SE 5.0. As I said in my previous message, though, even without those fixes, with ongoing activity, class descriptor cache entries would eventually get cleared.
-- Peter
On Apr 12, 2010, at 5:39 PM, Peter Firmstone wrote:
> Awesome thanks, I found the bug database record.
>
> http://bugs.sun.com/view_bug.do?bug_id=6232010
>
> I'll have to see if I can track down the CDC source code and see if it too has the fix.
>
> For what I'd like to do with ClassLoaders, this bug would have been a show stopper.
>
> You can just imagine my thoughts, stumbling across it, while theorising about how to reduce duplication of class files, by sharing code between services for services utilising common packages. I'd also like to return results from lookup as a stream, ordered by common bytecode, so the local jvm garbage collects unwanted results during stream inspection, to avoid a memory explosion.
>
> Cheers,
>
> Peter.
>
> Gregg Wonderly wrote:
>> It was my understanding that this issue had been solved as well.
>>
>> Gregg Wonderly
>>
>> Christopher Dolan wrote:
>>> This is news to me. Looking at the JDK 1.5 source code, I can see that
>>> ObjectStreamClass$Caches uses weak references for the classes. So
>>> either this is a solved problem as of 1.5, or there's something
>>> non-obvious in the implementation that I've overlooked.
>>>
>>> Chris
>>>
>>> -----Original Message-----
>>> From: Peter Firmstone [mailto:jini@zeus.net.au] Sent: Saturday, April 10, 2010 8:25 PM
>>> To: river-dev@incubator.apache.org
>>> Subject: ClassLoader and Class Garbage Collection issues with
>>> Serialization.
>>>
>>> I have stumbled across a troubling problem with Serialization relating to Garbage Collection of Classes and ClassLoaders and was hoping someone
>>>
>>> might be able to shed some light on the issue.
>>>
>>> Is it really true that the more objects you distribute, the greater your
>>>
>>> memory consumption because Class files and ClassLoaders cannot be garbage collected?
>>>
>>> Regards,
>>>
>>> Peter.
>>>
>>> The issue can be found here:
>>> http://www.ibm.com/developerworks/java/library/j-dclp3/index.html
>>>
>>> And here's the relevant information, pasted from the link:
>>>
>>>
>>> Problems related to garbage collection and serialization
>>>
>>> The garbage collector interacts closely with the class loader. Among other things, the collector examines the class loader data structures to
>>>
>>> determine which classes are /live/ -- that is, are not garbage collectable. This can often lead to some unexpected problems.
>>>
>>> Figure 2 illustrates a situation where serialization affects the garbage
>>>
>>> collection (GC) of classes and a class loader in an unexpected way:
>>>
>>>
>>> *Figure 2. Serialization example*
>>>
>>> Serialization example
>>>
>>> In this example, |SerializationTest| instantiates a |URLClassLoader|, called |loader|. After loading |SerializationClass|, the class loader is
>>>
>>> dereferenced. The expectation is that this will allow the classes loaded
>>>
>>> by it to be garbage collected. The code for these classes is illustrated
>>>
>>> in Listings 9 and 10:
>>>
>>>
>>> *Listing 9. SerializationTest.java*
>>>
>>> import java.net.MalformedURLException;
>>> import java.net.URL;
>>> import java.net.URLClassLoader;
>>>
>>> public class SerializationTest extends ClassLoader {
>>>
>>> public static void main(String args[]) {
>>> try {
>>> URLClassLoader loader = new URLClassLoader(new URL[] { new URL(
>>> "file://C:/CL_Article/Serialization/dir1/") });
>>> System.out.println("Loading SerializationClass");
>>> Class c = loader.loadClass("SerializationClass");
>>> System.out.println("Creating an instance of
>>> SerializationClass");
>>> c.newInstance();
>>> System.out.println("Dereferencing the class loader");
>>> c = null;
>>> loader = null;
>>> System.out.println("Running GC...");
>>> System.gc();
>>> System.out.println("Triggering a Javadump");
>>> com.ibm.jvm.Dump.JavaDump();
>>> } catch (MalformedURLException e) {
>>> e.printStackTrace();
>>> } catch (InstantiationException e) {
>>> e.printStackTrace();
>>> } catch (IllegalAccessException e) {
>>> e.printStackTrace();
>>> } catch (ClassNotFoundException e) {
>>> e.printStackTrace();
>>> }
>>> }
>>> }
>>>
>>>
>>>
>>> *Listing 10. SerializationClass.java*
>>>
>>> import java.io.File;
>>> import java.io.FileOutputStream;
>>> import java.io.ObjectOutputStream;
>>> import java.io.Serializable;
>>>
>>> public class SerializationClass implements Serializable {
>>>
>>> private static final long serialVersionUID = 5024741671582526226L;
>>>
>>> public SerializationClass() {
>>> try {
>>> File file = new
>>> File("C:/CL_Article/Serialization/test.txt");
>>> FileOutputStream fos = new FileOutputStream(file);
>>> ObjectOutputStream oos = new ObjectOutputStream(fos);
>>> oos.writeObject(this);
>>> oos.reset();
>>> oos.close();
>>> fos.close();
>>> oos = null;
>>> fos = null;
>>> file = null;
>>> } catch (Exception e) {
>>> e.printStackTrace();
>>> }
>>> }
>>> }
>>>
>>>
>>> Using a Javadump, it is possible to discover whether the class loader has been garbage collected. (See the first article in this series for more on using Javadump.) If the following section appears in the list of
>>>
>>> class loaders, then it has not been collected:
>>>
>>> ------a- Loader java/net/URLClassLoader(0x44DC6DE0), Shadow 0x00ADB6D8,
>>> Parent sun/misc/Launcher$AppClassLoader(0x00ADB7B0) Number of loaded classes 1 Number of cached classes 11 Allocation used for loaded classes 1 Package owner 0x00ADB6D8
>>>
>>> Though dereferencing a user-defined class loader seems like a way to ensure that the classes are garbage collected, this is not actually the case. In the previous example, the problem stems from the use of |java.io.ObjectOutputStream.writeObject(Object obj)| and its implications on GC.
>>>
>>> When |writeObject()| is invoked (to serialize |SerializationClass|), a reference to this class object is passed internally to |ObjectStreamClass| and stored in a lookup table (that is, in an internal cache). This reference is kept to speed up future serialization
>>>
>>> of the same class.
>>>
>>> When the class loader is dereferenced, the classes that it loaded are not garbage collectable. This is because there is a live reference to the |SerializationClass| class from the |ObjectStreamClass| lookup table. |ObjectStreamClass| is a primordial class and therefore is never garbage collected. The lookup table is referenced from a static field in
>>>
>>> |ObjectStreamClass| and is kept in the class itself rather than in an instance of it. As a result, the reference to |SerializationClass| exists for the lifetime of the JVM, and the class thus cannot be garbage
>>>
>>> collected. Importantly, the |SerializationClass| class has a reference to its defining class loader, and so it cannot be completely dereferenced either.
>>>
>>> To avoid this problem, any classes that are to be serialized should be loaded by a class loader that does not need to be garbage collected -- by the system class loader, for example.
>>>
>>>
>>>
>>
>>
>
Re: ClassLoader and Class Garbage Collection issues with Serialization.
Posted by Peter Firmstone <ji...@zeus.net.au>.
Awesome thanks, I found the bug database record.
http://bugs.sun.com/view_bug.do?bug_id=6232010
I'll have to see if I can track down the CDC source code and see if it
too has the fix.
For what I'd like to do with ClassLoaders, this bug would have been a
show stopper.
You can just imagine my thoughts, stumbling across it, while theorising
about how to reduce duplication of class files, by sharing code between
services for services utilising common packages. I'd also like to
return results from lookup as a stream, ordered by common bytecode, so
the local jvm garbage collects unwanted results during stream
inspection, to avoid a memory explosion.
Cheers,
Peter.
Gregg Wonderly wrote:
> It was my understanding that this issue had been solved as well.
>
> Gregg Wonderly
>
> Christopher Dolan wrote:
>> This is news to me. Looking at the JDK 1.5 source code, I can see that
>> ObjectStreamClass$Caches uses weak references for the classes. So
>> either this is a solved problem as of 1.5, or there's something
>> non-obvious in the implementation that I've overlooked.
>>
>> Chris
>>
>> -----Original Message-----
>> From: Peter Firmstone [mailto:jini@zeus.net.au] Sent: Saturday, April
>> 10, 2010 8:25 PM
>> To: river-dev@incubator.apache.org
>> Subject: ClassLoader and Class Garbage Collection issues with
>> Serialization.
>>
>> I have stumbled across a troubling problem with Serialization
>> relating to Garbage Collection of Classes and ClassLoaders and was
>> hoping someone
>>
>> might be able to shed some light on the issue.
>>
>> Is it really true that the more objects you distribute, the greater your
>>
>> memory consumption because Class files and ClassLoaders cannot be
>> garbage collected?
>>
>> Regards,
>>
>> Peter.
>>
>> The issue can be found here:
>> http://www.ibm.com/developerworks/java/library/j-dclp3/index.html
>>
>> And here's the relevant information, pasted from the link:
>>
>>
>> Problems related to garbage collection and serialization
>>
>> The garbage collector interacts closely with the class loader. Among
>> other things, the collector examines the class loader data structures to
>>
>> determine which classes are /live/ -- that is, are not garbage
>> collectable. This can often lead to some unexpected problems.
>>
>> Figure 2 illustrates a situation where serialization affects the garbage
>>
>> collection (GC) of classes and a class loader in an unexpected way:
>>
>>
>> *Figure 2. Serialization example*
>>
>> Serialization example
>>
>> In this example, |SerializationTest| instantiates a |URLClassLoader|,
>> called |loader|. After loading |SerializationClass|, the class loader is
>>
>> dereferenced. The expectation is that this will allow the classes loaded
>>
>> by it to be garbage collected. The code for these classes is illustrated
>>
>> in Listings 9 and 10:
>>
>>
>> *Listing 9. SerializationTest.java*
>>
>> import java.net.MalformedURLException;
>> import java.net.URL;
>> import java.net.URLClassLoader;
>>
>> public class SerializationTest extends ClassLoader {
>>
>> public static void main(String args[]) {
>> try {
>> URLClassLoader loader = new URLClassLoader(new URL[] { new URL(
>> "file://C:/CL_Article/Serialization/dir1/") });
>> System.out.println("Loading SerializationClass");
>> Class c = loader.loadClass("SerializationClass");
>> System.out.println("Creating an instance of
>> SerializationClass");
>> c.newInstance();
>> System.out.println("Dereferencing the class loader");
>> c = null;
>> loader = null;
>> System.out.println("Running GC...");
>> System.gc();
>> System.out.println("Triggering a Javadump");
>> com.ibm.jvm.Dump.JavaDump();
>> } catch (MalformedURLException e) {
>> e.printStackTrace();
>> } catch (InstantiationException e) {
>> e.printStackTrace();
>> } catch (IllegalAccessException e) {
>> e.printStackTrace();
>> } catch (ClassNotFoundException e) {
>> e.printStackTrace();
>> }
>> }
>> }
>>
>>
>>
>> *Listing 10. SerializationClass.java*
>>
>> import java.io.File;
>> import java.io.FileOutputStream;
>> import java.io.ObjectOutputStream;
>> import java.io.Serializable;
>>
>> public class SerializationClass implements Serializable {
>>
>> private static final long serialVersionUID = 5024741671582526226L;
>>
>> public SerializationClass() {
>> try {
>> File file = new
>> File("C:/CL_Article/Serialization/test.txt");
>> FileOutputStream fos = new FileOutputStream(file);
>> ObjectOutputStream oos = new ObjectOutputStream(fos);
>> oos.writeObject(this);
>> oos.reset();
>> oos.close();
>> fos.close();
>> oos = null;
>> fos = null;
>> file = null;
>> } catch (Exception e) {
>> e.printStackTrace();
>> }
>> }
>> }
>>
>>
>> Using a Javadump, it is possible to discover whether the class loader
>> has been garbage collected. (See the first article in this series for
>> more on using Javadump.) If the following section appears in the list of
>>
>> class loaders, then it has not been collected:
>>
>> ------a- Loader java/net/URLClassLoader(0x44DC6DE0), Shadow 0x00ADB6D8,
>> Parent sun/misc/Launcher$AppClassLoader(0x00ADB7B0)
>> Number of loaded classes 1 Number of cached classes 11
>> Allocation used for loaded classes 1 Package
>> owner 0x00ADB6D8
>>
>>
>> Though dereferencing a user-defined class loader seems like a way to
>> ensure that the classes are garbage collected, this is not actually
>> the case. In the previous example, the problem stems from the use of
>> |java.io.ObjectOutputStream.writeObject(Object obj)| and its
>> implications on GC.
>>
>> When |writeObject()| is invoked (to serialize |SerializationClass|),
>> a reference to this class object is passed internally to
>> |ObjectStreamClass| and stored in a lookup table (that is, in an
>> internal cache). This reference is kept to speed up future serialization
>>
>> of the same class.
>>
>> When the class loader is dereferenced, the classes that it loaded are
>> not garbage collectable. This is because there is a live reference to
>> the |SerializationClass| class from the |ObjectStreamClass| lookup
>> table. |ObjectStreamClass| is a primordial class and therefore is
>> never garbage collected. The lookup table is referenced from a static
>> field in
>>
>> |ObjectStreamClass| and is kept in the class itself rather than in an
>> instance of it. As a result, the reference to |SerializationClass|
>> exists for the lifetime of the JVM, and the class thus cannot be garbage
>>
>> collected. Importantly, the |SerializationClass| class has a
>> reference to its defining class loader, and so it cannot be
>> completely dereferenced either.
>>
>> To avoid this problem, any classes that are to be serialized should
>> be loaded by a class loader that does not need to be garbage
>> collected -- by the system class loader, for example.
>>
>>
>>
>
>
Re: ClassLoader and Class Garbage Collection issues with Serialization.
Posted by Gregg Wonderly <gr...@wonderly.org>.
It was my understanding that this issue had been solved as well.
Gregg Wonderly
Christopher Dolan wrote:
> This is news to me. Looking at the JDK 1.5 source code, I can see that
> ObjectStreamClass$Caches uses weak references for the classes. So
> either this is a solved problem as of 1.5, or there's something
> non-obvious in the implementation that I've overlooked.
>
> Chris
>
> -----Original Message-----
> From: Peter Firmstone [mailto:jini@zeus.net.au]
> Sent: Saturday, April 10, 2010 8:25 PM
> To: river-dev@incubator.apache.org
> Subject: ClassLoader and Class Garbage Collection issues with
> Serialization.
>
> I have stumbled across a troubling problem with Serialization relating
> to Garbage Collection of Classes and ClassLoaders and was hoping someone
>
> might be able to shed some light on the issue.
>
> Is it really true that the more objects you distribute, the greater your
>
> memory consumption because Class files and ClassLoaders cannot be
> garbage collected?
>
> Regards,
>
> Peter.
>
> The issue can be found here:
> http://www.ibm.com/developerworks/java/library/j-dclp3/index.html
>
> And here's the relevant information, pasted from the link:
>
>
> Problems related to garbage collection and serialization
>
> The garbage collector interacts closely with the class loader. Among
> other things, the collector examines the class loader data structures to
>
> determine which classes are /live/ -- that is, are not garbage
> collectable. This can often lead to some unexpected problems.
>
> Figure 2 illustrates a situation where serialization affects the garbage
>
> collection (GC) of classes and a class loader in an unexpected way:
>
>
> *Figure 2. Serialization example*
>
> Serialization example
>
> In this example, |SerializationTest| instantiates a |URLClassLoader|,
> called |loader|. After loading |SerializationClass|, the class loader is
>
> dereferenced. The expectation is that this will allow the classes loaded
>
> by it to be garbage collected. The code for these classes is illustrated
>
> in Listings 9 and 10:
>
>
> *Listing 9. SerializationTest.java*
>
> import java.net.MalformedURLException;
> import java.net.URL;
> import java.net.URLClassLoader;
>
> public class SerializationTest extends ClassLoader {
>
> public static void main(String args[]) {
> try {
> URLClassLoader loader = new URLClassLoader(new URL[] { new URL(
> "file://C:/CL_Article/Serialization/dir1/") });
> System.out.println("Loading SerializationClass");
> Class c = loader.loadClass("SerializationClass");
> System.out.println("Creating an instance of
> SerializationClass");
> c.newInstance();
> System.out.println("Dereferencing the class loader");
> c = null;
> loader = null;
>
> System.out.println("Running GC...");
> System.gc();
> System.out.println("Triggering a Javadump");
> com.ibm.jvm.Dump.JavaDump();
>
> } catch (MalformedURLException e) {
> e.printStackTrace();
> } catch (InstantiationException e) {
> e.printStackTrace();
> } catch (IllegalAccessException e) {
> e.printStackTrace();
> } catch (ClassNotFoundException e) {
> e.printStackTrace();
> }
> }
> }
>
>
>
> *Listing 10. SerializationClass.java*
>
> import java.io.File;
> import java.io.FileOutputStream;
> import java.io.ObjectOutputStream;
> import java.io.Serializable;
>
> public class SerializationClass implements Serializable {
>
> private static final long serialVersionUID = 5024741671582526226L;
>
> public SerializationClass() {
> try {
> File file = new
> File("C:/CL_Article/Serialization/test.txt");
> FileOutputStream fos = new FileOutputStream(file);
> ObjectOutputStream oos = new ObjectOutputStream(fos);
> oos.writeObject(this);
> oos.reset();
> oos.close();
> fos.close();
> oos = null;
> fos = null;
> file = null;
> } catch (Exception e) {
> e.printStackTrace();
> }
> }
> }
>
>
> Using a Javadump, it is possible to discover whether the class loader
> has been garbage collected. (See the first article in this series for
> more on using Javadump.) If the following section appears in the list of
>
> class loaders, then it has not been collected:
>
> ------a- Loader java/net/URLClassLoader(0x44DC6DE0), Shadow 0x00ADB6D8,
> Parent sun/misc/Launcher$AppClassLoader(0x00ADB7B0)
> Number of loaded classes 1
> Number of cached classes 11
> Allocation used for loaded classes 1
> Package owner 0x00ADB6D8
>
>
>
> Though dereferencing a user-defined class loader seems like a way to
> ensure that the classes are garbage collected, this is not actually the
> case. In the previous example, the problem stems from the use of
> |java.io.ObjectOutputStream.writeObject(Object obj)| and its
> implications on GC.
>
> When |writeObject()| is invoked (to serialize |SerializationClass|), a
> reference to this class object is passed internally to
> |ObjectStreamClass| and stored in a lookup table (that is, in an
> internal cache). This reference is kept to speed up future serialization
>
> of the same class.
>
> When the class loader is dereferenced, the classes that it loaded are
> not garbage collectable. This is because there is a live reference to
> the |SerializationClass| class from the |ObjectStreamClass| lookup
> table. |ObjectStreamClass| is a primordial class and therefore is never
> garbage collected. The lookup table is referenced from a static field in
>
> |ObjectStreamClass| and is kept in the class itself rather than in an
> instance of it. As a result, the reference to |SerializationClass|
> exists for the lifetime of the JVM, and the class thus cannot be garbage
>
> collected. Importantly, the |SerializationClass| class has a reference
> to its defining class loader, and so it cannot be completely
> dereferenced either.
>
> To avoid this problem, any classes that are to be serialized should be
> loaded by a class loader that does not need to be garbage collected --
> by the system class loader, for example.
>
>
>
RE: ClassLoader and Class Garbage Collection issues with Serialization.
Posted by Christopher Dolan <ch...@avid.com>.
This is news to me. Looking at the JDK 1.5 source code, I can see that
ObjectStreamClass$Caches uses weak references for the classes. So
either this is a solved problem as of 1.5, or there's something
non-obvious in the implementation that I've overlooked.
Chris
-----Original Message-----
From: Peter Firmstone [mailto:jini@zeus.net.au]
Sent: Saturday, April 10, 2010 8:25 PM
To: river-dev@incubator.apache.org
Subject: ClassLoader and Class Garbage Collection issues with
Serialization.
I have stumbled across a troubling problem with Serialization relating
to Garbage Collection of Classes and ClassLoaders and was hoping someone
might be able to shed some light on the issue.
Is it really true that the more objects you distribute, the greater your
memory consumption because Class files and ClassLoaders cannot be
garbage collected?
Regards,
Peter.
The issue can be found here:
http://www.ibm.com/developerworks/java/library/j-dclp3/index.html
And here's the relevant information, pasted from the link:
Problems related to garbage collection and serialization
The garbage collector interacts closely with the class loader. Among
other things, the collector examines the class loader data structures to
determine which classes are /live/ -- that is, are not garbage
collectable. This can often lead to some unexpected problems.
Figure 2 illustrates a situation where serialization affects the garbage
collection (GC) of classes and a class loader in an unexpected way:
*Figure 2. Serialization example*
Serialization example
In this example, |SerializationTest| instantiates a |URLClassLoader|,
called |loader|. After loading |SerializationClass|, the class loader is
dereferenced. The expectation is that this will allow the classes loaded
by it to be garbage collected. The code for these classes is illustrated
in Listings 9 and 10:
*Listing 9. SerializationTest.java*
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLClassLoader;
public class SerializationTest extends ClassLoader {
public static void main(String args[]) {
try {
URLClassLoader loader = new URLClassLoader(new URL[] { new URL(
"file://C:/CL_Article/Serialization/dir1/") });
System.out.println("Loading SerializationClass");
Class c = loader.loadClass("SerializationClass");
System.out.println("Creating an instance of
SerializationClass");
c.newInstance();
System.out.println("Dereferencing the class loader");
c = null;
loader = null;
System.out.println("Running GC...");
System.gc();
System.out.println("Triggering a Javadump");
com.ibm.jvm.Dump.JavaDump();
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (InstantiationException e) {
e.printStackTrace();
} catch (IllegalAccessException e) {
e.printStackTrace();
} catch (ClassNotFoundException e) {
e.printStackTrace();
}
}
}
*Listing 10. SerializationClass.java*
import java.io.File;
import java.io.FileOutputStream;
import java.io.ObjectOutputStream;
import java.io.Serializable;
public class SerializationClass implements Serializable {
private static final long serialVersionUID = 5024741671582526226L;
public SerializationClass() {
try {
File file = new
File("C:/CL_Article/Serialization/test.txt");
FileOutputStream fos = new FileOutputStream(file);
ObjectOutputStream oos = new ObjectOutputStream(fos);
oos.writeObject(this);
oos.reset();
oos.close();
fos.close();
oos = null;
fos = null;
file = null;
} catch (Exception e) {
e.printStackTrace();
}
}
}
Using a Javadump, it is possible to discover whether the class loader
has been garbage collected. (See the first article in this series for
more on using Javadump.) If the following section appears in the list of
class loaders, then it has not been collected:
------a- Loader java/net/URLClassLoader(0x44DC6DE0), Shadow 0x00ADB6D8,
Parent sun/misc/Launcher$AppClassLoader(0x00ADB7B0)
Number of loaded classes 1
Number of cached classes 11
Allocation used for loaded classes 1
Package owner 0x00ADB6D8
Though dereferencing a user-defined class loader seems like a way to
ensure that the classes are garbage collected, this is not actually the
case. In the previous example, the problem stems from the use of
|java.io.ObjectOutputStream.writeObject(Object obj)| and its
implications on GC.
When |writeObject()| is invoked (to serialize |SerializationClass|), a
reference to this class object is passed internally to
|ObjectStreamClass| and stored in a lookup table (that is, in an
internal cache). This reference is kept to speed up future serialization
of the same class.
When the class loader is dereferenced, the classes that it loaded are
not garbage collectable. This is because there is a live reference to
the |SerializationClass| class from the |ObjectStreamClass| lookup
table. |ObjectStreamClass| is a primordial class and therefore is never
garbage collected. The lookup table is referenced from a static field in
|ObjectStreamClass| and is kept in the class itself rather than in an
instance of it. As a result, the reference to |SerializationClass|
exists for the lifetime of the JVM, and the class thus cannot be garbage
collected. Importantly, the |SerializationClass| class has a reference
to its defining class loader, and so it cannot be completely
dereferenced either.
To avoid this problem, any classes that are to be serialized should be
loaded by a class loader that does not need to be garbage collected --
by the system class loader, for example.