You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Sangjin Lee (JIRA)" <ji...@apache.org> on 2016/03/25 00:10:25 UTC
[jira] [Updated] (HADOOP-12958) PhantomReference for filesystem
statistics can trigger OOM
[ https://issues.apache.org/jira/browse/HADOOP-12958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sangjin Lee updated HADOOP-12958:
---------------------------------
Attachment: HADOOP-12958.01.patch
Posted patch v.1.
After looking at this some more, it seems that {{WeakReference}} is a better choice than {{PhantomReference}} after all. We are able to do clean-up based on the reference queue with either type, but the hold on the referent seems tighter (ironically) with the phantom reference in this case. Weak references will be cleared out by the garbage collector, unlike the phantom references. We never access the actual referent in the clean-up logic, so it is basically a matter of substituting the reference type.
I distilled this use case and the {{FileSystem}} code to reproduce this case, and the following is the verbose output of that test with the *phantom* reference:
{noformat}
$ java -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintReferenceGC -Xms1600m -Xmx1600m Test
allocating 1 G
creating a phantom reference to this thread
sleeping for 10 seconds
done sleeping
allocating another 1 G
10.528: [GC (Allocation Failure) 10.550: [SoftReference, 0 refs, 0.0000145 secs]10.550: [WeakReference, 9 refs, 0.0000054 secs]10.550: [FinalReference, 16 refs, 0.0000157 secs]10.550: [PhantomReference, 1 refs, 0 refs, 0.0000060 secs]10.550: [JNI Weak Reference, 0.0000061 secs][PSYoungGen: 24576K->416K(477696K)] 1073152K->1049000K(1570304K), 0.0222061 secs] [Times: user=0.17 sys=0.00, real=0.02 secs]
10.550: [Full GC (Ergonomics) 10.550: [SoftReference, 0 refs, 0.0000250 secs]10.550: [WeakReference, 2 refs, 0.0000048 secs]10.550: [FinalReference, 4 refs, 0.0000044 secs]10.550: [PhantomReference, 0 refs, 0 refs, 0.0000049 secs]10.550: [JNI Weak Reference, 0.0000031 secs][PSYoungGen: 416K->0K(477696K)] [ParOldGen: 1048584K->1048857K(1092608K)] 1049000K->1048857K(1570304K), [Metaspace: 2614K->2614K(1056768K)], 0.0216855 secs] [Times: user=0.02 sys=0.01, real=0.02 secs]
10.572: [GC (Allocation Failure) 10.584: [SoftReference, 0 refs, 0.0000340 secs]10.584: [WeakReference, 0 refs, 0.0000049 secs]10.584: [FinalReference, 0 refs, 0.0000043 secs]10.584: [PhantomReference, 0 refs, 0 refs, 0.0000055 secs]10.584: [JNI Weak Reference, 0.0000037 secs][PSYoungGen: 0K->0K(477696K)] 1048857K->1048857K(1570304K), 0.0125265 secs] [Times: user=0.07 sys=0.00, real=0.02 secs]
10.584: [Full GC (Allocation Failure) 10.585: [SoftReference, 37 refs, 0.0000157 secs]10.585: [WeakReference, 2 refs, 0.0000050 secs]10.585: [FinalReference, 4 refs, 0.0000047 secs]10.585: [PhantomReference, 0 refs, 0 refs, 0.0000053 secs]10.585: [JNI Weak Reference, 0.0000032 secs][PSYoungGen: 0K->0K(477696K)] [ParOldGen: 1048857K->1048845K(1092608K)] 1048857K->1048845K(1570304K), [Metaspace: 2614K->2614K(1056768K)], 0.0114447 secs] [Times: user=0.02 sys=0.01, real=0.01 secs]
phantom reference dequeued
Exception in thread "main"
java.lang.OutOfMemoryError: Java heap space
at Test$BigBufferContainer.<init>(Test.java:26)
at Test.main(Test.java:18)
Heap
PSYoungGen total 477696K, used 20480K [0x000000079eb00000, 0x00000007c0000000, 0x00000007c0000000)
eden space 409600K, 5% used [0x000000079eb00000,0x000000079ff001a8,0x00000007b7b00000)
from space 68096K, 0% used [0x00000007bbd80000,0x00000007bbd80000,0x00000007c0000000)
to space 68096K, 0% used [0x00000007b7b00000,0x00000007b7b00000,0x00000007bbd80000)
ParOldGen total 1092608K, used 1048845K [0x000000075c000000, 0x000000079eb00000, 0x000000079eb00000)
object space 1092608K, 95% used [0x000000075c000000,0x000000079c043718,0x000000079eb00000)
Metaspace used 2646K, capacity 4496K, committed 4864K, reserved 1056768K
class space used 291K, capacity 388K, committed 512K, reserved 1048576K
{noformat}
With the *weak* reference:
{noformat}
$ java -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintReferenceGC -Xms1600m -Xmx1600m Test
allocating 1 G
creating a weak reference to this thread
sleeping for 10 seconds
done sleeping
allocating another 1 G
10.673: [GC (Allocation Failure) 10.695: [SoftReference, 0 refs, 0.0000244 secs]10.695: [WeakReference, 10 refs, 0.0000064 secs]10.695: [FinalReference, 16 refs, 0.0000300 secs]10.695: [PhantomReference, 0 refs, 0 refs, 0.0000051 secs]10.695: [JNI Weak Reference, 0.0000073 secs][PSYoungGen: 24576K->416K(477696K)] 1073152K->1049000K(1570304K), 0.0229661 secs] [Times: user=0.17 sys=0.00, real=0.02 secs]
10.696: [Full GC (Ergonomics) 10.696: [SoftReference, 0 refs, 0.0000143 secs]10.696: [WeakReference, 2 refs, 0.0000047 secs]10.696: [FinalReference, 0 refs, 0.0000042 secs]10.696: [PhantomReference, 0 refs, 0 refs, 0.0000049 secs]10.696: [JNI Weak Reference, 0.0000032 secs][PSYoungGen: 416K->0K(477696K)] [ParOldGen: 1048584K->280K(1092608K)] 1049000K->280K(1570304K), [Metaspace: 2614K->2614K(1056768K)], 0.0209466 secs] [Times: user=0.02 sys=0.01, real=0.02 secs]
weak reference dequeued
completed normally
Heap
PSYoungGen total 477696K, used 20480K [0x000000079eb00000, 0x00000007c0000000, 0x00000007c0000000)
eden space 409600K, 5% used [0x000000079eb00000,0x000000079ff001a8,0x00000007b7b00000)
from space 68096K, 0% used [0x00000007b7b00000,0x00000007b7b00000,0x00000007bbd80000)
to space 68096K, 0% used [0x00000007bbd80000,0x00000007bbd80000,0x00000007c0000000)
ParOldGen total 1092608K, used 1048856K [0x000000075c000000, 0x000000079eb00000, 0x000000079eb00000)
object space 1092608K, 95% used [0x000000075c000000,0x000000079c046378,0x000000079eb00000)
Metaspace used 2621K, capacity 4496K, committed 4864K, reserved 1056768K
class space used 288K, capacity 388K, committed 512K, reserved 1048576K
{noformat}
> PhantomReference for filesystem statistics can trigger OOM
> ----------------------------------------------------------
>
> Key: HADOOP-12958
> URL: https://issues.apache.org/jira/browse/HADOOP-12958
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 2.7.3, 2.6.4
> Reporter: Jason Lowe
> Assignee: Sangjin Lee
> Fix For: 2.7.3, 2.6.5
>
> Attachments: HADOOP-12958.01.patch
>
>
> I saw an OOM that appears to have been caused by the phantom references introduced for file system statistics management. I'll post details in a followup comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)