You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2013/12/19 17:52:08 UTC
[jira] [Commented] (PIG-3486) Pig hitting OOM while using
PigRunner.run()
[ https://issues.apache.org/jira/browse/PIG-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853028#comment-13853028 ]
Rohini Palaniswamy commented on PIG-3486:
-----------------------------------------
When removing the hook on reset(), it is not added back if we launch another job with the same Launcher as it is done in the constructor. We can put it in finalize but having only there is not a good idea as it is not guaranteed to be run. If we are putting in reset() then we should add the shutdown hook in launchPig instead of the constructor.
> Pig hitting OOM while using PigRunner.run()
> -------------------------------------------
>
> Key: PIG-3486
> URL: https://issues.apache.org/jira/browse/PIG-3486
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.11.1
> Reporter: Vivek Padmanabhan
> Attachments: PIG-3486.patch1, histolive.txt
>
>
> I have a timer based class, which will trigger a pig script execution every 5 minutes using PigRunner.run(args, null).
> But it looks like the heap usage is gradually increasing after around 15days it crossed 1G, ie after invoking the above method 4k times.
> The top entries of the histo live goes like this;
> num #instances #bytes class name
> ----------------------------------------------
> 1: 2430178 433053080 [C
> 2: 3055280 97768960 java.util.Hashtable$Entry
> 3: 2454870 78555840 java.lang.String
> 4: 1585204 50726528 java.util.HashMap$Entry
> 5: 260310 37503984 <constMethodKlass>
> 6: 260310 35413536 <methodKlass>
> 7: 35024 23724672 [Ljava.util.Hashtable$Entry;
> 8: 7599 18141016 <constantPoolKlass>
> 9: 47551 18066696 [Ljava.util.HashMap$Entry;
> 10: 209516 16761280 java.lang.reflect.Method
> 11: 212292 16732008 [I
> 12: 6881 11332896 <constantPoolCacheKlass>
> 13: 7599 7160920 <instanceKlassKlass>
> 14: 79412 4447072 java.util.ResourceBundle$CacheKey
> 15: 10787 3958464 [S
> 16: 79412 3811776 java.util.ResourceBundle$BundleReference
> 17: 26634 3458160 [B
> 18: 133701 3208824 java.util.LinkedList$Node
> 19: 85492 2735744 java.util.concurrent.ConcurrentHashMap$HashEntry
> 20: 79412 2541184 java.util.ResourceBundle$LoaderReference
> 21: 47515 2280720 java.util.HashMap
> 22: 37298 2274416 [Ljava.lang.Object;
> 23: 70638 2260416 java.util.LinkedList
> 24: 2949 1994376 <methodDataKlass>
> 25: 7914 1749080 java.lang.Class
> 26: 62746 1505904 org.apache.commons.logging.impl.Log4JLogger
> 27: 16639 1463824 [[I
> 28: 21279 1361856 java.net.URL
> 29: 28090 1348320 java.util.Hashtable
> 30: 14167 1231856 [Ljava.util.WeakHashMap$Entry;
> 31: 17770 710800 java.lang.ref.Finalizer
> 32: 10626 680064 java.util.jar.JarFile
> 33: 14167 680016 java.util.WeakHashMap
> 34: 14238 569520 java.util.WeakHashMap$Entry
> 35: 7104 568320 java.util.jar.JarFile$JarFileEntry
> 36: 165 567264 [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
> 37: 10637 510576 sun.nio.cs.UTF_8$Encoder
> 38: 10633 510384 sun.misc.URLClassPath$JarLoader
> 39: 14176 453632 java.lang.ref.ReferenceQueue
> 40: 17747 409752 [Ljava.lang.Class;
> 41: 3463 387856 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$HangingJobKiller
> 42: 15355 368520 java.util.ArrayList
> 43: 10632 340224 java.util.zip.ZipCoder
> 44: 6932 332736 java.util.Properties
> 45: 4060 292320 java.lang.reflect.Constructor
> 46: 7143 285720 java.util.LinkedHashMap$Entry
> 47: 3517 281360 org.apache.pig.impl.PigContext$ContextClassLoader
> 48: 3476 278144 [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;
> 49: 3458 276640 java.net.URI
> 50: 8576 274432 antlr.ANTLRHashString
> 51: 10632 255168 java.util.ArrayDeque
> There are way too many instances of MapReduceLauncher$HangingJobKiller.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)