[jira] [Commented] (MAPREDUCE-4379) Node Manager throws java.lang.OutOfMemoryError: Java heap space due to org.apache.hadoop.fs.LocalDirAllocator.contexts

["Devaraj K (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402244#comment-13402244 ] 

Devaraj K commented on MAPREDUCE-4379:
--------------------------------------

{code:title=ContainerLocalizer.java|borderStyle=solid}
    this.appDirs =
      new LocalDirAllocator(String.format(APPCACHE_CTXT_FMT, appId));
    this.userDirs =
      new LocalDirAllocator(String.format(USERCACHE_CTXT_FMT, appId));
    this.pendingResources = new HashMap<LocalResource,Future<Path>>();
{code}

Here for every application during localization, it creates two LocalDirAllocator instances.


{code:title=LocalDirAllocator.java|borderStyle=solid}
  private AllocatorPerContext obtainContext(String contextCfgItemName) {
    synchronized (contexts) {
      AllocatorPerContext l = contexts.get(contextCfgItemName);
      if (l == null) {
        contexts.put(contextCfgItemName, 
                    (l = new AllocatorPerContext(contextCfgItemName)));
      }
      return l;
    }
  }
{code}

 Those two instances will internally creates AllocatorPerContext instances and add those into contexts while obtaining contexts. It will keep on adding for every application and no where else these are getting removed from the map. It is leading to OOM after running for some time.
                
> Node Manager throws java.lang.OutOfMemoryError: Java heap space due to org.apache.hadoop.fs.LocalDirAllocator.contexts
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4379
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4379
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2, nodemanager
>    Affects Versions: 2.0.0-alpha, 3.0.0
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>            Priority: Critical
>
> {code:xml}
> Exception in thread "Container Monitor" java.lang.OutOfMemoryError: Java heap space
> 	at java.io.BufferedReader.<init>(BufferedReader.java:80)
> 	at java.io.BufferedReader.<init>(BufferedReader.java:91)
> 	at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:410)
> 	at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:171)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:389)
> 	Exception in thread "LocalizerRunner for container_1340690914008_10890_01_000003" java.lang.OutOfMemoryError: Java heap space
> 	at java.util.Arrays.copyOfRange(Arrays.java:3209)
> 	at java.lang.String.<init>(String.java:215)
> 	at com.sun.org.apache.xerces.internal.xni.XMLString.toString(XMLString.java:185)
> 	at com.sun.org.apache.xerces.internal.parsers.AbstractDOMParser.characters(AbstractDOMParser.java:1188)
> 	at com.sun.org.apache.xerces.internal.xinclude.XIncludeHandler.characters(XIncludeHandler.java:1084)
> 	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:464)
> 	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
> 	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
> 	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
> 	at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:235)
> 	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
> 	at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
> 	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1738)
> 	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1689)
> 	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1635)
> 	at org.apache.hadoop.conf.Configuration.set(Configuration.java:722)
> 	at org.apache.hadoop.conf.Configuration.setStrings(Configuration.java:1300)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:375)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:127)
> 	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:103)
> 	at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:862)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira