You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Ivan Veselovsky (JIRA)" <ji...@apache.org> on 2016/06/09 09:25:21 UTC
[jira] [Created] (IGNITE-3285) A reference to HadoopClassLoader may
be held in IGFS service pool threads after Job finish.
Ivan Veselovsky created IGNITE-3285:
---------------------------------------
Summary: A reference to HadoopClassLoader may be held in IGFS service pool threads after Job finish.
Key: IGNITE-3285
URL: https://issues.apache.org/jira/browse/IGNITE-3285
Project: Ignite
Issue Type: Bug
Reporter: Ivan Veselovsky
Assignee: Ivan Veselovsky
Memory profiling shows that an instance of HadoopClassLoader used for Hadoop job may still be referenced after that job finish. This happens for 2 reasons.
1) When a new thread in IGFS pool is created by a thread that has HadoopClassLoader as the current context class loader, this class loader inherently propagated as a context class loader to the created thread:
{code}
java.lang.Throwable: #### 00igfs-#85%null%.<init>: set hadoop class loader: HadoopClassLoader [name=hadoop-task-6b4d1037-65df-4e83-a7f8-7338e13ab1cf_1-SETUP-0]. Current cl = HadoopClassLoader [name=hadoop-task-6b4d1037-65df-4e83-a7f8-7338e13ab1cf_1-SETUP-0]
at org.apache.ignite.thread.IgniteThread.<init>(IgniteThread.java:83)
at org.apache.ignite.thread.IgniteThread.<init>(IgniteThread.java:62)
at org.apache.ignite.thread.IgniteThreadFactory$1.<init>(IgniteThreadFactory.java:62)
at org.apache.ignite.thread.IgniteThreadFactory.newThread(IgniteThreadFactory.java:62)
at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:610)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:924)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1360)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:132)
at org.apache.ignite.internal.processors.igfs.IgfsDataManager.callIgfsLocalSafe(IgfsDataManager.java:1133)
at org.apache.ignite.internal.processors.igfs.IgfsDataManager.processBatch(IgfsDataManager.java:1024)
at org.apache.ignite.internal.processors.igfs.IgfsDataManager.access$2500(IgfsDataManager.java:100)
at org.apache.ignite.internal.processors.igfs.IgfsDataManager$BlocksWriter.storeDataBlocks(IgfsDataManager.java:1416)
at org.apache.ignite.internal.processors.igfs.IgfsDataManager.storeDataBlocks(IgfsDataManager.java:538)
at org.apache.ignite.internal.processors.igfs.IgfsOutputStreamImpl.storeDataBlock(IgfsOutputStreamImpl.java:193)
at org.apache.ignite.internal.processors.igfs.IgfsOutputStreamAdapter.sendData(IgfsOutputStreamAdapter.java:252)
at org.apache.ignite.internal.processors.igfs.IgfsOutputStreamAdapter.write(IgfsOutputStreamAdapter.java:135)
at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsInProc.writeData(HadoopIgfsInProc.java:440)
at org.apache.ignite.internal.processors.hadoop.igfs.HadoopIgfsOutputStream.write(HadoopIgfsOutputStream.java:112)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1333)
at org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat$1.write(SequenceFileOutputFormat.java:83)
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Context.write(HadoopV2Context.java:144)
at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at org.apache.hadoop.examples.RandomTextWriter$RandomTextMapper.map(RandomTextWriter.java:140)
at org.apache.hadoop.examples.RandomTextWriter$RandomTextMapper.map(RandomTextWriter.java:102)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2MapTask.run0(HadoopV2MapTask.java:74)
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2Task.run(HadoopV2Task.java:54)
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.run(HadoopV2TaskContext.java:249)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.runTask(HadoopRunnableTask.java:201)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call0(HadoopRunnableTask.java:144)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call(HadoopRunnableTask.java:116)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask$1.call(HadoopRunnableTask.java:114)
at org.apache.ignite.internal.processors.hadoop.v2.HadoopV2TaskContext.runAsJobOwner(HadoopV2TaskContext.java:544)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:114)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopRunnableTask.call(HadoopRunnableTask.java:46)
at org.apache.ignite.internal.processors.hadoop.taskexecutor.HadoopExecutorService$2.body(HadoopExecutorService.java:186)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
{code}
2. java.lang.Thread#inheritedAccessControlContext field after new thread construction may have reference to the parent Thread context class loader through the java.security.ProtectionDomain#classloader field.
{code}
this - value: org.apache.ignite.internal.processors.hadoop.HadoopClassLoader #2
<- classloader - class: java.security.ProtectionDomain, value: org.apache.ignite.internal.processors.hadoop.HadoopClassLoader #2
<- [4] - class: java.security.ProtectionDomain[], value: java.security.ProtectionDomain #123
<- context - class: java.security.AccessControlContext, value: java.security.ProtectionDomain[] #67 (8 items)
<- inheritedAccessControlContext (thread object) - class: org.apache.ignite.thread.IgniteThreadFactory$1, value: java.security.AccessControlContext #105
{code}
Note that strictly speaking these references do not form leak , since number of threads in the pool is limited by a constant. Nevertheless, it should be fixed since means not effective memory usage.
Currently we suggest to fix this simply be pre-creating all the threads in igfs service pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)