You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by "Hadanny, Ido" <ih...@paypal.com> on 2011/12/25 15:35:51 UTC
hadoop-pig mapper runs out of memory in what appears to be a log4j
memory leak?
Hey guys,
I'm running a hadoop-pig job, and it fails after an hour :( on the last mapper, with what appears to me as a memory leak in log4j - could this be the case???
My job has 3370 mappers, and 3369 are reported to finish successfully, and then the last one starts failing and reports 4 failed attempts, each on a different machine in my 6-nodes cluster , and all on the same reason "Error: Java heap space", although I'm running them with -Xmx1024m, which is plenty enough.
In the failed task attempts stderr logs I can see log4j:WARN No appenders could be found for logger (org.apache.pig.impl.util.SpillableMemoryManager). And in their stdout, I always see the exception below. The offending line is:
LogUtils.warn(this,
"Unable to interpret value " + Arrays.toString(b) + " in field being " +
"converted to int, caught NumberFormatException <" +
nfe.getMessage() + "> field discarded",
PigWarning.FIELD_DISCARDED_TYPE_CONVERSION_FAILED, mLog);
But I don't see even a single WARN line of this type anywhere.
Any ideas? (also posted here: http://stackoverflow.com/questions/8630038/hadoop-pig-mapper-runs-out-of-memory-in-what-appears-to-be-a-log4j-memory-leak )
The exception:
2011-12-25 13:12:57,663 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2882) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390) at java.lang.StringBuffer.append(StringBuffer.java:224) at org.apache.log4j.helpers.PatternParser$LiteralPatternConverter.format(PatternParser.java:420) at org.apache.log4j.PatternLayout.format(PatternLayout.java:502) at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302) at org.apache.log4j.WriterAppender.append(WriterAppender.java:160) at org.apache.hadoop.mapred.TaskLogAppender.append(TaskLogAppender.java:83) at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66) at org.apache.log4j.Category.callAppenders(Category.java:206) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.log(Category.java:856) at org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:157) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigHadoopLogger.warn(PigHadoopLogger.java:69) at org.apache.pig.impl.util.LogUtils.warn(LogUtils.java:40) at org.apache.pig.builtin.Utf8StorageConverter.bytesToInteger(Utf8StorageConverter.java:389) at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.getNext(POCast.java:155) at ... org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)