You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by A Laxmi <a....@gmail.com> on 2013/10/22 01:47:20 UTC
[Nutch 2.2.1] Error java.lang.OutOfMemoryError: GC overhead limit exceeded
Hi,
I saw this strange out of memory error today that said GC overhead limit
exceeded -* Error java.lang.OutOfMemoryError: GC overhead limit exceeded *in
parsing stage. I have seen OOM errors; heap space before but never that
said "GC overhead limit exceeded"
Below is the log. Can anyone please comment?
2013-10-21 08:59:33,879 INFO parse.ParserJob - Parsing
http://www.###.##.###/pub/###2011.pdf
2013-10-21 09:02:01,697 ERROR zookeeper.ClientCnxn - from
pool-1-thread-1-SendThread(isscvm.isscdomain:2181)
*java.lang.OutOfMemoryError: GC overhead limit exceeded*
at java.util.Arrays.copyOf(Arrays.java:2367)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
at java.lang.StringBuilder.append(StringBuilder.java:132)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157)
2013-10-21 09:05:55,650 WARN parse.ParseUtil - Error parsing
http://www.fs.fed.us/pnw/olympia/silv/publications/opt/622_HarringtonDevine2011.pdf
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC
overhead limit exceeded
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:202)
at org.apache.nutch.parse.ParseUtil.runParser(ParseUtil.java:148)
at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:129)
at org.apache.nutch.parse.ParseUtil.process(ParseUtil.java:176)
at
org.apache.nutch.parse.ParserJob$ParserMapper.map(ParserJob.java:131)
at
org.apache.nutch.parse.ParserJob$ParserMapper.map(ParserJob.java:78)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:55,651 WARN parse.ParseUtil - Unable to successfully
parse content
http://www.fs.fed.us/pnw/olympia/silv/publications/opt/622_HarringtonDevine2011.pdfof
type application/pdf
2013-10-21 09:05:44,429 ERROR zookeeper.ClientCnxn - from
pool-1-thread-1-SendThread(centosvm.centosdomain:2181)
java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:38,851 ERROR zookeeper.ClientCnxn - from
pool-1-thread-1-SendThread(centosvm.centosdomain:2181)
java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:27,363 ERROR zookeeper.ClientCnxn - Error while calling
watcher
java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:23,990 ERROR zookeeper.ClientCnxn - from
pool-1-thread-1-SendThread(centosvm.centosdomain:2181)
java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:14,381 ERROR zookeeper.ClientCnxn - from
pool-1-thread-1-SendThread(centosvm.centosdomain:2181)
java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:09,330 ERROR zookeeper.ClientCnxn - Caught unexpected
throwable
java.lang.OutOfMemoryError: GC overhead limit exceeded
Yan: [Nutch 2.2.1] Error java.lang.OutOfMemoryError: GC overhead
limit exceeded
Posted by Talat Uyarer <ta...@agmlab.com>.
Hi Laxmi,
I think you should tune your hbase. This informations is not enough for understanding your problem.
you can add below lines in $HBASE_HOME/conf/hbase-env.sh for debuging.
export HBASE_OPTS="$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/usr/local/hbase/logs/gc-hbase.log"
In our cluster, I use Concurrent-Mark-Sweep GC(CMS). Just for training you add your hbase-env.sh below lines:
export HBASE_OPTS="$HBASE_OPTS -XX:CMSInitiatingOccupancyFraction=60"
I think these configuration will be solved your problem. But if you want to learn how to tune my hbase below documents will be good start point.
http://www.packtpub.com/article/hbase-basic-performance-tuning
http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html
I hope you solve your problems :)
Talat
----- Orijinal Mesaj -----
Kimden: A Laxmi <a....@gmail.com>
Kime: user@nutch.apache.org
Gönderilenler: Tue, 22 Oct 2013 02:47:20 +0300 (EEST)
Konu: [Nutch 2.2.1] Error java.lang.OutOfMemoryError: GC overhead limit exceeded
Hi,
I saw this strange out of memory error today that said GC overhead limit
exceeded -* Error java.lang.OutOfMemoryError: GC overhead limit exceeded *in
parsing stage. I have seen OOM errors; heap space before but never that
said "GC overhead limit exceeded"
Below is the log. Can anyone please comment?
2013-10-21 08:59:33,879 INFO parse.ParserJob - Parsing
http://www.###.##.###/pub/###2011.pdf
2013-10-21 09:02:01,697 ERROR zookeeper.ClientCnxn - from
pool-1-thread-1-SendThread(isscvm.isscdomain:2181)
*java.lang.OutOfMemoryError: GC overhead limit exceeded*
at java.util.Arrays.copyOf(Arrays.java:2367)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:130)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:114)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:415)
at java.lang.StringBuilder.append(StringBuilder.java:132)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1157)
2013-10-21 09:05:55,650 WARN parse.ParseUtil - Error parsing
http://www.fs.fed.us/pnw/olympia/silv/publications/opt/622_HarringtonDevine2011.pdf
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: GC
overhead limit exceeded
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:202)
at org.apache.nutch.parse.ParseUtil.runParser(ParseUtil.java:148)
at org.apache.nutch.parse.ParseUtil.parse(ParseUtil.java:129)
at org.apache.nutch.parse.ParseUtil.process(ParseUtil.java:176)
at
org.apache.nutch.parse.ParserJob$ParserMapper.map(ParserJob.java:131)
at
org.apache.nutch.parse.ParserJob$ParserMapper.map(ParserJob.java:78)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:55,651 WARN parse.ParseUtil - Unable to successfully
parse content
http://www.fs.fed.us/pnw/olympia/silv/publications/opt/622_HarringtonDevine2011.pdfof
type application/pdf
2013-10-21 09:05:44,429 ERROR zookeeper.ClientCnxn - from
pool-1-thread-1-SendThread(centosvm.centosdomain:2181)
java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:38,851 ERROR zookeeper.ClientCnxn - from
pool-1-thread-1-SendThread(centosvm.centosdomain:2181)
java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:27,363 ERROR zookeeper.ClientCnxn - Error while calling
watcher
java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:23,990 ERROR zookeeper.ClientCnxn - from
pool-1-thread-1-SendThread(centosvm.centosdomain:2181)
java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:14,381 ERROR zookeeper.ClientCnxn - from
pool-1-thread-1-SendThread(centosvm.centosdomain:2181)
java.lang.OutOfMemoryError: GC overhead limit exceeded
2013-10-21 09:05:09,330 ERROR zookeeper.ClientCnxn - Caught unexpected
throwable
java.lang.OutOfMemoryError: GC overhead limit exceeded