You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Patricio Galeas <pg...@yahoo.de> on 2010/03/05 22:14:14 UTC

By Indexing I get: OutOfMemoryError: GC overhead limit exceeded ...

Hello all,
I am  running Nutch in a Virtual Machine (Debian) with 8 GB RAM and 1,5TB for the hadoop temporal folder.
Running the index process with a 1.3GB segments folder,  I got  "OutOfMemoryError: GC overhead limit exceeded"  (see below)

I created the segments using slice=50000
and I also set HADOOP_HEAPSIZE with the maximal physical memory (8000).

Do I need more memory to run the index process?
Are there some limitation to run Nutch in a Virtual Machine?

Thank you!
Pato

...
...                                                                                                                                                                        
2010-03-05 19:52:13,864 INFO  plugin.PluginRepository -         Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)                                                                                      
2010-03-05 19:52:13,864 INFO  plugin.PluginRepository -         Ontology Model Loader (org.apache.nutch.ontology.Ontology)                                                                                  
2010-03-05 19:52:13,867 INFO  lang.LanguageIdentifier - Language identifier configuration [1-4/2048]  
2010-03-05 19:52:22,961 INFO  lang.LanguageIdentifier - Language identifier plugin supports: it(1000) is(1000) hu(1000) th(1000) sv(1000) sq(1000) fr(1000) ru(1000) fi(1000) es(1000) en(1000) el(1000) ee(1000) pt(1000) de(1000) da(1000) pl(1000) no(1000) nl(1000)                                           
2010-03-05 19:52:22,961 INFO  indexer.IndexingFilters - Adding org.apache.nutch.analysis.lang.LanguageIndexingFilter                                                                                        
2010-03-05 19:52:22,963 INFO  indexer.IndexingFilters - Adding org.apache.nutch.indexer.basic.BasicIndexingFilter                                                                                           
2010-03-05 19:52:22,964 INFO  indexer.IndexingFilters - Adding org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2010-03-05 19:52:36,278 WARN  mapred.LocalJobRunner - job_local_0001
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
        at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:775)
        at org.apache.hadoop.io.Text.encode(Text.java:388)
        at org.apache.hadoop.io.Text.encode(Text.java:369)
        at org.apache.hadoop.io.Text.writeString(Text.java:409)
        at org.apache.nutch.parse.Outlink.write(Outlink.java:52)
        at org.apache.nutch.parse.ParseData.write(ParseData.java:152)
        at org.apache.hadoop.io.GenericWritable.write(GenericWritable.java:135)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:613)
        at org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:67)
        at org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:50)
        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
2010-03-05 19:52:37,277 FATAL indexer.Indexer - Indexer: java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
        at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
        at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)

__________________________________________________
Do You Yahoo!?
Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails. 
http://mail.yahoo.com 

AW: By Indexing I get: OutOfMemoryError: GC overhead limit exceeded ...

Posted by Patricio Galeas <pg...@yahoo.de>.
Hello Ted,

I ran the command 'ps -aux' and I confirmed that only 1GB was defined.
I adjust  NUTCH_HEAPSIZE to 8GB (physical RAM) and ran it again
successfully.

Do you know which parameters need to be adjusted if not enough physical RAM is available on the server? For example for 2GB RAM.
I ran a web crawl (depth=6) without the parameter topN and the segments grew exponetially.
Later, I had a lot of problems by merging the segments and by indexing
(not enough memory, too many opened files, etc.).

Thank you for your help
Pato



----- Ursprüngliche Mail ----
Von: Ted Yu <yu...@gmail.com>
An: nutch-user@lucene.apache.org
Gesendet: Samstag, den 6. März 2010, 15:42:38 Uhr
Betreff: Re: By Indexing I get: OutOfMemoryError: GC overhead limit exceeded  ...

Can you use 'ps aux' to find out the -Xmx commandine parameter passed to
java for the following action ?

On Fri, Mar 5, 2010 at 1:14 PM, Patricio Galeas <pg...@yahoo.de> wrote:

> Hello all,
> I am  running Nutch in a Virtual Machine (Debian) with 8 GB RAM and 1,5TB
> for the hadoop temporal folder.
> Running the index process with a 1.3GB segments folder,  I got
>  "OutOfMemoryError: GC overhead limit exceeded"  (see below)
>
> I created the segments using slice=50000
> and I also set HADOOP_HEAPSIZE with the maximal physical memory (8000).
>
> Do I need more memory to run the index process?
> Are there some limitation to run Nutch in a Virtual Machine?
>
> Thank you!
> Pato
>
> ...
> ...
> 2010-03-05 19:52:13,864 INFO  plugin.PluginRepository -         Nutch
> Scoring (org.apache.nutch.scoring.ScoringFilter)
> 2010-03-05 19:52:13,864 INFO  plugin.PluginRepository -         Ontology
> Model Loader (org.apache.nutch.ontology.Ontology)
> 2010-03-05 19:52:13,867 INFO  lang.LanguageIdentifier - Language identifier
> configuration [1-4/2048]
> 2010-03-05 19:52:22,961 INFO  lang.LanguageIdentifier - Language identifier
> plugin supports: it(1000) is(1000) hu(1000) th(1000) sv(1000) sq(1000)
> fr(1000) ru(1000) fi(1000) es(1000) en(1000) el(1000) ee(1000) pt(1000)
> de(1000) da(1000) pl(1000) no(1000) nl(1000)
> 2010-03-05 19:52:22,961 INFO  indexer.IndexingFilters - Adding
> org.apache.nutch.analysis.lang.LanguageIndexingFilter
> 2010-03-05 19:52:22,963 INFO  indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2010-03-05 19:52:22,964 INFO  indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2010-03-05 19:52:36,278 WARN  mapred.LocalJobRunner - job_local_0001
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>        at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
>        at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:775)
>        at org.apache.hadoop.io.Text.encode(Text.java:388)
>        at org.apache.hadoop.io.Text.encode(Text.java:369)
>        at org.apache.hadoop.io.Text.writeString(Text.java:409)
>        at org.apache.nutch.parse.Outlink.write(Outlink.java:52)
>        at org.apache.nutch.parse.ParseData.write(ParseData.java:152)
>        at
> org.apache.hadoop.io.GenericWritable.write(GenericWritable.java:135)
>        at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
>        at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
>        at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:613)
>        at
> org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:67)
>        at
> org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:50)
>        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>        at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
> 2010-03-05 19:52:37,277 FATAL indexer.Indexer - Indexer:
> java.io.IOException: Job failed!
>        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
>        at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
>        at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
>
> __________________________________________________
> Do You Yahoo!?
> Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz
> gegen Massenmails.
> http://mail.yahoo.com
>


__________________________________________________
Do You Yahoo!?
Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails. 
http://mail.yahoo.com 

Re: By Indexing I get: OutOfMemoryError: GC overhead limit exceeded ...

Posted by Ted Yu <yu...@gmail.com>.
Can you use 'ps aux' to find out the -Xmx commandine parameter passed to
java for the following action ?

On Fri, Mar 5, 2010 at 1:14 PM, Patricio Galeas <pg...@yahoo.de> wrote:

> Hello all,
> I am  running Nutch in a Virtual Machine (Debian) with 8 GB RAM and 1,5TB
> for the hadoop temporal folder.
> Running the index process with a 1.3GB segments folder,  I got
>  "OutOfMemoryError: GC overhead limit exceeded"  (see below)
>
> I created the segments using slice=50000
> and I also set HADOOP_HEAPSIZE with the maximal physical memory (8000).
>
> Do I need more memory to run the index process?
> Are there some limitation to run Nutch in a Virtual Machine?
>
> Thank you!
> Pato
>
> ...
> ...
> 2010-03-05 19:52:13,864 INFO  plugin.PluginRepository -         Nutch
> Scoring (org.apache.nutch.scoring.ScoringFilter)
> 2010-03-05 19:52:13,864 INFO  plugin.PluginRepository -         Ontology
> Model Loader (org.apache.nutch.ontology.Ontology)
> 2010-03-05 19:52:13,867 INFO  lang.LanguageIdentifier - Language identifier
> configuration [1-4/2048]
> 2010-03-05 19:52:22,961 INFO  lang.LanguageIdentifier - Language identifier
> plugin supports: it(1000) is(1000) hu(1000) th(1000) sv(1000) sq(1000)
> fr(1000) ru(1000) fi(1000) es(1000) en(1000) el(1000) ee(1000) pt(1000)
> de(1000) da(1000) pl(1000) no(1000) nl(1000)
> 2010-03-05 19:52:22,961 INFO  indexer.IndexingFilters - Adding
> org.apache.nutch.analysis.lang.LanguageIndexingFilter
> 2010-03-05 19:52:22,963 INFO  indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2010-03-05 19:52:22,964 INFO  indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2010-03-05 19:52:36,278 WARN  mapred.LocalJobRunner - job_local_0001
> java.lang.OutOfMemoryError: GC overhead limit exceeded
>        at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
>        at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:775)
>        at org.apache.hadoop.io.Text.encode(Text.java:388)
>        at org.apache.hadoop.io.Text.encode(Text.java:369)
>        at org.apache.hadoop.io.Text.writeString(Text.java:409)
>        at org.apache.nutch.parse.Outlink.write(Outlink.java:52)
>        at org.apache.nutch.parse.ParseData.write(ParseData.java:152)
>        at
> org.apache.hadoop.io.GenericWritable.write(GenericWritable.java:135)
>        at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
>        at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
>        at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:613)
>        at
> org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:67)
>        at
> org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:50)
>        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
>        at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
> 2010-03-05 19:52:37,277 FATAL indexer.Indexer - Indexer:
> java.io.IOException: Job failed!
>        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
>        at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
>        at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
>
> __________________________________________________
> Do You Yahoo!?
> Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz
> gegen Massenmails.
> http://mail.yahoo.com
>