You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Bogdan Kecman <bo...@alteray.com> on 2006/09/11 01:42:26 UTC

java.lang.OutOfMemoryError: Java heap space

 
Hi,
I have strange prob with nutch 0.7.2 not sure how to fix this, must be some config tweak but tried few different settings and still have the same prob ... 
how to increare this "java heap space" ?
 
thanks in advance
here is the log:
 
 
060909 160139 SEVERE error writing output:java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
060909 160139 SEVERE error writing output:java.io.IOException: key out of order: 161408 after 161408
java.io.IOException: key out of order: 161408 after 161408
        at org.apache.nutch.io.MapFile$Writer.checkKey(MapFile.java:134)
        at org.apache.nutch.io.MapFile$Writer.append(MapFile.java:120)
        at org.apache.nutch.io.ArrayFile$Writer.append(ArrayFile.java:39)
        at org.apache.nutch.fetcher.Fetcher$FetcherThread.outputPage(Fetcher.java:280)
        at org.apache.nutch.fetcher.Fetcher$FetcherThread.handleFetch(Fetcher.java:261)
        at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:193)
060909 160139 SEVERE error writing output:java.io.IOException: key out of order: 161408 after 161408
java.io.IOException: key out of order: 161408 after 161408
        at org.apache.nutch.io.MapFile$Writer.checkKey(MapFile.java:134)
        at org.apache.nutch.io.MapFile$Writer.append(MapFile.java:120)
        at org.apache.nutch.io.ArrayFile$Writer.append(ArrayFile.java:39)
        at org.apache.nutch.fetcher.Fetcher$FetcherThread.outputPage(Fetcher.java:280)
        at org.apache.nutch.fetcher.Fetcher$FetcherThread.handleFetch(Fetcher.java:261)
        at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:193)
Exception in thread "main" java.lang.RuntimeException: SEVERE error logged.  Exiting fetcher.
        at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:354)
        at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:488)
        at com.alteray.linex.aggregation.StartHere.main(StartHere.java:183)
060909 160313 SEVERE error writing output:java.io.IOException: key out of order: 161408 after 161408
...
...

RE: java.lang.OutOfMemoryError: Java heap space

Posted by Bogdan Kecman <bo...@alteray.com>.
Hi,
Big thanks .. I found it .. For nutch 0.7.2 

export NUTCH_HEAPSIZE=1500

Does the job (the nutch shell script automatically adds "m" at the parameter)

Thanks again
Bogdan 

> -----Original Message-----
> From: Dennis Kubes [mailto:nutch-dev@dragonflymc.com] 
> Sent: Monday, September 11, 2006 1:49 AM
> To: nutch-user@lucene.apache.org
> Subject: Re: java.lang.OutOfMemoryError: Java heap space
> 
> I don't know if it is the same in 7.2 but in .8 there is a 
> hadoop-env.sh file where you can uncomment the JAVA_OPTS 
> variable and give the heap more memory.  Either way the JVM 
> must be started with more memory, something like this vm 
> option -Xmx1024M for a 1Gig heap.
> 
> Dennis
> 
> Bogdan Kecman wrote:
> >  
> > Hi,
> > I have strange prob with nutch 0.7.2 not sure how to fix 
> this, must be some config tweak but tried few different 
> settings and still have the same prob ... 
> > how to increare this "java heap space" ?
> >  
> > thanks in advance
> > here is the log:
> >  
> >  
> > 060909 160139 SEVERE error writing 
> output:java.lang.OutOfMemoryError: 
> > Java heap space
> > java.lang.OutOfMemoryError: Java heap space
> > 060909 160139 SEVERE error writing 
> output:java.io.IOException: key out 
> > of order: 161408 after 161408
> > java.io.IOException: key out of order: 161408 after 161408
> >         at 
> org.apache.nutch.io.MapFile$Writer.checkKey(MapFile.java:134)
> >         at 
> org.apache.nutch.io.MapFile$Writer.append(MapFile.java:120)
> >         at 
> org.apache.nutch.io.ArrayFile$Writer.append(ArrayFile.java:39)
> >         at 
> org.apache.nutch.fetcher.Fetcher$FetcherThread.outputPage(Fetc
> her.java:280)
> >         at 
> org.apache.nutch.fetcher.Fetcher$FetcherThread.handleFetch(Fet
> cher.java:261)
> >         at 
> > org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:193)
> > 060909 160139 SEVERE error writing 
> output:java.io.IOException: key out 
> > of order: 161408 after 161408
> > java.io.IOException: key out of order: 161408 after 161408
> >         at 
> org.apache.nutch.io.MapFile$Writer.checkKey(MapFile.java:134)
> >         at 
> org.apache.nutch.io.MapFile$Writer.append(MapFile.java:120)
> >         at 
> org.apache.nutch.io.ArrayFile$Writer.append(ArrayFile.java:39)
> >         at 
> org.apache.nutch.fetcher.Fetcher$FetcherThread.outputPage(Fetc
> her.java:280)
> >         at 
> org.apache.nutch.fetcher.Fetcher$FetcherThread.handleFetch(Fet
> cher.java:261)
> >         at 
> > org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:193)
> > Exception in thread "main" java.lang.RuntimeException: 
> SEVERE error logged.  Exiting fetcher.
> >         at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:354)
> >         at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:488)
> >         at 
> > com.alteray.linex.aggregation.StartHere.main(StartHere.java:183)
> > 060909 160313 SEVERE error writing 
> output:java.io.IOException: key out 
> > of order: 161408 after 161408 ...
> > ...
> >
> >

Re: java.lang.OutOfMemoryError: Java heap space

Posted by Dennis Kubes <nu...@dragonflymc.com>.
I don't know if it is the same in 7.2 but in .8 there is a hadoop-env.sh 
file where you can uncomment the JAVA_OPTS variable and give the heap 
more memory.  Either way the JVM must be started with more memory, 
something like this vm option -Xmx1024M for a 1Gig heap.

Dennis

Bogdan Kecman wrote:
>  
> Hi,
> I have strange prob with nutch 0.7.2 not sure how to fix this, must be some config tweak but tried few different settings and still have the same prob ... 
> how to increare this "java heap space" ?
>  
> thanks in advance
> here is the log:
>  
>  
> 060909 160139 SEVERE error writing output:java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> 060909 160139 SEVERE error writing output:java.io.IOException: key out of order: 161408 after 161408
> java.io.IOException: key out of order: 161408 after 161408
>         at org.apache.nutch.io.MapFile$Writer.checkKey(MapFile.java:134)
>         at org.apache.nutch.io.MapFile$Writer.append(MapFile.java:120)
>         at org.apache.nutch.io.ArrayFile$Writer.append(ArrayFile.java:39)
>         at org.apache.nutch.fetcher.Fetcher$FetcherThread.outputPage(Fetcher.java:280)
>         at org.apache.nutch.fetcher.Fetcher$FetcherThread.handleFetch(Fetcher.java:261)
>         at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:193)
> 060909 160139 SEVERE error writing output:java.io.IOException: key out of order: 161408 after 161408
> java.io.IOException: key out of order: 161408 after 161408
>         at org.apache.nutch.io.MapFile$Writer.checkKey(MapFile.java:134)
>         at org.apache.nutch.io.MapFile$Writer.append(MapFile.java:120)
>         at org.apache.nutch.io.ArrayFile$Writer.append(ArrayFile.java:39)
>         at org.apache.nutch.fetcher.Fetcher$FetcherThread.outputPage(Fetcher.java:280)
>         at org.apache.nutch.fetcher.Fetcher$FetcherThread.handleFetch(Fetcher.java:261)
>         at org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:193)
> Exception in thread "main" java.lang.RuntimeException: SEVERE error logged.  Exiting fetcher.
>         at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:354)
>         at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:488)
>         at com.alteray.linex.aggregation.StartHere.main(StartHere.java:183)
> 060909 160313 SEVERE error writing output:java.io.IOException: key out of order: 161408 after 161408
> ...
> ...
>
>