You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by MilleBii <mi...@gmail.com> on 2011/06/16 13:53:22 UTC

Fetcher keeps having heap-size problems

The fetcher is creating me weird problems on the master node only and  not
on data node despite the following actions :

+ increased HADOOP_HEAPSIZE from 1000 to 2000
+ reduced the number of threads
+ reduced the number URL's
+ increase swap on master node

I'm using Nutch1.0 with patches 721, 769, 770

What can be the issue ? If it is a memory leak why is it not happenning on
datanode.


-- 
-MilleBii-

Re: Fetcher keeps having heap-size problems

Posted by Markus Jelsma <ma...@openindex.io>.

Are you parsing during the fetch cycle?

On Thursday 16 June 2011 13:53:22 MilleBii wrote:
> The fetcher is creating me weird problems on the master node only and  not
> on data node despite the following actions :
> 
> + increased HADOOP_HEAPSIZE from 1000 to 2000
> + reduced the number of threads
> + reduced the number URL's
> + increase swap on master node
> 
> I'm using Nutch1.0 with patches 721, 769, 770
> 
> What can be the issue ? If it is a memory leak why is it not happenning on
> datanode.

-- 
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350