Long crawl keeps failing in fetch phase

[Amit Sela <am...@infolinks.com>]

Hi all,
I'm trying to execute a long crawl with no depth. I am crawling over a seed
list of ~15M sites with depth 1 because I just want to index these sites
and nothing else.
I'm running with nutch 1.6 over a 3 machines cluster.
Once I get to 79% (takes a few days...) my map and reduce tasks get killed
with "Lost task tracker: ..." and after that I start getting tasks failed
with: "Child Error...IOException: Task process exit with nonzero status of
255" and "Child Error... IOException: Creation of symlink from [path to
log].cleanup to [path ti mapred-tmp-dir log].cleanup failed.

I read about "mapred.task.timeout" might be the cause of "Lost task
tracker" so I changed it to  10800000 (default is 600000) but it keeps
happening. 600000 is 10 minuites and 10800000 is 3 hours.. maybe increase
even more ?

Thanks for the help, I'm really puzzled here.

Amit.

RE: Long crawl keeps failing in fetch phase

[Markus Jelsma <ma...@openindex.io>]

Hi,

Begin with decreasing the number of records your process per mapper and increase the number of mappers. Better parallelism and less recovery work in case of problems. We usually don't do more than 25k URL's per mapper but do a lot of mappers instead! Easier to control and debug.

Cheers.
 
-----Original message-----
> From:Amit Sela <am...@infolinks.com>
> Sent: Saturday 6th July 2013 10:43
> To: user@nutch.apache.org
> Subject: Long crawl keeps failing in fetch phase
> 
> Hi all,
> I'm trying to execute a long crawl with no depth. I am crawling over a seed
> list of ~15M sites with depth 1 because I just want to index these sites
> and nothing else.
> I'm running with nutch 1.6 over a 3 machines cluster.
> Once I get to 79% (takes a few days...) my map and reduce tasks get killed
> with "Lost task tracker: ..." and after that I start getting tasks failed
> with: "Child Error...IOException: Task process exit with nonzero status of
> 255" and "Child Error... IOException: Creation of symlink from [path to
> log].cleanup to [path ti mapred-tmp-dir log].cleanup failed.
> 
> I read about "mapred.task.timeout" might be the cause of "Lost task
> tracker" so I changed it to  10800000 (default is 600000) but it keeps
> happening. 600000 is 10 minuites and 10800000 is 3 hours.. maybe increase
> even more ?
> 
> Thanks for the help, I'm really puzzled here.
> 
> Amit.
> 

Re: Long crawl keeps failing in fetch phase

[Amit Sela <am...@infolinks.com>]

Forgot to mention that this happens in the fetch job. Thanks.
On Jul 6, 2013 11:43 AM, "Amit Sela" <am...@infolinks.com> wrote:

> Hi all,
> I'm trying to execute a long crawl with no depth. I am crawling over a
> seed list of ~15M sites with depth 1 because I just want to index these
> sites and nothing else.
> I'm running with nutch 1.6 over a 3 machines cluster.
> Once I get to 79% (takes a few days...) my map and reduce tasks get killed
> with "Lost task tracker: ..." and after that I start getting tasks failed
> with: "Child Error...IOException: Task process exit with nonzero status of
> 255" and "Child Error... IOException: Creation of symlink from [path to
> log].cleanup to [path ti mapred-tmp-dir log].cleanup failed.
>
> I read about "mapred.task.timeout" might be the cause of "Lost task
> tracker" so I changed it to  10800000 (default is 600000) but it keeps
> happening. 600000 is 10 minuites and 10800000 is 3 hours.. maybe increase
> even more ?
>
> Thanks for the help, I'm really puzzled here.
>
> Amit.
>