RE: Necessary to send parse command after merge?

["McGibbney, Lewis John" <Le...@gcu.ac.uk>]

Thanks for both of your replies Markus

Further to your last point below. My current configuration is a dual boot Windows (not currently being used) and Ubuntu. To get past the input path does not exist error, I took your advice and added the hadoop.tmp.dir property to nutch-site.xml and set the value to a location on my HDD. In my case there is more free space on the windows partition than on my Ubuntu one.

This is hopefully beneficial to anyone else who might experience this in the future!

Thanks Lewis
________________________________________
From: Markus Jelsma [markus.jelsma@openindex.io]
Sent: 31 March 2011 21:22
To: user@nutch.apache.org
Cc: McGibbney, Lewis John
Subject: Re: Necessary to send parse command after merge?

Lewis,

You don't need to merge before parsing the fetch. In large fetches it is
recommended not to parse while fetching, hence the default. You need to
execute a parse command after the fetch.

The dreaded input path does not exist is most likely caused by a corrupted
fetch, such as running out of diskspace in the /tmp dir or hadoop.tmp.dir. If
this happens, all commands on the segment will somehow fail.

Cheers,

> By default within <!--fetcher properties --> of nutch-site.xml the
> following setting is set to false
>
> <property>
>   <name>fetcher.parse</name>
>   <value>false</value>
>   <description>If true, fetcher will parse content. Default is false, which
> means that a separate parsing step is required after fetching is
> finished.</description> </property>
>
> Setting to true removes the requirement for an additional command in a
> custom crawl script
>
> Lewis
> ________________________________________
> From: McGibbney, Lewis John [Lewis.McGibbney@gcu.ac.uk]
> Sent: 31 March 2011 20:24
> To: user@nutch.apache.org
> Subject: Necessary to send parse command after merge?
>
> Hi list,
>
> I am curious to know whether we need to send a command to parse segments
> after merging them. I get an exception as follows when I run my script
>
> LinkDb: org.apache.hadoop.mapred.InvalidInputException: Input path does not
> exist:
> file:/home/lewis/workspace/branch-1.3/runtime/local/crawl/segments/2011033
> 1194928/parse_data at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:1
> 90) at
> org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileIn
> putFormat.java:44) at
> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:20
> 1) at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730) at
> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249) at
> org.apache.nutch.crawl.LinkDb.invert(LinkDb.java:175)
>         at org.apache.nutch.crawl.LinkDb.run(LinkDb.java:292)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.nutch.crawl.LinkDb.main(LinkDb.java:255)
>
> Do I need to add something to my script in-between Merge Segments and
> Invert Links?
>
> Using branch-1.3
>
> Thank you for any help
> Lewis
>
> Glasgow Caledonian University is a registered Scottish charity, number
> SC021474
>
> Winner: Times Higher Education’s Widening Participation Initiative of the
> Year 2009 and Herald Society’s Education Initiative of the Year 2009.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,
> en.html
>
> Winner: Times Higher Education’s Outstanding Support for Early Career
> Researchers of the Year 2010, GCU as a lead with Universities Scotland
> partners.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691
> ,en.html
>
> Email has been scanned for viruses by Altman Technologies' email management
> service - www.altman.co.uk/emailsystems
>
> Glasgow Caledonian University is a registered Scottish charity, number
> SC021474
>
> Winner: Times Higher Education’s Widening Participation Initiative of the
> Year 2009 and Herald Society’s Education Initiative of the Year 2009.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,
> en.html
>
> Winner: Times Higher Education’s Outstanding Support for Early Career
> Researchers of the Year 2010, GCU as a lead with Universities Scotland
> partners.
> http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691
> ,en.html
Email has been scanned for viruses by Altman Technologies' email management service - www.altman.co.uk/emailsystems

Glasgow Caledonian University is a registered Scottish charity, number SC021474

Winner: Times Higher Education’s Widening Participation Initiative of the Year 2009 and Herald Society’s Education Initiative of the Year 2009.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,6219,en.html

Winner: Times Higher Education’s Outstanding Support for Early Career Researchers of the Year 2010, GCU as a lead with Universities Scotland partners.
http://www.gcu.ac.uk/newsevents/news/bycategory/theuniversity/1/name,15691,en.html