You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Armel T. Nene" <ar...@idna-solutions.com> on 2007/01/23 20:26:35 UTC

is crawldb format in Nutch 0.8 compatible with Nutch0.7

Hi guys,

 

I am running in some nightmares when trying to iterate over values in the
Nutch 0.8.2 crawldb. I am getting some hadoop exception such as the
following:

 

07/01/23 18:33:56 INFO conf.Configuration: parsing
jar:file:/C:/nutch-0.8.2-dev/lib/hadoop-0.4.0-patched.jar!/hadoop-default.xm
l

07/01/23 18:33:56 INFO conf.Configuration: parsing
jar:file:/C:/nutch-0.8.2-dev/nutch-0.8.2-dev.jar!/nutch-default.xml

07/01/23 18:33:56 INFO conf.Configuration: parsing
jar:file:/C:/nutch-0.8.2-dev/nutch-0.8.2-dev.jar!/nutch-site.xml

Exception in thread "main" java.lang.ArithmeticException: / by zero

        at
org.apache.hadoop.mapred.lib.HashPartitioner.getPartition(HashPartitioner.ja
va:33)

        at
org.apache.hadoop.mapred.MapFileOutputFormat.getEntry(MapFileOutputFormat.ja
va:88)

        at org.apache.nutch.crawl.CrawlDbReader.get(CrawlDbReader.java:321)

therefore, if I can iterate over the values contained in the crawldb using
Nutch 0.7 API, I should think this will fix the issue. So the question is;

 

is Nutch 0.8 backward compatible with Nutch 0.7.2

 

Thanks,

 

Armel

 

-------------------------------------------------

Armel T. Nene

iDNA Solutions

Tel: +44 (207) 257 6124

Mobile: +44 (788) 695 0483 

 <http://blog.idna-solutions.com/> http://blog.idna-solutions.com

 


Re: is crawldb format in Nutch 0.8 compatible with Nutch0.7

Posted by Andrzej Bialecki <ab...@getopt.org>.
Armel T. Nene wrote:
> therefore, if I can iterate over the values contained in the crawldb using
> Nutch 0.7 API, I should think this will fix the issue. So the question is;
>
>  
>
> is Nutch 0.8 backward compatible with Nutch 0.7.2
>   


This subject was discussed several times in the past. The answer is 
still no - Nutch 0.8 is NOT compatible with 0.7. It is possible to write 
converters for some of the data, but it's a lot of tedious work. The 
recommended upgrade path is to dump your 0.7 WebDB to a text file, and 
then inject this text file to a 0.8 CrawlDb.

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com