You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by rishi pathak <ma...@gmail.com> on 2011/02/03 15:06:20 UTC

Upgrade to hadoop-0.21.0

Hi,
         I tried upgrading upgrading to hadoop-0.21.0 in nutch-1.2. 'ant
package' does not report any errors and build is sucessfull but nutch crawl
is failing with following errors:



./bin/nutch crawl /home2/nazgul/hadoop/dfs/local/urls/ -dir
/home2/nazgul/hadoop/dfs/local/crawl-data/ -depth 3
11/02/03 19:13:29 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
11/02/03 19:13:29 INFO crawl.Crawl: crawl started in:
/home2/internal/sysadmin/nazgul/hadoop/dfs/local/crawl-data
11/02/03 19:13:29 INFO crawl.Crawl: rootUrlDir =
/home2/nazgul/hadoop/dfs/local/urls
11/02/03 19:13:29 INFO crawl.Crawl: threads = 2
11/02/03 19:13:29 INFO crawl.Crawl: depth = 3
11/02/03 19:13:29 INFO crawl.Crawl: indexer=lucene
11/02/03 19:13:29 INFO crawl.Injector: Injector: starting at 2011-02-03
19:13:29
11/02/03 19:13:29 INFO crawl.Injector: Injector: crawlDb:
/home2/nazgul/hadoop/dfs/local/crawl-data/crawldb
11/02/03 19:13:29 INFO crawl.Injector: Injector: urlDir:
/home2/nazgul/hadoop/dfs/local/urls
11/02/03 19:13:29 WARN conf.Configuration: mapred.temp.dir is deprecated.
Instead, use mapreduce.cluster.temp.dir
11/02/03 19:13:29 INFO crawl.Injector: Injector: Converting injected urls to
crawl db entries.
Exception in thread "main" java.lang.NoClassDefFoundError:
org/codehaus/jackson/map/JsonMappingException
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:960)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:534)
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:779)
        at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
        at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)
Caused by: java.lang.ClassNotFoundException:
org.codehaus.jackson.map.JsonMappingException
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)




-- 
---
Rishi Pathak
National PARAM Supercomputing Facility
C-DAC, Pune, India

move from a single node to 4 node structure

Posted by Patricio Galeas <pg...@yahoo.de>.
Hi,

running Nutch-1.2 on a single machine I produce a 2GB index (157 segments, slice 
50.000).
Because the performance is low, I would like to test further crawls in Amazon 
EC2.
Q1: If I start with 4 nodes,  should I divide the number of the segments 
proportionally for each node, and then start a new crawl?
Q2: Analysing the actual log files (single machine), I found out that the most 
time consuming are Index Database (13 hours), Actualize Crawl Database (11 
hours),  Actualize LinkDB (7 hours 24), etc. Assuming the 4 node structure, do 
the index and database actualization time proportionally decrease?

Thanks
Patricio



Re: Upgrade to hadoop-0.21.0

Posted by rishi pathak <ma...@gmail.com>.
Following steps were followed and it seems like working:

#Do an upgrade as described in
http://wiki.apache.org/nutch/Upgrading_Hadoop
#Download and put jackson-all-1.7.2 in $NUTCH_HOME/lib from
http://jackson.codehaus.org/
#Download and put avro-1.3.3 in $NUTCH_HOME/lib from
avro.apache.org(avro-1.4.x breaks)




On Thu, Feb 3, 2011 at 7:36 PM, rishi pathak <ma...@gmail.com>wrote:

> Hi,
>          I tried upgrading upgrading to hadoop-0.21.0 in nutch-1.2. 'ant
> package' does not report any errors and build is sucessfull but nutch crawl
> is failing with following errors:
>
>
>
> ./bin/nutch crawl /home2/nazgul/hadoop/dfs/local/urls/ -dir
> /home2/nazgul/hadoop/dfs/local/crawl-data/ -depth 3
> 11/02/03 19:13:29 INFO security.Groups: Group mapping
> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
> cacheTimeout=300000
> 11/02/03 19:13:29 INFO crawl.Crawl: crawl started in:
> /home2/internal/sysadmin/nazgul/hadoop/dfs/local/crawl-data
> 11/02/03 19:13:29 INFO crawl.Crawl: rootUrlDir =
> /home2/nazgul/hadoop/dfs/local/urls
> 11/02/03 19:13:29 INFO crawl.Crawl: threads = 2
> 11/02/03 19:13:29 INFO crawl.Crawl: depth = 3
> 11/02/03 19:13:29 INFO crawl.Crawl: indexer=lucene
> 11/02/03 19:13:29 INFO crawl.Injector: Injector: starting at 2011-02-03
> 19:13:29
> 11/02/03 19:13:29 INFO crawl.Injector: Injector: crawlDb:
> /home2/nazgul/hadoop/dfs/local/crawl-data/crawldb
> 11/02/03 19:13:29 INFO crawl.Injector: Injector: urlDir:
> /home2/nazgul/hadoop/dfs/local/urls
> 11/02/03 19:13:29 WARN conf.Configuration: mapred.temp.dir is deprecated.
> Instead, use mapreduce.cluster.temp.dir
> 11/02/03 19:13:29 INFO crawl.Injector: Injector: Converting injected urls
> to crawl db entries.
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/codehaus/jackson/map/JsonMappingException
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:960)
>         at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:534)
>         at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:779)
>         at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
>         at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)
> Caused by: java.lang.ClassNotFoundException:
> org.codehaus.jackson.map.JsonMappingException
>         at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
>         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>         at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>
>
>
>
> --
> ---
> Rishi Pathak
> National PARAM Supercomputing Facility
> C-DAC, Pune, India
>
>
>


-- 
---
Rishi Pathak
National PARAM Supercomputing Facility
C-DAC, Pune, India