You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by rishi pathak <ma...@gmail.com> on 2011/02/03 15:06:20 UTC
Upgrade to hadoop-0.21.0
Hi,
I tried upgrading upgrading to hadoop-0.21.0 in nutch-1.2. 'ant
package' does not report any errors and build is sucessfull but nutch crawl
is failing with following errors:
./bin/nutch crawl /home2/nazgul/hadoop/dfs/local/urls/ -dir
/home2/nazgul/hadoop/dfs/local/crawl-data/ -depth 3
11/02/03 19:13:29 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
11/02/03 19:13:29 INFO crawl.Crawl: crawl started in:
/home2/internal/sysadmin/nazgul/hadoop/dfs/local/crawl-data
11/02/03 19:13:29 INFO crawl.Crawl: rootUrlDir =
/home2/nazgul/hadoop/dfs/local/urls
11/02/03 19:13:29 INFO crawl.Crawl: threads = 2
11/02/03 19:13:29 INFO crawl.Crawl: depth = 3
11/02/03 19:13:29 INFO crawl.Crawl: indexer=lucene
11/02/03 19:13:29 INFO crawl.Injector: Injector: starting at 2011-02-03
19:13:29
11/02/03 19:13:29 INFO crawl.Injector: Injector: crawlDb:
/home2/nazgul/hadoop/dfs/local/crawl-data/crawldb
11/02/03 19:13:29 INFO crawl.Injector: Injector: urlDir:
/home2/nazgul/hadoop/dfs/local/urls
11/02/03 19:13:29 WARN conf.Configuration: mapred.temp.dir is deprecated.
Instead, use mapreduce.cluster.temp.dir
11/02/03 19:13:29 INFO crawl.Injector: Injector: Converting injected urls to
crawl db entries.
Exception in thread "main" java.lang.NoClassDefFoundError:
org/codehaus/jackson/map/JsonMappingException
at org.apache.hadoop.mapreduce.Job.submit(Job.java:960)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:534)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:779)
at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)
Caused by: java.lang.ClassNotFoundException:
org.codehaus.jackson.map.JsonMappingException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
--
---
Rishi Pathak
National PARAM Supercomputing Facility
C-DAC, Pune, India
move from a single node to 4 node structure
Posted by Patricio Galeas <pg...@yahoo.de>.
Hi,
running Nutch-1.2 on a single machine I produce a 2GB index (157 segments, slice
50.000).
Because the performance is low, I would like to test further crawls in Amazon
EC2.
Q1: If I start with 4 nodes, should I divide the number of the segments
proportionally for each node, and then start a new crawl?
Q2: Analysing the actual log files (single machine), I found out that the most
time consuming are Index Database (13 hours), Actualize Crawl Database (11
hours), Actualize LinkDB (7 hours 24), etc. Assuming the 4 node structure, do
the index and database actualization time proportionally decrease?
Thanks
Patricio
Re: Upgrade to hadoop-0.21.0
Posted by rishi pathak <ma...@gmail.com>.
Following steps were followed and it seems like working:
#Do an upgrade as described in
http://wiki.apache.org/nutch/Upgrading_Hadoop
#Download and put jackson-all-1.7.2 in $NUTCH_HOME/lib from
http://jackson.codehaus.org/
#Download and put avro-1.3.3 in $NUTCH_HOME/lib from
avro.apache.org(avro-1.4.x breaks)
On Thu, Feb 3, 2011 at 7:36 PM, rishi pathak <ma...@gmail.com>wrote:
> Hi,
> I tried upgrading upgrading to hadoop-0.21.0 in nutch-1.2. 'ant
> package' does not report any errors and build is sucessfull but nutch crawl
> is failing with following errors:
>
>
>
> ./bin/nutch crawl /home2/nazgul/hadoop/dfs/local/urls/ -dir
> /home2/nazgul/hadoop/dfs/local/crawl-data/ -depth 3
> 11/02/03 19:13:29 INFO security.Groups: Group mapping
> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
> cacheTimeout=300000
> 11/02/03 19:13:29 INFO crawl.Crawl: crawl started in:
> /home2/internal/sysadmin/nazgul/hadoop/dfs/local/crawl-data
> 11/02/03 19:13:29 INFO crawl.Crawl: rootUrlDir =
> /home2/nazgul/hadoop/dfs/local/urls
> 11/02/03 19:13:29 INFO crawl.Crawl: threads = 2
> 11/02/03 19:13:29 INFO crawl.Crawl: depth = 3
> 11/02/03 19:13:29 INFO crawl.Crawl: indexer=lucene
> 11/02/03 19:13:29 INFO crawl.Injector: Injector: starting at 2011-02-03
> 19:13:29
> 11/02/03 19:13:29 INFO crawl.Injector: Injector: crawlDb:
> /home2/nazgul/hadoop/dfs/local/crawl-data/crawldb
> 11/02/03 19:13:29 INFO crawl.Injector: Injector: urlDir:
> /home2/nazgul/hadoop/dfs/local/urls
> 11/02/03 19:13:29 WARN conf.Configuration: mapred.temp.dir is deprecated.
> Instead, use mapreduce.cluster.temp.dir
> 11/02/03 19:13:29 INFO crawl.Injector: Injector: Converting injected urls
> to crawl db entries.
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/codehaus/jackson/map/JsonMappingException
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:960)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:534)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:779)
> at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
> at org.apache.nutch.crawl.Crawl.main(Crawl.java:124)
> Caused by: java.lang.ClassNotFoundException:
> org.codehaus.jackson.map.JsonMappingException
> at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
>
>
>
>
> --
> ---
> Rishi Pathak
> National PARAM Supercomputing Facility
> C-DAC, Pune, India
>
>
>
--
---
Rishi Pathak
National PARAM Supercomputing Facility
C-DAC, Pune, India