You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Massimo Schiavon <ms...@volunia.com> on 2010/10/12 12:59:04 UTC

Class loading problem running Nutch on existing Hadoop cluster.

I've created a custom .job file (renamed jar file) containing:

- a lib folder with all the jars present in the Nutch lib dir (except 
all those already present in my Hadoop installation lib dir) and the 
Nutch-1.2.jar file

- a class that launch programatically some jobs (some Nutch jobs and 
some custom written) in a similar way like org.apache.nutch.crawl.Crawl 
does.

Trying to crawl starting from a single url  (for simplicity) the jobs 
are all completed but none of the pages are parsed.

This is an extract from the logs (with some censored content):

2010-10-12 11:37:42,517 INFO org.apache.nutch.fetcher.Fetcher: 
-finishing thread FetcherThread, activeThreads=1
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http: 
http.proxy.host = null
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http: 
http.proxy.port = 8080
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http: 
http.timeout = 10000
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http: 
http.content.limit = 65536
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http: 
http.agent = *****
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http: 
http.accept.language = en-us,en-gb,en;q=0.7,*;q=0.3
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http: 
protocol.plugin.check.blocking = false
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http: 
protocol.plugin.check.robots = false
2010-10-12 11:37:43,530 INFO org.apache.nutch.fetcher.Fetcher: 
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2010-10-12 11:37:43,978 INFO 
org.apache.nutch.protocol.http.api.RobotRulesParser: Couldn't get 
robots.txt for http://www.*****: java.lang.NoClassDefFoundError: 
org/apache/commons/lang/StringUtils
2010-10-12 11:37:44,464 ERROR org.apache.nutch.protocol.http.Http: 
java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
2010-10-12 11:37:44,464 ERROR org.apache.nutch.protocol.http.Http: at 
org.apache.nutch.metadata.SpellCheckedMetadata.getNormalizedName(SpellCheckedMetadata.java:117)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at 
org.apache.nutch.metadata.SpellCheckedMetadata.set(SpellCheckedMetadata.java:147)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at 
org.apache.nutch.protocol.http.HttpResponse.processHeaderLine(HttpResponse.java:363)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at 
org.apache.nutch.protocol.http.HttpResponse.parseHeaders(HttpResponse.java:396)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at 
org.apache.nutch.protocol.http.HttpResponse.<init>(HttpResponse.java:154)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at 
org.apache.nutch.protocol.http.Http.getResponse(Http.java:64)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at 
org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:224)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at 
org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:628)
2010-10-12 11:37:44,468 INFO org.apache.nutch.fetcher.Fetcher: fetch of 
http://www.***** failed with: java.lang.NoClassDefFoundError: 
org/apache/commons/lang/StringUtils

Seems that the jvm started for the Fetcher task execution couldn't load 
StringUtils class. The weird thing is that the commons-lang jar file is 
included into the job file in the lib dir.

Starting mapred daemon including all the libraries in the Hadoop 
classpath resolves the problem but I don't want to do that way because I 
want to keep Hadoop installation clean (and I don't want to restart 
mapred to add a library)

I've tried the same process with two different configurations with the 
same results:

Ubuntu Linux 10.04 desktop
kernel 2.6.32-25-generic
jvm 1.6.0_20 64-bit
Hadoop 0.20.2 in single node configuration
Nutch 1.2

and

CentOS 5.5
kernel 2.6.18-194.el5
jvm 1.6.0_21 64-bit
Hadoop 0.20.2 in a three node mapred cluster
Nutch 1.2


Thanks
Massimo