You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Massimo Schiavon <ms...@volunia.com> on 2010/10/12 12:59:04 UTC
Class loading problem running Nutch on existing Hadoop cluster.
I've created a custom .job file (renamed jar file) containing:
- a lib folder with all the jars present in the Nutch lib dir (except
all those already present in my Hadoop installation lib dir) and the
Nutch-1.2.jar file
- a class that launch programatically some jobs (some Nutch jobs and
some custom written) in a similar way like org.apache.nutch.crawl.Crawl
does.
Trying to crawl starting from a single url (for simplicity) the jobs
are all completed but none of the pages are parsed.
This is an extract from the logs (with some censored content):
2010-10-12 11:37:42,517 INFO org.apache.nutch.fetcher.Fetcher:
-finishing thread FetcherThread, activeThreads=1
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http:
http.proxy.host = null
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http:
http.proxy.port = 8080
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http:
http.timeout = 10000
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http:
http.content.limit = 65536
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http:
http.agent = *****
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http:
http.accept.language = en-us,en-gb,en;q=0.7,*;q=0.3
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http:
protocol.plugin.check.blocking = false
2010-10-12 11:37:42,517 INFO org.apache.nutch.protocol.http.Http:
protocol.plugin.check.robots = false
2010-10-12 11:37:43,530 INFO org.apache.nutch.fetcher.Fetcher:
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0
2010-10-12 11:37:43,978 INFO
org.apache.nutch.protocol.http.api.RobotRulesParser: Couldn't get
robots.txt for http://www.*****: java.lang.NoClassDefFoundError:
org/apache/commons/lang/StringUtils
2010-10-12 11:37:44,464 ERROR org.apache.nutch.protocol.http.Http:
java.lang.NoClassDefFoundError: org/apache/commons/lang/StringUtils
2010-10-12 11:37:44,464 ERROR org.apache.nutch.protocol.http.Http: at
org.apache.nutch.metadata.SpellCheckedMetadata.getNormalizedName(SpellCheckedMetadata.java:117)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at
org.apache.nutch.metadata.SpellCheckedMetadata.set(SpellCheckedMetadata.java:147)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at
org.apache.nutch.protocol.http.HttpResponse.processHeaderLine(HttpResponse.java:363)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at
org.apache.nutch.protocol.http.HttpResponse.parseHeaders(HttpResponse.java:396)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at
org.apache.nutch.protocol.http.HttpResponse.<init>(HttpResponse.java:154)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at
org.apache.nutch.protocol.http.Http.getResponse(Http.java:64)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at
org.apache.nutch.protocol.http.api.HttpBase.getProtocolOutput(HttpBase.java:224)
2010-10-12 11:37:44,465 ERROR org.apache.nutch.protocol.http.Http: at
org.apache.nutch.fetcher.Fetcher$FetcherThread.run(Fetcher.java:628)
2010-10-12 11:37:44,468 INFO org.apache.nutch.fetcher.Fetcher: fetch of
http://www.***** failed with: java.lang.NoClassDefFoundError:
org/apache/commons/lang/StringUtils
Seems that the jvm started for the Fetcher task execution couldn't load
StringUtils class. The weird thing is that the commons-lang jar file is
included into the job file in the lib dir.
Starting mapred daemon including all the libraries in the Hadoop
classpath resolves the problem but I don't want to do that way because I
want to keep Hadoop installation clean (and I don't want to restart
mapred to add a library)
I've tried the same process with two different configurations with the
same results:
Ubuntu Linux 10.04 desktop
kernel 2.6.32-25-generic
jvm 1.6.0_20 64-bit
Hadoop 0.20.2 in single node configuration
Nutch 1.2
and
CentOS 5.5
kernel 2.6.18-194.el5
jvm 1.6.0_21 64-bit
Hadoop 0.20.2 in a three node mapred cluster
Nutch 1.2
Thanks
Massimo