You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Andy Xue <an...@gmail.com> on 2012/04/10 03:37:23 UTC

Run Nutch Crawl in Eclipse

Hi all:

I'd like to run Nutch Crawl in Eclipse. I have followed the
"RunNutchInEclipse" tutorial (http://wiki.apache.org/nutch/RunNutchInEclipse).
However when I tried to run the crawler, the following exception occurred:

==============================================================================
solrUrl is not set, indexing will be skipped...
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
 at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
at org.apache.nutch.crawl.Crawl.run(Crawl.java:127)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
==============================================================================


The Eclipse Run Configurations are all set according to the tutorial.
==============================================================================

Main Class: org.apache.nutch.crawl.Crawl
Program Arguments: urls -dir crawl -depth 2 -topN 10
VM arguments: -Dhadoop.log.dir=logs -Dhadoop.log.file=hadoop.log
Working directory: Default

==============================================================================
And I have set the classpath, src path, ivy path, etc according to the
tutorial too.


I can build using Ant in Eclipse. Afterwards I can successfully manually
run the crawl script using

$NUTCH_HOME/runtime/local/bin/nutch crawl urls -dir crawl -depth 2 -topN 10


And it does do the crawl process correctly and give the correct result. But
when I try to run it in Eclipse, it always fail.



Does anyone have the similar problem and knows how it can be solved?
Appreciate your time and help.


Andy

Re: Run Nutch Crawl in Eclipse

Posted by Ferdy Galema <fe...@kalooga.com>.

Hi,

Please only post to the user list, instead of both user and dev.

The "Job failed" is a general exception that is always caused by a more
specific error. You need to inspect the logs in the Eclipse console for the
specific error that caused the job to fail. (NullPointerException,
ClassNotFoundException, something like that).

Ferdy

On Tue, Apr 10, 2012 at 3:37 AM, Andy Xue <an...@gmail.com> wrote:

> Hi all:
>
> I'd like to run Nutch Crawl in Eclipse. I have followed the
> "RunNutchInEclipse" tutorial (
> http://wiki.apache.org/nutch/RunNutchInEclipse).
> However when I tried to run the crawler, the following exception occurred:
>
>
> ==============================================================================
> solrUrl is not set, indexing will be skipped...
> Exception in thread "main" java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1252)
>  at org.apache.nutch.crawl.Injector.inject(Injector.java:217)
> at org.apache.nutch.crawl.Crawl.run(Crawl.java:127)
>  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.crawl.Crawl.main(Crawl.java:55)
>
> ==============================================================================
>
>
> The Eclipse Run Configurations are all set according to the tutorial.
>
> ==============================================================================
>
> Main Class: org.apache.nutch.crawl.Crawl
> Program Arguments: urls -dir crawl -depth 2 -topN 10
> VM arguments: -Dhadoop.log.dir=logs -Dhadoop.log.file=hadoop.log
> Working directory: Default
>
>
> ==============================================================================
> And I have set the classpath, src path, ivy path, etc according to the
> tutorial too.
>
>
> I can build using Ant in Eclipse. Afterwards I can successfully manually
> run the crawl script using
>
> $NUTCH_HOME/runtime/local/bin/nutch crawl urls -dir crawl -depth 2 -topN 10
>
>
> And it does do the crawl process correctly and give the correct result. But
> when I try to run it in Eclipse, it always fail.
>
>
>
> Does anyone have the similar problem and knows how it can be solved?
> Appreciate your time and help.
>
>
> Andy
>

Re: Run Nutch Crawl in Eclipse

Posted by Mustafa Elkhiat <me...@gmail.com>.

Hi Feng,
thanks for your reply
and i do this action (changing value of property of plugin.folder)but the
problem still exist


On Sat, Mar 16, 2013 at 4:15 PM, feng lu <am...@gmail.com> wrote:

> Hi Mustafa
>
> Maybe it not found the plugin.folders in your application. you can set the
> absolute path to this property like this
>
> <property>
>   <name>plugin.folders</name>
>   <value>/home/workspace/Nutch16/runtime/local/plugins</value>
>   <description>Directories where nutch plugins are located.  Each
>   element may be a relative or absolute path.  If absolute, it is used
>   as is.  If relative, it is searched for on the classpath.</description>
> </property>
>
>
> On Sat, Mar 16, 2013 at 12:34 PM, Mustafa_elkhiat <melkhiat@gmail.com
> >wrote:
>
> > On 03/16/2013 02:55 AM, Lewis John Mcgibbney wrote:
> >
> >> Hi Mustafa,
> >> 1. Always tell us what version of the software you are using. It also
> >> helps
> >> to mention whether your are using a binary version or src.
> >> 2. Please read the responses from users@, you haven't answered which
> >> version of Nutch your using
> >> 3. As I explained, If you check out conf/log4j.properties you can alter
> >> specific tools, or the entire logging policy to obtain the coarseness
> you
> >> require. In this instance, I would either set the logging for Injector
> to
> >> DEBUG, or of course manually debug that particular job to see what is
> >> wrong.
> >>
> >> It makes it significantly more difficult to help when the information is
> >> low.
> >> Thanks
> >> Lewis
> >>
> >> On Fri, Mar 15, 2013 at 5:07 PM, Mustafa_elkhiat <me...@gmail.com>
> >> wrote:
> >>
> >>  thank you Lewis for your reply
> >>> but i'm not clearly understand you .what should i do?
> >>>
> >>>
> >>>
> >>  i use apache-nutch-1.6
> >
>
>
>
> --
> Don't Grow Old, Grow Up... :-)
>