Nutch/Hadoop: Error (FreeGenerator job did not succeed)

[Mike <mz...@gmail.com>]

Hi!

I've been using Nutch for a while but I'm new to hadoop. got a cluster with
hadoop 3.2.3 installed.

do i have to install nutch on the hadoop filesystem or can i run it
"local"? the clients don't need more from nutch than the info on master in
the command line: hadoop jar /home/debian/nutch40/lib/apache-nutch-1.19.jar
org.apache.nutch.tools.FreeGenerator -conf /home/debian/
nutch40/conf/nutch-default.xml
-Dplugin.folder=/home/debian/nutch40/plugins/
/crawl/urls//tranco-top350k-20221007.txt /home/debian/crawl/segments/

I get an error on the command:

Exception in thread "main" java.lang.RuntimeException: FreeGenerator job
did not succeed, job id: job_1665751705815_0007, job status: FAILED,
reason: Task failed task_1665751705815_0007_m_000000


Since I'm new I can't find the logs in hadoop properly yet.

Is there a guide how to install Natch (1.19) on Hadoop that I can't find?

Thanks
Mike

Re: Nutch/Hadoop: Error (FreeGenerator job did not succeed)

[Markus Jelsma <ma...@openindex.io>]

Hello,

You cannot just run Nutch's JAR like that on Hadoop, you need the large
.job file instead. If you build Nutch from source, you will get a
runtime/deploy directory. Upload its contents to a Hadoop client and run
Nutch commands using bin/nutch ... You will then automatically use the
large .job file that is on the same level as the bin directory.

Application log files on Hadoop are to be found everywhere. Select
individuel mapper or reduce subtasks, click deeper, and look to inspect
their logs. That is where the application logs are to be found.

Good luck!
Markus

Op vr 14 okt. 2022 om 16:18 schreef Mike <mz...@gmail.com>:

> Hi!
>
> I've been using Nutch for a while but I'm new to hadoop. got a cluster with
> hadoop 3.2.3 installed.
>
> do i have to install nutch on the hadoop filesystem or can i run it
> "local"? the clients don't need more from nutch than the info on master in
> the command line: hadoop jar /home/debian/nutch40/lib/apache-nutch-1.19.jar
> org.apache.nutch.tools.FreeGenerator -conf /home/debian/
> nutch40/conf/nutch-default.xml
> -Dplugin.folder=/home/debian/nutch40/plugins/
> /crawl/urls//tranco-top350k-20221007.txt /home/debian/crawl/segments/
>
> I get an error on the command:
>
> Exception in thread "main" java.lang.RuntimeException: FreeGenerator job
> did not succeed, job id: job_1665751705815_0007, job status: FAILED,
> reason: Task failed task_1665751705815_0007_m_000000
>
>
> Since I'm new I can't find the logs in hadoop properly yet.
>
> Is there a guide how to install Natch (1.19) on Hadoop that I can't find?
>
> Thanks
> Mike
>