You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by "shubham.gupta" <sh...@orkash.com> on 2016/10/14 10:15:42 UTC
Injector and Generator Job Failing
Hey
Whenever i run the nutch application, only the injector and generate job
fails.
The path of the plugin folders in conf/nutch-site.xml is correct.
The following error occurs:
INFO mapreduce.Job: Job job_1476273924585_1272 failed with state FAILED
due to: Task failed task_1476273924585_1272_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
INFO mapreduce.Job: Counters: 9
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=17594276
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=2513468
Total vcore-seconds taken by all map tasks=2513468
Total megabyte-seconds taken by all map tasks=8978107696
ERROR crawl.*InjectorJob*: InjectorJob: java.lang.RuntimeException: job
failed: name=inject jobid=job_1476273924585_1272
at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:231)
at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
mapreduce.Job: Job job_1476273924585_0973 failed with state FAILED due
to: Task failed task_1476273924585_0973_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
INFO mapreduce.Job: Counters: 8
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=4
Total time spent by all maps in occupied slots
(ms)=17585463
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=2512209
Total vcore-seconds taken by all map tasks=2512209
Total megabyte-seconds taken by all map tasks=8973610548
16/10/14 04:15:49 ERROR crawl.*GeneratorJob*: GeneratorJob:
java.lang.RuntimeException: job failed: name=generate:
1476396190-679771226, jobid=job_1476273924585_0973
at
org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:227)
at
org.apache.nutch.crawl.GeneratorJob.generate(GeneratorJob.java:256)
at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:322)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.nutch.crawl.GeneratorJob.main(GeneratorJob.java:330)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Any suggestions.
--
Thanks and Regards,
Shubham Gupta
RE: Injector and Generator Job Failing
Posted by Markus Jelsma <ma...@openindex.io>.
Well, ERROR log is probably all we need to see, this is the application's main stdout which doesnt tell what is wrong.
M.
-----Original message-----
> From:shubham.gupta <sh...@orkash.com>
> Sent: Friday 14th October 2016 12:53
> To: user@nutch.apache.org
> Subject: Re: Injector and Generator Job Failing
>
> Due to huge amount of hadoop logging I had only allowed the logging of
> ERROR messages and above of both hadoop and nutch. Also, enabled the
> periodic deletion of logs as lot of disk was being utilized. So, I am
> kind of in the dark here.
>
> Thanks and Regards,
> Shubham Gupta
>
> On Friday 14 October 2016 04:11 PM, Markus Jelsma wrote:
> > Check the logs, this only tells you that i failed, not why.
> > M.
> >
> >
> >
> > -----Original message-----
> >> From:shubham.gupta <sh...@orkash.com>
> >> Sent: Friday 14th October 2016 12:15
> >> To: user@nutch.apache.org
> >> Subject: Injector and Generator Job Failing
> >>
> >> Hey
> >>
> >> Whenever i run the nutch application, only the injector and generate job
> >> fails.
> >> The path of the plugin folders in conf/nutch-site.xml is correct.
> >>
> >> The following error occurs:
> >> INFO mapreduce.Job: Job job_1476273924585_1272 failed with state FAILED
> >> due to: Task failed task_1476273924585_1272_m_000000
> >> Job failed as tasks failed. failedMaps:1 failedReduces:0
> >>
> >> INFO mapreduce.Job: Counters: 9
> >> Job Counters
> >> Failed map tasks=4
> >> Launched map tasks=4
> >> Other local map tasks=3
> >> Data-local map tasks=1
> >> Total time spent by all maps in occupied slots (ms)=17594276
> >> Total time spent by all reduces in occupied slots (ms)=0
> >> Total time spent by all map tasks (ms)=2513468
> >> Total vcore-seconds taken by all map tasks=2513468
> >> Total megabyte-seconds taken by all map tasks=8978107696
> >> ERROR crawl.*InjectorJob*: InjectorJob: java.lang.RuntimeException: job
> >> failed: name=inject jobid=job_1476273924585_1272
> >> at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
> >> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:231)
> >> at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
> >> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >> at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> >>
> >> mapreduce.Job: Job job_1476273924585_0973 failed with state FAILED due
> >> to: Task failed task_1476273924585_0973_m_000000
> >> Job failed as tasks failed. failedMaps:1 failedReduces:0
> >>
> >> INFO mapreduce.Job: Counters: 8
> >> Job Counters
> >> Failed map tasks=4
> >> Launched map tasks=4
> >> Other local map tasks=4
> >> Total time spent by all maps in occupied slots
> >> (ms)=17585463
> >> Total time spent by all reduces in occupied slots (ms)=0
> >> Total time spent by all map tasks (ms)=2512209
> >> Total vcore-seconds taken by all map tasks=2512209
> >> Total megabyte-seconds taken by all map tasks=8973610548
> >> 16/10/14 04:15:49 ERROR crawl.*GeneratorJob*: GeneratorJob:
> >> java.lang.RuntimeException: job failed: name=generate:
> >> 1476396190-679771226, jobid=job_1476273924585_0973
> >> at
> >> org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
> >> at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:227)
> >> at
> >> org.apache.nutch.crawl.GeneratorJob.generate(GeneratorJob.java:256)
> >> at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:322)
> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >> at org.apache.nutch.crawl.GeneratorJob.main(GeneratorJob.java:330)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> >>
> >> Any suggestions.
> >>
> >> --
> >> Thanks and Regards,
> >> Shubham Gupta
> >>
> >>
>
>
Re: Injector and Generator Job Failing
Posted by "shubham.gupta" <sh...@orkash.com>.
Due to huge amount of hadoop logging I had only allowed the logging of
ERROR messages and above of both hadoop and nutch. Also, enabled the
periodic deletion of logs as lot of disk was being utilized. So, I am
kind of in the dark here.
Thanks and Regards,
Shubham Gupta
On Friday 14 October 2016 04:11 PM, Markus Jelsma wrote:
> Check the logs, this only tells you that i failed, not why.
> M.
>
>
>
> -----Original message-----
>> From:shubham.gupta <sh...@orkash.com>
>> Sent: Friday 14th October 2016 12:15
>> To: user@nutch.apache.org
>> Subject: Injector and Generator Job Failing
>>
>> Hey
>>
>> Whenever i run the nutch application, only the injector and generate job
>> fails.
>> The path of the plugin folders in conf/nutch-site.xml is correct.
>>
>> The following error occurs:
>> INFO mapreduce.Job: Job job_1476273924585_1272 failed with state FAILED
>> due to: Task failed task_1476273924585_1272_m_000000
>> Job failed as tasks failed. failedMaps:1 failedReduces:0
>>
>> INFO mapreduce.Job: Counters: 9
>> Job Counters
>> Failed map tasks=4
>> Launched map tasks=4
>> Other local map tasks=3
>> Data-local map tasks=1
>> Total time spent by all maps in occupied slots (ms)=17594276
>> Total time spent by all reduces in occupied slots (ms)=0
>> Total time spent by all map tasks (ms)=2513468
>> Total vcore-seconds taken by all map tasks=2513468
>> Total megabyte-seconds taken by all map tasks=8978107696
>> ERROR crawl.*InjectorJob*: InjectorJob: java.lang.RuntimeException: job
>> failed: name=inject jobid=job_1476273924585_1272
>> at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
>> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:231)
>> at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
>> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>
>> mapreduce.Job: Job job_1476273924585_0973 failed with state FAILED due
>> to: Task failed task_1476273924585_0973_m_000000
>> Job failed as tasks failed. failedMaps:1 failedReduces:0
>>
>> INFO mapreduce.Job: Counters: 8
>> Job Counters
>> Failed map tasks=4
>> Launched map tasks=4
>> Other local map tasks=4
>> Total time spent by all maps in occupied slots
>> (ms)=17585463
>> Total time spent by all reduces in occupied slots (ms)=0
>> Total time spent by all map tasks (ms)=2512209
>> Total vcore-seconds taken by all map tasks=2512209
>> Total megabyte-seconds taken by all map tasks=8973610548
>> 16/10/14 04:15:49 ERROR crawl.*GeneratorJob*: GeneratorJob:
>> java.lang.RuntimeException: job failed: name=generate:
>> 1476396190-679771226, jobid=job_1476273924585_0973
>> at
>> org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
>> at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:227)
>> at
>> org.apache.nutch.crawl.GeneratorJob.generate(GeneratorJob.java:256)
>> at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:322)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.nutch.crawl.GeneratorJob.main(GeneratorJob.java:330)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>>
>> Any suggestions.
>>
>> --
>> Thanks and Regards,
>> Shubham Gupta
>>
>>
RE: Injector and Generator Job Failing
Posted by Markus Jelsma <ma...@openindex.io>.
Check the logs, this only tells you that i failed, not why.
M.
-----Original message-----
> From:shubham.gupta <sh...@orkash.com>
> Sent: Friday 14th October 2016 12:15
> To: user@nutch.apache.org
> Subject: Injector and Generator Job Failing
>
> Hey
>
> Whenever i run the nutch application, only the injector and generate job
> fails.
> The path of the plugin folders in conf/nutch-site.xml is correct.
>
> The following error occurs:
> INFO mapreduce.Job: Job job_1476273924585_1272 failed with state FAILED
> due to: Task failed task_1476273924585_1272_m_000000
> Job failed as tasks failed. failedMaps:1 failedReduces:0
>
> INFO mapreduce.Job: Counters: 9
> Job Counters
> Failed map tasks=4
> Launched map tasks=4
> Other local map tasks=3
> Data-local map tasks=1
> Total time spent by all maps in occupied slots (ms)=17594276
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=2513468
> Total vcore-seconds taken by all map tasks=2513468
> Total megabyte-seconds taken by all map tasks=8978107696
> ERROR crawl.*InjectorJob*: InjectorJob: java.lang.RuntimeException: job
> failed: name=inject jobid=job_1476273924585_1272
> at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:231)
> at org.apache.nutch.crawl.InjectorJob.inject(InjectorJob.java:252)
> at org.apache.nutch.crawl.InjectorJob.run(InjectorJob.java:275)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.nutch.crawl.InjectorJob.main(InjectorJob.java:284)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
> mapreduce.Job: Job job_1476273924585_0973 failed with state FAILED due
> to: Task failed task_1476273924585_0973_m_000000
> Job failed as tasks failed. failedMaps:1 failedReduces:0
>
> INFO mapreduce.Job: Counters: 8
> Job Counters
> Failed map tasks=4
> Launched map tasks=4
> Other local map tasks=4
> Total time spent by all maps in occupied slots
> (ms)=17585463
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=2512209
> Total vcore-seconds taken by all map tasks=2512209
> Total megabyte-seconds taken by all map tasks=8973610548
> 16/10/14 04:15:49 ERROR crawl.*GeneratorJob*: GeneratorJob:
> java.lang.RuntimeException: job failed: name=generate:
> 1476396190-679771226, jobid=job_1476273924585_0973
> at
> org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:120)
> at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:227)
> at
> org.apache.nutch.crawl.GeneratorJob.generate(GeneratorJob.java:256)
> at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:322)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.nutch.crawl.GeneratorJob.main(GeneratorJob.java:330)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
>
> Any suggestions.
>
> --
> Thanks and Regards,
> Shubham Gupta
>
>