You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Sznajder ForMailingList <bs...@gmail.com> on 2013/07/02 15:25:06 UTC
Distributed mode and java/lang/OutOfMemoryError
Hi,
I am running Nutch 1.7 on a cluster of 6 nodes.
I tempted to launch the bin/crawl script in this configuration and I am
getting a very strange error (an error I did not get in the local mode):
13/07/02 16:04:23 INFO fetcher.Fetcher: Fetcher Timelimit set for :
1372781063368
13/07/02 16:04:25 INFO mapred.FileInputFormat: Total input paths to process
: 1
13/07/02 16:04:27 INFO mapred.JobClient: Running job: job_201207110955_1173
13/07/02 16:04:28 INFO mapred.JobClient: map 0% reduce 0%
13/07/02 16:04:46 INFO mapred.JobClient: map 100% reduce 0%
13/07/02 16:05:01 INFO mapred.JobClient: Task Id :
attempt_201207110955_1173_r_000000_0, Status : FAILED
Error: null
attempt_201207110955_1173_r_000000_0: JVMDUMP006I Processing dump event
"systhrow", detail "java/lang/OutOfMemoryError" - please wait.
attempt_201207110955_1173_r_000000_0: JVMDUMP032I JVM requested Heap dump
using
'/local/hadoop/tmp/mapred/local/taskTracker/jobcache/job_201207110955_1173/attempt_201207110955_1173_r_000000_0/work/heapdump.20130702.160821.24995.0001.phd'
in response to an event
Do you have an idea how to solve that?
Benjamin
Re: Distributed mode and java/lang/OutOfMemoryError
Posted by Julien Nioche <li...@gmail.com>.
Neither. You leave it in $NUTCH/conf and compile a job file with 'ant job'
which gets used from runtime/deploy/bin
BTW new users should at least do the basic Hadoop tutorial.
On 2 July 2013 16:23, Lewis John Mcgibbney <le...@gmail.com>wrote:
> I'm assuming that if your running on an established hadoop cluster you will
> wish to keep it over there.
> Lewis
>
> On Tuesday, July 2, 2013, Sznajder ForMailingList <
> bs4mailinglist@gmail.com>
> wrote:
> > Thanks
> > Can I copy this file to my $NUTCH/conf directory, or must I keep it in
> the
> > $HADOOP/conf directory?
> >
> > Benjamin
> >
> >
> > On Tue, Jul 2, 2013 at 5:10 PM, Lewis John Mcgibbney <
> > lewis.mcgibbney@gmail.com> wrote:
> >
> >> in mapred-site.xml
> >> It is your Mapreduce configuration override.
> >> hth
> >>
> >> On Tuesday, July 2, 2013, Sznajder ForMailingList <
> >> bs4mailinglist@gmail.com>
> >> wrote:
> >> > Thanks a lot Markus!
> >> >
> >> > Where do we define this parameter, please?
> >> >
> >> > Benjamin
> >> >
> >> >
> >> > On Tue, Jul 2, 2013 at 4:28 PM, Markus Jelsma <
> >> markus.jelsma@openindex.io
> >> >wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> Increase your memory in the task trackers by setting your Xmx in
> >> >> mapred.map.child.java.opts.
> >> >>
> >> >> Cheers
> >> >>
> >> >>
> >> >>
> >> >> -----Original message-----
> >> >> > From:Sznajder ForMailingList <bs...@gmail.com>
> >> >> > Sent: Tuesday 2nd July 2013 15:25
> >> >> > To: user@nutch.apache.org
> >> >> > Subject: Distributed mode and java/lang/OutOfMemoryError
> >> >> >
> >> >> > Hi,
> >> >> >
> >> >> > I am running Nutch 1.7 on a cluster of 6 nodes.
> >> >> >
> >> >> > I tempted to launch the bin/crawl script in this configuration and
> I
> >> am
> >> >> > getting a very strange error (an error I did not get in the local
> >> mode):
> >> >> >
> >> >> > 13/07/02 16:04:23 INFO fetcher.Fetcher: Fetcher Timelimit set for :
> >> >> > 1372781063368
> >> >> > 13/07/02 16:04:25 INFO mapred.FileInputFormat: Total input paths to
> >> >> process
> >> >> > : 1
> >> >> > 13/07/02 16:04:27 INFO mapred.JobClient: Running job:
> >> >> job_201207110955_1173
> >> >> > 13/07/02 16:04:28 INFO mapred.JobClient: map 0% reduce 0%
> >> >> > 13/07/02 16:04:46 INFO mapred.JobClient: map 100% reduce 0%
> >> >> > 13/07/02 16:05:01 INFO mapred.JobClient: Task Id :
> >> >> > attempt_201207110955_1173_r_000000_0, Status : FAILED
> >> >> > Error: null
> >> >> > attempt_201207110955_1173_r_000000_0: JVMDUMP006I Processing dump
> >> event
> >> >> > "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
> >> >> > attempt_201207110955_1173_r_000000_0: JVMDUMP032I JVM requested
> Heap
> >> dump
> >> >> > using
> >> >> >
> >> >>
> >>
> >>
>
> '/local/hadoop/tmp/mapred/local/taskTracker/jobcache/job_201207110955_1173/attempt_201207110955_1173_r_000000_0/work/heapdump.20130702.160821.24995.0001.phd'
> >> >> > in response to an event
> >> >> >
> >> >> >
> >> >> > Do you have an idea how to solve that?
> >> >> >
> >> >> > Benjamin
> >> >> >
> >> >>
> >> >
> >>
> >> --
> >> *Lewis*
> >>
> >
>
> --
> *Lewis*
>
--
*
*Open Source Solutions for Text Engineering
http://digitalpebble.blogspot.com/
http://www.digitalpebble.com
http://twitter.com/digitalpebble
Re: Distributed mode and java/lang/OutOfMemoryError
Posted by Lewis John Mcgibbney <le...@gmail.com>.
I'm assuming that if your running on an established hadoop cluster you will
wish to keep it over there.
Lewis
On Tuesday, July 2, 2013, Sznajder ForMailingList <bs...@gmail.com>
wrote:
> Thanks
> Can I copy this file to my $NUTCH/conf directory, or must I keep it in the
> $HADOOP/conf directory?
>
> Benjamin
>
>
> On Tue, Jul 2, 2013 at 5:10 PM, Lewis John Mcgibbney <
> lewis.mcgibbney@gmail.com> wrote:
>
>> in mapred-site.xml
>> It is your Mapreduce configuration override.
>> hth
>>
>> On Tuesday, July 2, 2013, Sznajder ForMailingList <
>> bs4mailinglist@gmail.com>
>> wrote:
>> > Thanks a lot Markus!
>> >
>> > Where do we define this parameter, please?
>> >
>> > Benjamin
>> >
>> >
>> > On Tue, Jul 2, 2013 at 4:28 PM, Markus Jelsma <
>> markus.jelsma@openindex.io
>> >wrote:
>> >
>> >> Hi,
>> >>
>> >> Increase your memory in the task trackers by setting your Xmx in
>> >> mapred.map.child.java.opts.
>> >>
>> >> Cheers
>> >>
>> >>
>> >>
>> >> -----Original message-----
>> >> > From:Sznajder ForMailingList <bs...@gmail.com>
>> >> > Sent: Tuesday 2nd July 2013 15:25
>> >> > To: user@nutch.apache.org
>> >> > Subject: Distributed mode and java/lang/OutOfMemoryError
>> >> >
>> >> > Hi,
>> >> >
>> >> > I am running Nutch 1.7 on a cluster of 6 nodes.
>> >> >
>> >> > I tempted to launch the bin/crawl script in this configuration and I
>> am
>> >> > getting a very strange error (an error I did not get in the local
>> mode):
>> >> >
>> >> > 13/07/02 16:04:23 INFO fetcher.Fetcher: Fetcher Timelimit set for :
>> >> > 1372781063368
>> >> > 13/07/02 16:04:25 INFO mapred.FileInputFormat: Total input paths to
>> >> process
>> >> > : 1
>> >> > 13/07/02 16:04:27 INFO mapred.JobClient: Running job:
>> >> job_201207110955_1173
>> >> > 13/07/02 16:04:28 INFO mapred.JobClient: map 0% reduce 0%
>> >> > 13/07/02 16:04:46 INFO mapred.JobClient: map 100% reduce 0%
>> >> > 13/07/02 16:05:01 INFO mapred.JobClient: Task Id :
>> >> > attempt_201207110955_1173_r_000000_0, Status : FAILED
>> >> > Error: null
>> >> > attempt_201207110955_1173_r_000000_0: JVMDUMP006I Processing dump
>> event
>> >> > "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
>> >> > attempt_201207110955_1173_r_000000_0: JVMDUMP032I JVM requested Heap
>> dump
>> >> > using
>> >> >
>> >>
>>
>>
'/local/hadoop/tmp/mapred/local/taskTracker/jobcache/job_201207110955_1173/attempt_201207110955_1173_r_000000_0/work/heapdump.20130702.160821.24995.0001.phd'
>> >> > in response to an event
>> >> >
>> >> >
>> >> > Do you have an idea how to solve that?
>> >> >
>> >> > Benjamin
>> >> >
>> >>
>> >
>>
>> --
>> *Lewis*
>>
>
--
*Lewis*
Re: Distributed mode and java/lang/OutOfMemoryError
Posted by Sznajder ForMailingList <bs...@gmail.com>.
Thanks
Can I copy this file to my $NUTCH/conf directory, or must I keep it in the
$HADOOP/conf directory?
Benjamin
On Tue, Jul 2, 2013 at 5:10 PM, Lewis John Mcgibbney <
lewis.mcgibbney@gmail.com> wrote:
> in mapred-site.xml
> It is your Mapreduce configuration override.
> hth
>
> On Tuesday, July 2, 2013, Sznajder ForMailingList <
> bs4mailinglist@gmail.com>
> wrote:
> > Thanks a lot Markus!
> >
> > Where do we define this parameter, please?
> >
> > Benjamin
> >
> >
> > On Tue, Jul 2, 2013 at 4:28 PM, Markus Jelsma <
> markus.jelsma@openindex.io
> >wrote:
> >
> >> Hi,
> >>
> >> Increase your memory in the task trackers by setting your Xmx in
> >> mapred.map.child.java.opts.
> >>
> >> Cheers
> >>
> >>
> >>
> >> -----Original message-----
> >> > From:Sznajder ForMailingList <bs...@gmail.com>
> >> > Sent: Tuesday 2nd July 2013 15:25
> >> > To: user@nutch.apache.org
> >> > Subject: Distributed mode and java/lang/OutOfMemoryError
> >> >
> >> > Hi,
> >> >
> >> > I am running Nutch 1.7 on a cluster of 6 nodes.
> >> >
> >> > I tempted to launch the bin/crawl script in this configuration and I
> am
> >> > getting a very strange error (an error I did not get in the local
> mode):
> >> >
> >> > 13/07/02 16:04:23 INFO fetcher.Fetcher: Fetcher Timelimit set for :
> >> > 1372781063368
> >> > 13/07/02 16:04:25 INFO mapred.FileInputFormat: Total input paths to
> >> process
> >> > : 1
> >> > 13/07/02 16:04:27 INFO mapred.JobClient: Running job:
> >> job_201207110955_1173
> >> > 13/07/02 16:04:28 INFO mapred.JobClient: map 0% reduce 0%
> >> > 13/07/02 16:04:46 INFO mapred.JobClient: map 100% reduce 0%
> >> > 13/07/02 16:05:01 INFO mapred.JobClient: Task Id :
> >> > attempt_201207110955_1173_r_000000_0, Status : FAILED
> >> > Error: null
> >> > attempt_201207110955_1173_r_000000_0: JVMDUMP006I Processing dump
> event
> >> > "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
> >> > attempt_201207110955_1173_r_000000_0: JVMDUMP032I JVM requested Heap
> dump
> >> > using
> >> >
> >>
>
> '/local/hadoop/tmp/mapred/local/taskTracker/jobcache/job_201207110955_1173/attempt_201207110955_1173_r_000000_0/work/heapdump.20130702.160821.24995.0001.phd'
> >> > in response to an event
> >> >
> >> >
> >> > Do you have an idea how to solve that?
> >> >
> >> > Benjamin
> >> >
> >>
> >
>
> --
> *Lewis*
>
Re: Distributed mode and java/lang/OutOfMemoryError
Posted by Lewis John Mcgibbney <le...@gmail.com>.
in mapred-site.xml
It is your Mapreduce configuration override.
hth
On Tuesday, July 2, 2013, Sznajder ForMailingList <bs...@gmail.com>
wrote:
> Thanks a lot Markus!
>
> Where do we define this parameter, please?
>
> Benjamin
>
>
> On Tue, Jul 2, 2013 at 4:28 PM, Markus Jelsma <markus.jelsma@openindex.io
>wrote:
>
>> Hi,
>>
>> Increase your memory in the task trackers by setting your Xmx in
>> mapred.map.child.java.opts.
>>
>> Cheers
>>
>>
>>
>> -----Original message-----
>> > From:Sznajder ForMailingList <bs...@gmail.com>
>> > Sent: Tuesday 2nd July 2013 15:25
>> > To: user@nutch.apache.org
>> > Subject: Distributed mode and java/lang/OutOfMemoryError
>> >
>> > Hi,
>> >
>> > I am running Nutch 1.7 on a cluster of 6 nodes.
>> >
>> > I tempted to launch the bin/crawl script in this configuration and I am
>> > getting a very strange error (an error I did not get in the local
mode):
>> >
>> > 13/07/02 16:04:23 INFO fetcher.Fetcher: Fetcher Timelimit set for :
>> > 1372781063368
>> > 13/07/02 16:04:25 INFO mapred.FileInputFormat: Total input paths to
>> process
>> > : 1
>> > 13/07/02 16:04:27 INFO mapred.JobClient: Running job:
>> job_201207110955_1173
>> > 13/07/02 16:04:28 INFO mapred.JobClient: map 0% reduce 0%
>> > 13/07/02 16:04:46 INFO mapred.JobClient: map 100% reduce 0%
>> > 13/07/02 16:05:01 INFO mapred.JobClient: Task Id :
>> > attempt_201207110955_1173_r_000000_0, Status : FAILED
>> > Error: null
>> > attempt_201207110955_1173_r_000000_0: JVMDUMP006I Processing dump event
>> > "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
>> > attempt_201207110955_1173_r_000000_0: JVMDUMP032I JVM requested Heap
dump
>> > using
>> >
>>
'/local/hadoop/tmp/mapred/local/taskTracker/jobcache/job_201207110955_1173/attempt_201207110955_1173_r_000000_0/work/heapdump.20130702.160821.24995.0001.phd'
>> > in response to an event
>> >
>> >
>> > Do you have an idea how to solve that?
>> >
>> > Benjamin
>> >
>>
>
--
*Lewis*
Re: Distributed mode and java/lang/OutOfMemoryError
Posted by Lewis John Mcgibbney <le...@gmail.com>.
in mapred-site.xml
It is your Mapreduce configuration override.
hth
On Tuesday, July 2, 2013, Sznajder ForMailingList <bs...@gmail.com>
wrote:
> Thanks a lot Markus!
>
> Where do we define this parameter, please?
>
> Benjamin
>
>
> On Tue, Jul 2, 2013 at 4:28 PM, Markus Jelsma <markus.jelsma@openindex.io
>wrote:
>
>> Hi,
>>
>> Increase your memory in the task trackers by setting your Xmx in
>> mapred.map.child.java.opts.
>>
>> Cheers
>>
>>
>>
>> -----Original message-----
>> > From:Sznajder ForMailingList <bs...@gmail.com>
>> > Sent: Tuesday 2nd July 2013 15:25
>> > To: user@nutch.apache.org
>> > Subject: Distributed mode and java/lang/OutOfMemoryError
>> >
>> > Hi,
>> >
>> > I am running Nutch 1.7 on a cluster of 6 nodes.
>> >
>> > I tempted to launch the bin/crawl script in this configuration and I am
>> > getting a very strange error (an error I did not get in the local
mode):
>> >
>> > 13/07/02 16:04:23 INFO fetcher.Fetcher: Fetcher Timelimit set for :
>> > 1372781063368
>> > 13/07/02 16:04:25 INFO mapred.FileInputFormat: Total input paths to
>> process
>> > : 1
>> > 13/07/02 16:04:27 INFO mapred.JobClient: Running job:
>> job_201207110955_1173
>> > 13/07/02 16:04:28 INFO mapred.JobClient: map 0% reduce 0%
>> > 13/07/02 16:04:46 INFO mapred.JobClient: map 100% reduce 0%
>> > 13/07/02 16:05:01 INFO mapred.JobClient: Task Id :
>> > attempt_201207110955_1173_r_000000_0, Status : FAILED
>> > Error: null
>> > attempt_201207110955_1173_r_000000_0: JVMDUMP006I Processing dump event
>> > "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
>> > attempt_201207110955_1173_r_000000_0: JVMDUMP032I JVM requested Heap
dump
>> > using
>> >
>>
'/local/hadoop/tmp/mapred/local/taskTracker/jobcache/job_201207110955_1173/attempt_201207110955_1173_r_000000_0/work/heapdump.20130702.160821.24995.0001.phd'
>> > in response to an event
>> >
>> >
>> > Do you have an idea how to solve that?
>> >
>> > Benjamin
>> >
>>
>
--
*Lewis*
Re: Distributed mode and java/lang/OutOfMemoryError
Posted by Lewis John Mcgibbney <le...@gmail.com>.
in mapred-site.xml
It is your Mapreduce configuration override.
hth
On Tuesday, July 2, 2013, Sznajder ForMailingList <bs...@gmail.com>
wrote:
> Thanks a lot Markus!
>
> Where do we define this parameter, please?
>
> Benjamin
>
>
> On Tue, Jul 2, 2013 at 4:28 PM, Markus Jelsma <markus.jelsma@openindex.io
>wrote:
>
>> Hi,
>>
>> Increase your memory in the task trackers by setting your Xmx in
>> mapred.map.child.java.opts.
>>
>> Cheers
>>
>>
>>
>> -----Original message-----
>> > From:Sznajder ForMailingList <bs...@gmail.com>
>> > Sent: Tuesday 2nd July 2013 15:25
>> > To: user@nutch.apache.org
>> > Subject: Distributed mode and java/lang/OutOfMemoryError
>> >
>> > Hi,
>> >
>> > I am running Nutch 1.7 on a cluster of 6 nodes.
>> >
>> > I tempted to launch the bin/crawl script in this configuration and I am
>> > getting a very strange error (an error I did not get in the local
mode):
>> >
>> > 13/07/02 16:04:23 INFO fetcher.Fetcher: Fetcher Timelimit set for :
>> > 1372781063368
>> > 13/07/02 16:04:25 INFO mapred.FileInputFormat: Total input paths to
>> process
>> > : 1
>> > 13/07/02 16:04:27 INFO mapred.JobClient: Running job:
>> job_201207110955_1173
>> > 13/07/02 16:04:28 INFO mapred.JobClient: map 0% reduce 0%
>> > 13/07/02 16:04:46 INFO mapred.JobClient: map 100% reduce 0%
>> > 13/07/02 16:05:01 INFO mapred.JobClient: Task Id :
>> > attempt_201207110955_1173_r_000000_0, Status : FAILED
>> > Error: null
>> > attempt_201207110955_1173_r_000000_0: JVMDUMP006I Processing dump event
>> > "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
>> > attempt_201207110955_1173_r_000000_0: JVMDUMP032I JVM requested Heap
dump
>> > using
>> >
>>
'/local/hadoop/tmp/mapred/local/taskTracker/jobcache/job_201207110955_1173/attempt_201207110955_1173_r_000000_0/work/heapdump.20130702.160821.24995.0001.phd'
>> > in response to an event
>> >
>> >
>> > Do you have an idea how to solve that?
>> >
>> > Benjamin
>> >
>>
>
--
*Lewis*
Re: Distributed mode and java/lang/OutOfMemoryError
Posted by Sznajder ForMailingList <bs...@gmail.com>.
Thanks a lot Markus!
Where do we define this parameter, please?
Benjamin
On Tue, Jul 2, 2013 at 4:28 PM, Markus Jelsma <ma...@openindex.io>wrote:
> Hi,
>
> Increase your memory in the task trackers by setting your Xmx in
> mapred.map.child.java.opts.
>
> Cheers
>
>
>
> -----Original message-----
> > From:Sznajder ForMailingList <bs...@gmail.com>
> > Sent: Tuesday 2nd July 2013 15:25
> > To: user@nutch.apache.org
> > Subject: Distributed mode and java/lang/OutOfMemoryError
> >
> > Hi,
> >
> > I am running Nutch 1.7 on a cluster of 6 nodes.
> >
> > I tempted to launch the bin/crawl script in this configuration and I am
> > getting a very strange error (an error I did not get in the local mode):
> >
> > 13/07/02 16:04:23 INFO fetcher.Fetcher: Fetcher Timelimit set for :
> > 1372781063368
> > 13/07/02 16:04:25 INFO mapred.FileInputFormat: Total input paths to
> process
> > : 1
> > 13/07/02 16:04:27 INFO mapred.JobClient: Running job:
> job_201207110955_1173
> > 13/07/02 16:04:28 INFO mapred.JobClient: map 0% reduce 0%
> > 13/07/02 16:04:46 INFO mapred.JobClient: map 100% reduce 0%
> > 13/07/02 16:05:01 INFO mapred.JobClient: Task Id :
> > attempt_201207110955_1173_r_000000_0, Status : FAILED
> > Error: null
> > attempt_201207110955_1173_r_000000_0: JVMDUMP006I Processing dump event
> > "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
> > attempt_201207110955_1173_r_000000_0: JVMDUMP032I JVM requested Heap dump
> > using
> >
> '/local/hadoop/tmp/mapred/local/taskTracker/jobcache/job_201207110955_1173/attempt_201207110955_1173_r_000000_0/work/heapdump.20130702.160821.24995.0001.phd'
> > in response to an event
> >
> >
> > Do you have an idea how to solve that?
> >
> > Benjamin
> >
>
RE: Distributed mode and java/lang/OutOfMemoryError
Posted by Markus Jelsma <ma...@openindex.io>.
Hi,
Increase your memory in the task trackers by setting your Xmx in mapred.map.child.java.opts.
Cheers
-----Original message-----
> From:Sznajder ForMailingList <bs...@gmail.com>
> Sent: Tuesday 2nd July 2013 15:25
> To: user@nutch.apache.org
> Subject: Distributed mode and java/lang/OutOfMemoryError
>
> Hi,
>
> I am running Nutch 1.7 on a cluster of 6 nodes.
>
> I tempted to launch the bin/crawl script in this configuration and I am
> getting a very strange error (an error I did not get in the local mode):
>
> 13/07/02 16:04:23 INFO fetcher.Fetcher: Fetcher Timelimit set for :
> 1372781063368
> 13/07/02 16:04:25 INFO mapred.FileInputFormat: Total input paths to process
> : 1
> 13/07/02 16:04:27 INFO mapred.JobClient: Running job: job_201207110955_1173
> 13/07/02 16:04:28 INFO mapred.JobClient: map 0% reduce 0%
> 13/07/02 16:04:46 INFO mapred.JobClient: map 100% reduce 0%
> 13/07/02 16:05:01 INFO mapred.JobClient: Task Id :
> attempt_201207110955_1173_r_000000_0, Status : FAILED
> Error: null
> attempt_201207110955_1173_r_000000_0: JVMDUMP006I Processing dump event
> "systhrow", detail "java/lang/OutOfMemoryError" - please wait.
> attempt_201207110955_1173_r_000000_0: JVMDUMP032I JVM requested Heap dump
> using
> '/local/hadoop/tmp/mapred/local/taskTracker/jobcache/job_201207110955_1173/attempt_201207110955_1173_r_000000_0/work/heapdump.20130702.160821.24995.0001.phd'
> in response to an event
>
>
> Do you have an idea how to solve that?
>
> Benjamin
>