You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by John Martyniak <jo...@beforedawnsolutions.com> on 2010/03/03 20:12:13 UTC
New version of nutch?
Does anybody have an idea of when a new version of nutch will be
availale, specifically supporting a latest version of hadoop. And
possibly hbase?
Thank you for any information.
-John
Re: Two Nutch parallel crawl with two conf folder.
Posted by Gora Mohanty <go...@srijan.in>.
On Tue, 9 Mar 2010 14:36:33 +0100
MilleBii <mi...@gmail.com> wrote:
> Never tried... Also you may want to check $NUTCH_HOME variable
> which should be different for each instance, otherwise it will
> only use one of the two conf dir.
[...]
Had meant to reply to the original poster, but had forgotten.
We have indeed run multiple instances of Nutch in separate
directories, without any problems.
I presume that you are using the crawl.sh script, or a derivative
of it. If so, as pointed out above, a likely cause of what you
are seeing is that the NUTCH_HOME variable in the script is set
to the same directory, so that the configuration from that directory
is the one picked up.
Regards,
Gora
Re: Two Nutch parallel crawl with two conf folder.
Posted by MilleBii <mi...@gmail.com>.
Never tried... Also you may want to check $NUTCH_HOME variable which
should be different for each instance, otherwise it will only use one
of the two conf dir.
2010/3/9, Pravin Karne <pr...@persistent.co.in>:
> Hi Millebii,
> Thanks for your valuable inputs.
>
> As per our requirements we need to run multiple nutch instances with each
> instance pointing to their own conf dir and crawlDB.
>
> crawl -urlfilter.txt is different in both conf folder. But in our case both
> nutch instances picking same conf dir instead of picking their own conf
> dir.
>
> So both crawlDB have same data. [Actually we have separate filter in both
> conf, so data in both crawlDB should be different]
>
> Have you tried such scenario? Since
>
>
> Thanks
> -Pravin
>
>
>
> -----Original Message-----
> From: MilleBii [mailto:millebii@gmail.com]
> Sent: Tuesday, March 09, 2010 1:06 PM
> To: nutch-user@lucene.apache.org
> Subject: Re: Two Nutch parallel crawl with two conf folder.
>
> Yes it should work, I personnaly run some tests crawl on the same
> hardware, even on the same nutch directory thus I share the conf
> directory.
> But If you don't want that I would use two nutch directory and of
> course two different crawl directory because with hadoop they will
> end-up on the same hdfs: (assuming you run in distribued or pseudo)
>
> 2010/3/9, Pravin Karne <pr...@persistent.co.in>:
>>
>> Can we share Hadoop cluster between two nutch instance.
>> So there will be two nutch instance and they will point to same Hadoop
>> cluster.
>>
>> This way I am able to share my hardware bandwidth. I know that Hadoop in
>> distributed mode serializes jobs.
>> But I will not affect my flow. I just want to share my hardware resource.
>>
>> I tried with two nutch setup , but somehow second instance overriding the
>> first one's configuration.
>>
>>
>> Any pointers ?????
>>
>> Thanks
>> -Pravin
>>
>>
>> -----Original Message-----
>> From: MilleBii [mailto:millebii@gmail.com]
>> Sent: Monday, March 08, 2010 8:02 PM
>> To: nutch-user@lucene.apache.org
>> Subject: Re: Two Nutch parallel crawl with two conf folder.
>>
>> How parallel is parallel in your case ?
>> Don't forget Hadoop in distributed mode will serialize your jobs anyhow.
>>
>> For the rest why don't you create two Nutch directories and run things
>> totally independently
>>
>>
>> 2010/3/8, Pravin Karne <pr...@persistent.co.in>:
>>> Hi guys any pointer on following.
>>> Your help will highly appreciated .
>>>
>>> Thanks
>>> -Pravin
>>>
>>> -----Original Message-----
>>> From: Pravin Karne
>>> Sent: Friday, March 05, 2010 12:57 PM
>>> To: nutch-user@lucene.apache.org
>>> Subject: Two Nutch parallel crawl with two conf folder.
>>>
>>> Hi,
>>>
>>> I want to do two Nutch parallel crawl with two conf folder.
>>>
>>> I am using crawl command to do this. I have two separate conf folders,
>>> all
>>> files from conf are same except crawl-urlfilter.txt . In this file we
>>> have
>>> different filters(domain filters).
>>>
>>> e.g . 1 st conf have -
>>> +.^http://([a-z0-9]*\.)*abc.com/
>>>
>>> 2nd conf have -
>>> +.^http://([a-z0-9]*\.)*xyz.com/
>>>
>>>
>>> I am starting two crawl with above configuration and on separate
>>> console.(one followed by other)
>>>
>>> I am using following crawl commands -
>>>
>>> bin/nutch --nutch_conf_dir=/home/conf1 crawl urls -dir test1
>>> -depth
>>> 1
>>>
>>> bin/nutch --nutch_conf_dir=/home/conf2 crawl urls -dir test2
>>> -depth
>>> 1
>>>
>>> [Note: We have modified nutch.sh for '--nutch_conf_dir']
>>>
>>> urls file have following entries-
>>>
>>> http://www.abc.com
>>> http://www.xyz.com
>>> http://www.pqr.com
>>>
>>>
>>> Expected Result:
>>>
>>> CrawlDB test1 should contains abc.com's data and CrawlDB test2
>>> should
>>> contains xyz.com's data.
>>>
>>> Actual Results:
>>>
>>> url filter of first run is overridden by url filter of second run.
>>>
>>> So Both CrawlDB have xyz.com's data.
>>>
>>>
>>> Please provide pointer regarding this.
>>>
>>> Thanks in advance.
>>>
>>> -Pravin
>>>
>>>
>>> DISCLAIMER
>>> ==========
>>> This e-mail may contain privileged and confidential information which is
>>> the
>>> property of Persistent Systems Ltd. It is intended only for the use of
>>> the
>>> individual or entity to which it is addressed. If you are not the
>>> intended
>>> recipient, you are not authorized to read, retain, copy, print,
>>> distribute
>>> or use this message. If you have received this communication in error,
>>> please notify the sender and delete all copies of this message.
>>> Persistent
>>> Systems Ltd. does not accept any liability for virus infected mails.
>>>
>>
>>
>> --
>> -MilleBii-
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is
>> the
>> property of Persistent Systems Ltd. It is intended only for the use of
>> the
>> individual or entity to which it is addressed. If you are not the
>> intended
>> recipient, you are not authorized to read, retain, copy, print,
>> distribute
>> or use this message. If you have received this communication in error,
>> please notify the sender and delete all copies of this message.
>> Persistent
>> Systems Ltd. does not accept any liability for virus infected mails.
>>
>
>
> --
> -MilleBii-
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the
> property of Persistent Systems Ltd. It is intended only for the use of the
> individual or entity to which it is addressed. If you are not the intended
> recipient, you are not authorized to read, retain, copy, print, distribute
> or use this message. If you have received this communication in error,
> please notify the sender and delete all copies of this message. Persistent
> Systems Ltd. does not accept any liability for virus infected mails.
>
--
-MilleBii-
RE: Two Nutch parallel crawl with two conf folder.
Posted by Pravin Karne <pr...@persistent.co.in>.
Hi Millebii,
Thanks for your valuable inputs.
As per our requirements we need to run multiple nutch instances with each instance pointing to their own conf dir and crawlDB.
crawl -urlfilter.txt is different in both conf folder. But in our case both nutch instances picking same conf dir instead of picking their own conf dir.
So both crawlDB have same data. [Actually we have separate filter in both conf, so data in both crawlDB should be different]
Have you tried such scenario? Since
Thanks
-Pravin
-----Original Message-----
From: MilleBii [mailto:millebii@gmail.com]
Sent: Tuesday, March 09, 2010 1:06 PM
To: nutch-user@lucene.apache.org
Subject: Re: Two Nutch parallel crawl with two conf folder.
Yes it should work, I personnaly run some tests crawl on the same
hardware, even on the same nutch directory thus I share the conf
directory.
But If you don't want that I would use two nutch directory and of
course two different crawl directory because with hadoop they will
end-up on the same hdfs: (assuming you run in distribued or pseudo)
2010/3/9, Pravin Karne <pr...@persistent.co.in>:
>
> Can we share Hadoop cluster between two nutch instance.
> So there will be two nutch instance and they will point to same Hadoop
> cluster.
>
> This way I am able to share my hardware bandwidth. I know that Hadoop in
> distributed mode serializes jobs.
> But I will not affect my flow. I just want to share my hardware resource.
>
> I tried with two nutch setup , but somehow second instance overriding the
> first one's configuration.
>
>
> Any pointers ?????
>
> Thanks
> -Pravin
>
>
> -----Original Message-----
> From: MilleBii [mailto:millebii@gmail.com]
> Sent: Monday, March 08, 2010 8:02 PM
> To: nutch-user@lucene.apache.org
> Subject: Re: Two Nutch parallel crawl with two conf folder.
>
> How parallel is parallel in your case ?
> Don't forget Hadoop in distributed mode will serialize your jobs anyhow.
>
> For the rest why don't you create two Nutch directories and run things
> totally independently
>
>
> 2010/3/8, Pravin Karne <pr...@persistent.co.in>:
>> Hi guys any pointer on following.
>> Your help will highly appreciated .
>>
>> Thanks
>> -Pravin
>>
>> -----Original Message-----
>> From: Pravin Karne
>> Sent: Friday, March 05, 2010 12:57 PM
>> To: nutch-user@lucene.apache.org
>> Subject: Two Nutch parallel crawl with two conf folder.
>>
>> Hi,
>>
>> I want to do two Nutch parallel crawl with two conf folder.
>>
>> I am using crawl command to do this. I have two separate conf folders,
>> all
>> files from conf are same except crawl-urlfilter.txt . In this file we
>> have
>> different filters(domain filters).
>>
>> e.g . 1 st conf have -
>> +.^http://([a-z0-9]*\.)*abc.com/
>>
>> 2nd conf have -
>> +.^http://([a-z0-9]*\.)*xyz.com/
>>
>>
>> I am starting two crawl with above configuration and on separate
>> console.(one followed by other)
>>
>> I am using following crawl commands -
>>
>> bin/nutch --nutch_conf_dir=/home/conf1 crawl urls -dir test1 -depth
>> 1
>>
>> bin/nutch --nutch_conf_dir=/home/conf2 crawl urls -dir test2 -depth
>> 1
>>
>> [Note: We have modified nutch.sh for '--nutch_conf_dir']
>>
>> urls file have following entries-
>>
>> http://www.abc.com
>> http://www.xyz.com
>> http://www.pqr.com
>>
>>
>> Expected Result:
>>
>> CrawlDB test1 should contains abc.com's data and CrawlDB test2
>> should
>> contains xyz.com's data.
>>
>> Actual Results:
>>
>> url filter of first run is overridden by url filter of second run.
>>
>> So Both CrawlDB have xyz.com's data.
>>
>>
>> Please provide pointer regarding this.
>>
>> Thanks in advance.
>>
>> -Pravin
>>
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is
>> the
>> property of Persistent Systems Ltd. It is intended only for the use of
>> the
>> individual or entity to which it is addressed. If you are not the
>> intended
>> recipient, you are not authorized to read, retain, copy, print,
>> distribute
>> or use this message. If you have received this communication in error,
>> please notify the sender and delete all copies of this message.
>> Persistent
>> Systems Ltd. does not accept any liability for virus infected mails.
>>
>
>
> --
> -MilleBii-
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the
> property of Persistent Systems Ltd. It is intended only for the use of the
> individual or entity to which it is addressed. If you are not the intended
> recipient, you are not authorized to read, retain, copy, print, distribute
> or use this message. If you have received this communication in error,
> please notify the sender and delete all copies of this message. Persistent
> Systems Ltd. does not accept any liability for virus infected mails.
>
--
-MilleBii-
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Two Nutch parallel crawl with two conf folder.
Posted by eks dev <ek...@yahoo.co.uk>.
sorry for the noise.. I've mixed up Emails
----- Original Message ----
> From: eks dev <ek...@yahoo.co.uk>
> To: nutch-user@lucene.apache.org
> Sent: Tue, 9 March, 2010 18:07:47
> Subject: Re: Two Nutch parallel crawl with two conf folder.
>
> coool answer
>
>
>
> ----- Original Message ----
> > From: MilleBii
> > To: nutch-user@lucene.apache.org
> > Sent: Tue, 9 March, 2010 8:35:42
> > Subject: Re: Two Nutch parallel crawl with two conf folder.
> >
> > Yes it should work, I personnaly run some tests crawl on the same
> > hardware, even on the same nutch directory thus I share the conf
> > directory.
> > But If you don't want that I would use two nutch directory and of
> > course two different crawl directory because with hadoop they will
> > end-up on the same hdfs: (assuming you run in distribued or pseudo)
> >
> > 2010/3/9, Pravin Karne :
> > >
> > > Can we share Hadoop cluster between two nutch instance.
> > > So there will be two nutch instance and they will point to same Hadoop
> > > cluster.
> > >
> > > This way I am able to share my hardware bandwidth. I know that Hadoop in
> > > distributed mode serializes jobs.
> > > But I will not affect my flow. I just want to share my hardware resource.
> > >
> > > I tried with two nutch setup , but somehow second instance overriding the
> > > first one's configuration.
> > >
> > >
> > > Any pointers ?????
> > >
> > > Thanks
> > > -Pravin
> > >
> > >
> > > -----Original Message-----
> > > From: MilleBii [mailto:millebii@gmail.com]
> > > Sent: Monday, March 08, 2010 8:02 PM
> > > To: nutch-user@lucene.apache.org
> > > Subject: Re: Two Nutch parallel crawl with two conf folder.
> > >
> > > How parallel is parallel in your case ?
> > > Don't forget Hadoop in distributed mode will serialize your jobs anyhow.
> > >
> > > For the rest why don't you create two Nutch directories and run things
> > > totally independently
> > >
> > >
> > > 2010/3/8, Pravin Karne :
> > >> Hi guys any pointer on following.
> > >> Your help will highly appreciated .
> > >>
> > >> Thanks
> > >> -Pravin
> > >>
> > >> -----Original Message-----
> > >> From: Pravin Karne
> > >> Sent: Friday, March 05, 2010 12:57 PM
> > >> To: nutch-user@lucene.apache.org
> > >> Subject: Two Nutch parallel crawl with two conf folder.
> > >>
> > >> Hi,
> > >>
> > >> I want to do two Nutch parallel crawl with two conf folder.
> > >>
> > >> I am using crawl command to do this. I have two separate conf folders,
> > >> all
> > >> files from conf are same except crawl-urlfilter.txt . In this file we
> > >> have
> > >> different filters(domain filters).
> > >>
> > >> e.g . 1 st conf have -
> > >> +.^http://([a-z0-9]*\.)*abc.com/
> > >>
> > >> 2nd conf have -
> > >> +.^http://([a-z0-9]*\.)*xyz.com/
> > >>
> > >>
> > >> I am starting two crawl with above configuration and on separate
> > >> console.(one followed by other)
> > >>
> > >> I am using following crawl commands -
> > >>
> > >> bin/nutch --nutch_conf_dir=/home/conf1 crawl urls -dir test1 -depth
> > >> 1
> > >>
> > >> bin/nutch --nutch_conf_dir=/home/conf2 crawl urls -dir test2 -depth
> > >> 1
> > >>
> > >> [Note: We have modified nutch.sh for '--nutch_conf_dir']
> > >>
> > >> urls file have following entries-
> > >>
> > >> http://www.abc.com
> > >> http://www.xyz.com
> > >> http://www.pqr.com
> > >>
> > >>
> > >> Expected Result:
> > >>
> > >> CrawlDB test1 should contains abc.com's data and CrawlDB test2
> > >> should
> > >> contains xyz.com's data.
> > >>
> > >> Actual Results:
> > >>
> > >> url filter of first run is overridden by url filter of second run.
> > >>
> > >> So Both CrawlDB have xyz.com's data.
> > >>
> > >>
> > >> Please provide pointer regarding this.
> > >>
> > >> Thanks in advance.
> > >>
> > >> -Pravin
> > >>
> > >>
> > >> DISCLAIMER
> > >> ==========
> > >> This e-mail may contain privileged and confidential information which is
> > >> the
> > >> property of Persistent Systems Ltd. It is intended only for the use of
> > >> the
> > >> individual or entity to which it is addressed. If you are not the
> > >> intended
> > >> recipient, you are not authorized to read, retain, copy, print,
> > >> distribute
> > >> or use this message. If you have received this communication in error,
> > >> please notify the sender and delete all copies of this message.
> > >> Persistent
> > >> Systems Ltd. does not accept any liability for virus infected mails.
> > >>
> > >
> > >
> > > --
> > > -MilleBii-
> > >
> > > DISCLAIMER
> > > ==========
> > > This e-mail may contain privileged and confidential information which is the
> > > property of Persistent Systems Ltd. It is intended only for the use of the
> > > individual or entity to which it is addressed. If you are not the intended
> > > recipient, you are not authorized to read, retain, copy, print, distribute
> > > or use this message. If you have received this communication in error,
> > > please notify the sender and delete all copies of this message. Persistent
> > > Systems Ltd. does not accept any liability for virus infected mails.
> > >
> >
> >
> > --
> > -MilleBii-
Re: Two Nutch parallel crawl with two conf folder.
Posted by eks dev <ek...@yahoo.co.uk>.
coool answer
----- Original Message ----
> From: MilleBii <mi...@gmail.com>
> To: nutch-user@lucene.apache.org
> Sent: Tue, 9 March, 2010 8:35:42
> Subject: Re: Two Nutch parallel crawl with two conf folder.
>
> Yes it should work, I personnaly run some tests crawl on the same
> hardware, even on the same nutch directory thus I share the conf
> directory.
> But If you don't want that I would use two nutch directory and of
> course two different crawl directory because with hadoop they will
> end-up on the same hdfs: (assuming you run in distribued or pseudo)
>
> 2010/3/9, Pravin Karne :
> >
> > Can we share Hadoop cluster between two nutch instance.
> > So there will be two nutch instance and they will point to same Hadoop
> > cluster.
> >
> > This way I am able to share my hardware bandwidth. I know that Hadoop in
> > distributed mode serializes jobs.
> > But I will not affect my flow. I just want to share my hardware resource.
> >
> > I tried with two nutch setup , but somehow second instance overriding the
> > first one's configuration.
> >
> >
> > Any pointers ?????
> >
> > Thanks
> > -Pravin
> >
> >
> > -----Original Message-----
> > From: MilleBii [mailto:millebii@gmail.com]
> > Sent: Monday, March 08, 2010 8:02 PM
> > To: nutch-user@lucene.apache.org
> > Subject: Re: Two Nutch parallel crawl with two conf folder.
> >
> > How parallel is parallel in your case ?
> > Don't forget Hadoop in distributed mode will serialize your jobs anyhow.
> >
> > For the rest why don't you create two Nutch directories and run things
> > totally independently
> >
> >
> > 2010/3/8, Pravin Karne :
> >> Hi guys any pointer on following.
> >> Your help will highly appreciated .
> >>
> >> Thanks
> >> -Pravin
> >>
> >> -----Original Message-----
> >> From: Pravin Karne
> >> Sent: Friday, March 05, 2010 12:57 PM
> >> To: nutch-user@lucene.apache.org
> >> Subject: Two Nutch parallel crawl with two conf folder.
> >>
> >> Hi,
> >>
> >> I want to do two Nutch parallel crawl with two conf folder.
> >>
> >> I am using crawl command to do this. I have two separate conf folders,
> >> all
> >> files from conf are same except crawl-urlfilter.txt . In this file we
> >> have
> >> different filters(domain filters).
> >>
> >> e.g . 1 st conf have -
> >> +.^http://([a-z0-9]*\.)*abc.com/
> >>
> >> 2nd conf have -
> >> +.^http://([a-z0-9]*\.)*xyz.com/
> >>
> >>
> >> I am starting two crawl with above configuration and on separate
> >> console.(one followed by other)
> >>
> >> I am using following crawl commands -
> >>
> >> bin/nutch --nutch_conf_dir=/home/conf1 crawl urls -dir test1 -depth
> >> 1
> >>
> >> bin/nutch --nutch_conf_dir=/home/conf2 crawl urls -dir test2 -depth
> >> 1
> >>
> >> [Note: We have modified nutch.sh for '--nutch_conf_dir']
> >>
> >> urls file have following entries-
> >>
> >> http://www.abc.com
> >> http://www.xyz.com
> >> http://www.pqr.com
> >>
> >>
> >> Expected Result:
> >>
> >> CrawlDB test1 should contains abc.com's data and CrawlDB test2
> >> should
> >> contains xyz.com's data.
> >>
> >> Actual Results:
> >>
> >> url filter of first run is overridden by url filter of second run.
> >>
> >> So Both CrawlDB have xyz.com's data.
> >>
> >>
> >> Please provide pointer regarding this.
> >>
> >> Thanks in advance.
> >>
> >> -Pravin
> >>
> >>
> >> DISCLAIMER
> >> ==========
> >> This e-mail may contain privileged and confidential information which is
> >> the
> >> property of Persistent Systems Ltd. It is intended only for the use of
> >> the
> >> individual or entity to which it is addressed. If you are not the
> >> intended
> >> recipient, you are not authorized to read, retain, copy, print,
> >> distribute
> >> or use this message. If you have received this communication in error,
> >> please notify the sender and delete all copies of this message.
> >> Persistent
> >> Systems Ltd. does not accept any liability for virus infected mails.
> >>
> >
> >
> > --
> > -MilleBii-
> >
> > DISCLAIMER
> > ==========
> > This e-mail may contain privileged and confidential information which is the
> > property of Persistent Systems Ltd. It is intended only for the use of the
> > individual or entity to which it is addressed. If you are not the intended
> > recipient, you are not authorized to read, retain, copy, print, distribute
> > or use this message. If you have received this communication in error,
> > please notify the sender and delete all copies of this message. Persistent
> > Systems Ltd. does not accept any liability for virus infected mails.
> >
>
>
> --
> -MilleBii-
Re: Two Nutch parallel crawl with two conf folder.
Posted by MilleBii <mi...@gmail.com>.
Yes it should work, I personnaly run some tests crawl on the same
hardware, even on the same nutch directory thus I share the conf
directory.
But If you don't want that I would use two nutch directory and of
course two different crawl directory because with hadoop they will
end-up on the same hdfs: (assuming you run in distribued or pseudo)
2010/3/9, Pravin Karne <pr...@persistent.co.in>:
>
> Can we share Hadoop cluster between two nutch instance.
> So there will be two nutch instance and they will point to same Hadoop
> cluster.
>
> This way I am able to share my hardware bandwidth. I know that Hadoop in
> distributed mode serializes jobs.
> But I will not affect my flow. I just want to share my hardware resource.
>
> I tried with two nutch setup , but somehow second instance overriding the
> first one's configuration.
>
>
> Any pointers ?????
>
> Thanks
> -Pravin
>
>
> -----Original Message-----
> From: MilleBii [mailto:millebii@gmail.com]
> Sent: Monday, March 08, 2010 8:02 PM
> To: nutch-user@lucene.apache.org
> Subject: Re: Two Nutch parallel crawl with two conf folder.
>
> How parallel is parallel in your case ?
> Don't forget Hadoop in distributed mode will serialize your jobs anyhow.
>
> For the rest why don't you create two Nutch directories and run things
> totally independently
>
>
> 2010/3/8, Pravin Karne <pr...@persistent.co.in>:
>> Hi guys any pointer on following.
>> Your help will highly appreciated .
>>
>> Thanks
>> -Pravin
>>
>> -----Original Message-----
>> From: Pravin Karne
>> Sent: Friday, March 05, 2010 12:57 PM
>> To: nutch-user@lucene.apache.org
>> Subject: Two Nutch parallel crawl with two conf folder.
>>
>> Hi,
>>
>> I want to do two Nutch parallel crawl with two conf folder.
>>
>> I am using crawl command to do this. I have two separate conf folders,
>> all
>> files from conf are same except crawl-urlfilter.txt . In this file we
>> have
>> different filters(domain filters).
>>
>> e.g . 1 st conf have -
>> +.^http://([a-z0-9]*\.)*abc.com/
>>
>> 2nd conf have -
>> +.^http://([a-z0-9]*\.)*xyz.com/
>>
>>
>> I am starting two crawl with above configuration and on separate
>> console.(one followed by other)
>>
>> I am using following crawl commands -
>>
>> bin/nutch --nutch_conf_dir=/home/conf1 crawl urls -dir test1 -depth
>> 1
>>
>> bin/nutch --nutch_conf_dir=/home/conf2 crawl urls -dir test2 -depth
>> 1
>>
>> [Note: We have modified nutch.sh for '--nutch_conf_dir']
>>
>> urls file have following entries-
>>
>> http://www.abc.com
>> http://www.xyz.com
>> http://www.pqr.com
>>
>>
>> Expected Result:
>>
>> CrawlDB test1 should contains abc.com's data and CrawlDB test2
>> should
>> contains xyz.com's data.
>>
>> Actual Results:
>>
>> url filter of first run is overridden by url filter of second run.
>>
>> So Both CrawlDB have xyz.com's data.
>>
>>
>> Please provide pointer regarding this.
>>
>> Thanks in advance.
>>
>> -Pravin
>>
>>
>> DISCLAIMER
>> ==========
>> This e-mail may contain privileged and confidential information which is
>> the
>> property of Persistent Systems Ltd. It is intended only for the use of
>> the
>> individual or entity to which it is addressed. If you are not the
>> intended
>> recipient, you are not authorized to read, retain, copy, print,
>> distribute
>> or use this message. If you have received this communication in error,
>> please notify the sender and delete all copies of this message.
>> Persistent
>> Systems Ltd. does not accept any liability for virus infected mails.
>>
>
>
> --
> -MilleBii-
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the
> property of Persistent Systems Ltd. It is intended only for the use of the
> individual or entity to which it is addressed. If you are not the intended
> recipient, you are not authorized to read, retain, copy, print, distribute
> or use this message. If you have received this communication in error,
> please notify the sender and delete all copies of this message. Persistent
> Systems Ltd. does not accept any liability for virus infected mails.
>
--
-MilleBii-
RE: Two Nutch parallel crawl with two conf folder.
Posted by Pravin Karne <pr...@persistent.co.in>.
Can we share Hadoop cluster between two nutch instance.
So there will be two nutch instance and they will point to same Hadoop cluster.
This way I am able to share my hardware bandwidth. I know that Hadoop in distributed mode serializes jobs.
But I will not affect my flow. I just want to share my hardware resource.
I tried with two nutch setup , but somehow second instance overriding the first one's configuration.
Any pointers ?????
Thanks
-Pravin
-----Original Message-----
From: MilleBii [mailto:millebii@gmail.com]
Sent: Monday, March 08, 2010 8:02 PM
To: nutch-user@lucene.apache.org
Subject: Re: Two Nutch parallel crawl with two conf folder.
How parallel is parallel in your case ?
Don't forget Hadoop in distributed mode will serialize your jobs anyhow.
For the rest why don't you create two Nutch directories and run things
totally independently
2010/3/8, Pravin Karne <pr...@persistent.co.in>:
> Hi guys any pointer on following.
> Your help will highly appreciated .
>
> Thanks
> -Pravin
>
> -----Original Message-----
> From: Pravin Karne
> Sent: Friday, March 05, 2010 12:57 PM
> To: nutch-user@lucene.apache.org
> Subject: Two Nutch parallel crawl with two conf folder.
>
> Hi,
>
> I want to do two Nutch parallel crawl with two conf folder.
>
> I am using crawl command to do this. I have two separate conf folders, all
> files from conf are same except crawl-urlfilter.txt . In this file we have
> different filters(domain filters).
>
> e.g . 1 st conf have -
> +.^http://([a-z0-9]*\.)*abc.com/
>
> 2nd conf have -
> +.^http://([a-z0-9]*\.)*xyz.com/
>
>
> I am starting two crawl with above configuration and on separate
> console.(one followed by other)
>
> I am using following crawl commands -
>
> bin/nutch --nutch_conf_dir=/home/conf1 crawl urls -dir test1 -depth 1
>
> bin/nutch --nutch_conf_dir=/home/conf2 crawl urls -dir test2 -depth 1
>
> [Note: We have modified nutch.sh for '--nutch_conf_dir']
>
> urls file have following entries-
>
> http://www.abc.com
> http://www.xyz.com
> http://www.pqr.com
>
>
> Expected Result:
>
> CrawlDB test1 should contains abc.com's data and CrawlDB test2 should
> contains xyz.com's data.
>
> Actual Results:
>
> url filter of first run is overridden by url filter of second run.
>
> So Both CrawlDB have xyz.com's data.
>
>
> Please provide pointer regarding this.
>
> Thanks in advance.
>
> -Pravin
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the
> property of Persistent Systems Ltd. It is intended only for the use of the
> individual or entity to which it is addressed. If you are not the intended
> recipient, you are not authorized to read, retain, copy, print, distribute
> or use this message. If you have received this communication in error,
> please notify the sender and delete all copies of this message. Persistent
> Systems Ltd. does not accept any liability for virus infected mails.
>
--
-MilleBii-
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Re: Two Nutch parallel crawl with two conf folder.
Posted by MilleBii <mi...@gmail.com>.
How parallel is parallel in your case ?
Don't forget Hadoop in distributed mode will serialize your jobs anyhow.
For the rest why don't you create two Nutch directories and run things
totally independently
2010/3/8, Pravin Karne <pr...@persistent.co.in>:
> Hi guys any pointer on following.
> Your help will highly appreciated .
>
> Thanks
> -Pravin
>
> -----Original Message-----
> From: Pravin Karne
> Sent: Friday, March 05, 2010 12:57 PM
> To: nutch-user@lucene.apache.org
> Subject: Two Nutch parallel crawl with two conf folder.
>
> Hi,
>
> I want to do two Nutch parallel crawl with two conf folder.
>
> I am using crawl command to do this. I have two separate conf folders, all
> files from conf are same except crawl-urlfilter.txt . In this file we have
> different filters(domain filters).
>
> e.g . 1 st conf have -
> +.^http://([a-z0-9]*\.)*abc.com/
>
> 2nd conf have -
> +.^http://([a-z0-9]*\.)*xyz.com/
>
>
> I am starting two crawl with above configuration and on separate
> console.(one followed by other)
>
> I am using following crawl commands -
>
> bin/nutch --nutch_conf_dir=/home/conf1 crawl urls -dir test1 -depth 1
>
> bin/nutch --nutch_conf_dir=/home/conf2 crawl urls -dir test2 -depth 1
>
> [Note: We have modified nutch.sh for '--nutch_conf_dir']
>
> urls file have following entries-
>
> http://www.abc.com
> http://www.xyz.com
> http://www.pqr.com
>
>
> Expected Result:
>
> CrawlDB test1 should contains abc.com's data and CrawlDB test2 should
> contains xyz.com's data.
>
> Actual Results:
>
> url filter of first run is overridden by url filter of second run.
>
> So Both CrawlDB have xyz.com's data.
>
>
> Please provide pointer regarding this.
>
> Thanks in advance.
>
> -Pravin
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is the
> property of Persistent Systems Ltd. It is intended only for the use of the
> individual or entity to which it is addressed. If you are not the intended
> recipient, you are not authorized to read, retain, copy, print, distribute
> or use this message. If you have received this communication in error,
> please notify the sender and delete all copies of this message. Persistent
> Systems Ltd. does not accept any liability for virus infected mails.
>
--
-MilleBii-
RE: Two Nutch parallel crawl with two conf folder.
Posted by Pravin Karne <pr...@persistent.co.in>.
Hi guys any pointer on following.
Your help will highly appreciated .
Thanks
-Pravin
-----Original Message-----
From: Pravin Karne
Sent: Friday, March 05, 2010 12:57 PM
To: nutch-user@lucene.apache.org
Subject: Two Nutch parallel crawl with two conf folder.
Hi,
I want to do two Nutch parallel crawl with two conf folder.
I am using crawl command to do this. I have two separate conf folders, all files from conf are same except crawl-urlfilter.txt . In this file we have different filters(domain filters).
e.g . 1 st conf have -
+.^http://([a-z0-9]*\.)*abc.com/
2nd conf have -
+.^http://([a-z0-9]*\.)*xyz.com/
I am starting two crawl with above configuration and on separate console.(one followed by other)
I am using following crawl commands -
bin/nutch --nutch_conf_dir=/home/conf1 crawl urls -dir test1 -depth 1
bin/nutch --nutch_conf_dir=/home/conf2 crawl urls -dir test2 -depth 1
[Note: We have modified nutch.sh for '--nutch_conf_dir']
urls file have following entries-
http://www.abc.com
http://www.xyz.com
http://www.pqr.com
Expected Result:
CrawlDB test1 should contains abc.com's data and CrawlDB test2 should contains xyz.com's data.
Actual Results:
url filter of first run is overridden by url filter of second run.
So Both CrawlDB have xyz.com's data.
Please provide pointer regarding this.
Thanks in advance.
-Pravin
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
Two Nutch parallel crawl with two conf folder.
Posted by Pravin Karne <pr...@persistent.co.in>.
Hi,
I want to do two Nutch parallel crawl with two conf folder.
I am using crawl command to do this. I have two separate conf folders, all files from conf are same except crawl-urlfilter.txt . In this file we have different filters(domain filters).
e.g . 1 st conf have -
+.^http://([a-z0-9]*\.)*abc.com/
2nd conf have -
+.^http://([a-z0-9]*\.)*xyz.com/
I am starting two crawl with above configuration and on separate console.(one followed by other)
I am using following crawl commands -
bin/nutch --nutch_conf_dir=/home/conf1 crawl urls -dir test1 -depth 1
bin/nutch --nutch_conf_dir=/home/conf2 crawl urls -dir test2 -depth 1
[Note: We have modified nutch.sh for '--nutch_conf_dir']
urls file have following entries-
http://www.abc.com
http://www.xyz.com
http://www.pqr.com
Expected Result:
CrawlDB test1 should contains abc.com's data and CrawlDB test2 should contains xyz.com's data.
Actual Results:
url filter of first run is overridden by url filter of second run.
So Both CrawlDB have xyz.com's data.
Please provide pointer regarding this.
Thanks in advance.
-Pravin
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.
AW: By Indexing I get: OutOfMemoryError: GC overhead limit exceeded ...
Posted by Patricio Galeas <pg...@yahoo.de>.
Hello Ted,
I ran the command 'ps -aux' and I confirmed that only 1GB was defined.
I adjust NUTCH_HEAPSIZE to 8GB (physical RAM) and ran it again
successfully.
Do you know which parameters need to be adjusted if not enough physical RAM is available on the server? For example for 2GB RAM.
I ran a web crawl (depth=6) without the parameter topN and the segments grew exponetially.
Later, I had a lot of problems by merging the segments and by indexing
(not enough memory, too many opened files, etc.).
Thank you for your help
Pato
----- Ursprüngliche Mail ----
Von: Ted Yu <yu...@gmail.com>
An: nutch-user@lucene.apache.org
Gesendet: Samstag, den 6. März 2010, 15:42:38 Uhr
Betreff: Re: By Indexing I get: OutOfMemoryError: GC overhead limit exceeded ...
Can you use 'ps aux' to find out the -Xmx commandine parameter passed to
java for the following action ?
On Fri, Mar 5, 2010 at 1:14 PM, Patricio Galeas <pg...@yahoo.de> wrote:
> Hello all,
> I am running Nutch in a Virtual Machine (Debian) with 8 GB RAM and 1,5TB
> for the hadoop temporal folder.
> Running the index process with a 1.3GB segments folder, I got
> "OutOfMemoryError: GC overhead limit exceeded" (see below)
>
> I created the segments using slice=50000
> and I also set HADOOP_HEAPSIZE with the maximal physical memory (8000).
>
> Do I need more memory to run the index process?
> Are there some limitation to run Nutch in a Virtual Machine?
>
> Thank you!
> Pato
>
> ...
> ...
> 2010-03-05 19:52:13,864 INFO plugin.PluginRepository - Nutch
> Scoring (org.apache.nutch.scoring.ScoringFilter)
> 2010-03-05 19:52:13,864 INFO plugin.PluginRepository - Ontology
> Model Loader (org.apache.nutch.ontology.Ontology)
> 2010-03-05 19:52:13,867 INFO lang.LanguageIdentifier - Language identifier
> configuration [1-4/2048]
> 2010-03-05 19:52:22,961 INFO lang.LanguageIdentifier - Language identifier
> plugin supports: it(1000) is(1000) hu(1000) th(1000) sv(1000) sq(1000)
> fr(1000) ru(1000) fi(1000) es(1000) en(1000) el(1000) ee(1000) pt(1000)
> de(1000) da(1000) pl(1000) no(1000) nl(1000)
> 2010-03-05 19:52:22,961 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.analysis.lang.LanguageIndexingFilter
> 2010-03-05 19:52:22,963 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2010-03-05 19:52:22,964 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2010-03-05 19:52:36,278 WARN mapred.LocalJobRunner - job_local_0001
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:775)
> at org.apache.hadoop.io.Text.encode(Text.java:388)
> at org.apache.hadoop.io.Text.encode(Text.java:369)
> at org.apache.hadoop.io.Text.writeString(Text.java:409)
> at org.apache.nutch.parse.Outlink.write(Outlink.java:52)
> at org.apache.nutch.parse.ParseData.write(ParseData.java:152)
> at
> org.apache.hadoop.io.GenericWritable.write(GenericWritable.java:135)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:613)
> at
> org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:67)
> at
> org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:50)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
> 2010-03-05 19:52:37,277 FATAL indexer.Indexer - Indexer:
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
> at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
> at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
>
> __________________________________________________
> Do You Yahoo!?
> Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz
> gegen Massenmails.
> http://mail.yahoo.com
>
__________________________________________________
Do You Yahoo!?
Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails.
http://mail.yahoo.com
Re: By Indexing I get: OutOfMemoryError: GC overhead limit exceeded
...
Posted by Ted Yu <yu...@gmail.com>.
Can you use 'ps aux' to find out the -Xmx commandine parameter passed to
java for the following action ?
On Fri, Mar 5, 2010 at 1:14 PM, Patricio Galeas <pg...@yahoo.de> wrote:
> Hello all,
> I am running Nutch in a Virtual Machine (Debian) with 8 GB RAM and 1,5TB
> for the hadoop temporal folder.
> Running the index process with a 1.3GB segments folder, I got
> "OutOfMemoryError: GC overhead limit exceeded" (see below)
>
> I created the segments using slice=50000
> and I also set HADOOP_HEAPSIZE with the maximal physical memory (8000).
>
> Do I need more memory to run the index process?
> Are there some limitation to run Nutch in a Virtual Machine?
>
> Thank you!
> Pato
>
> ...
> ...
> 2010-03-05 19:52:13,864 INFO plugin.PluginRepository - Nutch
> Scoring (org.apache.nutch.scoring.ScoringFilter)
> 2010-03-05 19:52:13,864 INFO plugin.PluginRepository - Ontology
> Model Loader (org.apache.nutch.ontology.Ontology)
> 2010-03-05 19:52:13,867 INFO lang.LanguageIdentifier - Language identifier
> configuration [1-4/2048]
> 2010-03-05 19:52:22,961 INFO lang.LanguageIdentifier - Language identifier
> plugin supports: it(1000) is(1000) hu(1000) th(1000) sv(1000) sq(1000)
> fr(1000) ru(1000) fi(1000) es(1000) en(1000) el(1000) ee(1000) pt(1000)
> de(1000) da(1000) pl(1000) no(1000) nl(1000)
> 2010-03-05 19:52:22,961 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.analysis.lang.LanguageIndexingFilter
> 2010-03-05 19:52:22,963 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.basic.BasicIndexingFilter
> 2010-03-05 19:52:22,964 INFO indexer.IndexingFilters - Adding
> org.apache.nutch.indexer.anchor.AnchorIndexingFilter
> 2010-03-05 19:52:36,278 WARN mapred.LocalJobRunner - job_local_0001
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
> at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:775)
> at org.apache.hadoop.io.Text.encode(Text.java:388)
> at org.apache.hadoop.io.Text.encode(Text.java:369)
> at org.apache.hadoop.io.Text.writeString(Text.java:409)
> at org.apache.nutch.parse.Outlink.write(Outlink.java:52)
> at org.apache.nutch.parse.ParseData.write(ParseData.java:152)
> at
> org.apache.hadoop.io.GenericWritable.write(GenericWritable.java:135)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:613)
> at
> org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:67)
> at
> org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:50)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
> 2010-03-05 19:52:37,277 FATAL indexer.Indexer - Indexer:
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
> at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
> at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
>
> __________________________________________________
> Do You Yahoo!?
> Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz
> gegen Massenmails.
> http://mail.yahoo.com
>
By Indexing I get: OutOfMemoryError: GC overhead limit exceeded ...
Posted by Patricio Galeas <pg...@yahoo.de>.
Hello all,
I am running Nutch in a Virtual Machine (Debian) with 8 GB RAM and 1,5TB for the hadoop temporal folder.
Running the index process with a 1.3GB segments folder, I got "OutOfMemoryError: GC overhead limit exceeded" (see below)
I created the segments using slice=50000
and I also set HADOOP_HEAPSIZE with the maximal physical memory (8000).
Do I need more memory to run the index process?
Are there some limitation to run Nutch in a Virtual Machine?
Thank you!
Pato
...
...
2010-03-05 19:52:13,864 INFO plugin.PluginRepository - Nutch Scoring (org.apache.nutch.scoring.ScoringFilter)
2010-03-05 19:52:13,864 INFO plugin.PluginRepository - Ontology Model Loader (org.apache.nutch.ontology.Ontology)
2010-03-05 19:52:13,867 INFO lang.LanguageIdentifier - Language identifier configuration [1-4/2048]
2010-03-05 19:52:22,961 INFO lang.LanguageIdentifier - Language identifier plugin supports: it(1000) is(1000) hu(1000) th(1000) sv(1000) sq(1000) fr(1000) ru(1000) fi(1000) es(1000) en(1000) el(1000) ee(1000) pt(1000) de(1000) da(1000) pl(1000) no(1000) nl(1000)
2010-03-05 19:52:22,961 INFO indexer.IndexingFilters - Adding org.apache.nutch.analysis.lang.LanguageIndexingFilter
2010-03-05 19:52:22,963 INFO indexer.IndexingFilters - Adding org.apache.nutch.indexer.basic.BasicIndexingFilter
2010-03-05 19:52:22,964 INFO indexer.IndexingFilters - Adding org.apache.nutch.indexer.anchor.AnchorIndexingFilter
2010-03-05 19:52:36,278 WARN mapred.LocalJobRunner - job_local_0001
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:775)
at org.apache.hadoop.io.Text.encode(Text.java:388)
at org.apache.hadoop.io.Text.encode(Text.java:369)
at org.apache.hadoop.io.Text.writeString(Text.java:409)
at org.apache.nutch.parse.Outlink.write(Outlink.java:52)
at org.apache.nutch.parse.ParseData.write(ParseData.java:152)
at org.apache.hadoop.io.GenericWritable.write(GenericWritable.java:135)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:613)
at org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:67)
at org.apache.nutch.indexer.IndexerMapReduce.map(IndexerMapReduce.java:50)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
2010-03-05 19:52:37,277 FATAL indexer.Indexer - Indexer: java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
at org.apache.nutch.indexer.Indexer.index(Indexer.java:72)
at org.apache.nutch.indexer.Indexer.run(Indexer.java:92)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.indexer.Indexer.main(Indexer.java:101)
__________________________________________________
Do You Yahoo!?
Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails.
http://mail.yahoo.com
Error by merging segments ...
Posted by Patricio Galeas <pg...@yahoo.de>.
Hello
By merging segments with ...
nutch mergesegs $crawldir/MERGEDsegments $crawldir/segments/* -slice 50000
... I got the following message in hadoop.log:
-------------------------
.
.
2010-03-03 03:27:01,849 INFO segment.SegmentMerger - Slice size: 50000 URLs.
2010-03-03 22:47:43,130 INFO segment.SegmentMerger - Slice size: 50000 URLs.
2010-03-03 22:47:43,149 INFO segment.SegmentMerger - Slice size: 50000 URLs.
2010-03-03 23:47:00,029 WARN mapred.LocalJobRunner - job_local_0001
java.lang.ThreadDeath
at java.lang.Thread.stop(Thread.java:715)
at org.apache.hadoop.mapred.LocalJobRunner.killJob(LocalJobRunner.java:310)
at org.apache.hadoop.mapred.JobClient$NetworkedJob.killJob(JobClient.java:315)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1239)
at org.apache.nutch.segment.SegmentMerger.merge(SegmentMerger.java:620)
at org.apache.nutch.segment.SegmentMerger.main(SegmentMerger.java:665)
-------------------------
In the folder $crawldir/MERGEDsegments, I have also a sub directory with the name "_temporary"
What I'm doing wrong?
Can I use the generated segments for indexing and continue later with the crawling process?
Thanks for your help.
Pat
__________________________________________________
Do You Yahoo!?
Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails.
http://mail.yahoo.com
Re: New version of nutch?
Posted by John Martyniak <jo...@beforedawnsolutions.com>.
Andrezj,
Thanks for the information. I anxiously await the update. The HBase
integration would be a nice to have but I don't think that it should hold up
the release.
-John
On Wed, Mar 3, 2010 at 5:04 PM, Andrzej Bialecki <ab...@getopt.org> wrote:
> On 2010-03-03 20:12, John Martyniak wrote:
>
>> Does anybody have an idea of when a new version of nutch will be
>> availale, specifically supporting a latest version of hadoop. And
>> possibly hbase?
>>
>> Thank you for any information.
>>
>
> We should roll out a 1.1 soon (a few weeks), the nutch+hbase is imho still
> a few months away.
>
>
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
--
John Martyniak
President/CEO
Before Dawn Solutions, Inc.
9457 S. University Blvd #266
Highlands Ranch, CO 80126
o: 877-499-1562
c: 303-522-1756
e: john@beforedawnsoutions.com
w: http://www.beforedawnsolutions.com
Re: New version of nutch?
Posted by Andrzej Bialecki <ab...@getopt.org>.
On 2010-03-03 20:12, John Martyniak wrote:
> Does anybody have an idea of when a new version of nutch will be
> availale, specifically supporting a latest version of hadoop. And
> possibly hbase?
>
> Thank you for any information.
We should roll out a 1.1 soon (a few weeks), the nutch+hbase is imho
still a few months away.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com