You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Felix Zimmermann <fe...@gmx.de> on 2009/12/16 22:26:28 UTC
Multiple Nutch instances for crawling?
Hi,
I would like to run at least two instances of nutch ONLY for crawling at
one time; one for very frequently updated sites and one for other sites.
Will the nutch instances get in trouble when running several
crawlscripts, especially the nutch confdir variable?
Thanks!
Felix.
RE: Multiple Nutch instances for crawling?
Posted by Jun Mao <Ju...@symantec.com>.
In my case, I am running many nucth instances. I called them spider pool.
From client side, someone will submit urls frequenly. Each time my server received a url,
It will use this url as seed, send out a nutch crawler to that site(limited only to that site),
Craw a few hundreds of pages and analyze them.
I guess I could not do this by the command line, so I write some code myself.
Thanks,
Jun
-----Original Message-----
From: Felix Zimmermann [mailto:felizimm@gmx.de]
Sent: 2009年12月17日 5:26
To: Nutch Mailinglist
Subject: Multiple Nutch instances for crawling?
Hi,
I would like to run at least two instances of nutch ONLY for crawling at
one time; one for very frequently updated sites and one for other sites.
Will the nutch instances get in trouble when running several
crawlscripts, especially the nutch confdir variable?
Thanks!
Felix.
Re: Multiple Nutch instances for crawling?
Posted by Felix Zimmermann <ma...@felix-zimmermann.eu>.
With the new nutch-gui (http://github.com/101tec/nutch) it is possible
to launch several instances on one machine at one time. How can I reach
this with shellscripts?
Felix.
Am Donnerstag, den 17.12.2009, 09:56 +0100 schrieb MilleBii:
> I guess because of the different nutch-site.xml & url filter that you want
> to use it won't work... but you could try installing nutch twice run the
> crawl/fetch/parse from those two locations. And joined the segments to
> recreate a unified searchable index (make sure you put all your segments
> under the same location).
>
> Just one comment though I think hadoop will serialize your jobs any how so
> you won't get a parallel execution of your hadoop jobs unless you run them
> from different hardware.
>
> 2009/12/16 Christopher Bader <cb...@gmail.com>
>
> > Felix,
> >
> > I've had trouble running multiple instances. I would be interested in
> > hearing from anyone who has done it successfully.
> >
> > CB
> >
> >
> > On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de> wrote:
> >
> > > Hi,
> > >
> > > I would like to run at least two instances of nutch ONLY for crawling at
> > > one time; one for very frequently updated sites and one for other sites.
> > > Will the nutch instances get in trouble when running several
> > > crawlscripts, especially the nutch confdir variable?
> > >
> > > Thanks!
> > > Felix.
> > >
> > >
> > >
> > >
> >
>
>
>
Re: Multiple Nutch instances for crawling?
Posted by Yves Petinot <yv...@snooth.com>.
Thanks a lot for the detailed explanation, Jason, this was most
definitely useful !
J.G.Konrad wrote:
> I have upgraded to 0.19.2 but the capacity scheduler feature is
> available in 0.19.1. You will need to download the hadoop common
> package ( http://hadoop.apache.org/common/releases.html ) to get the
> capacity scheduler jar. It is not included in the Nutch releases.
>
> After downloading you will want to copy the
> contrib/capacity-scheduler/hadoop-0.19.x-capacity-scheduler.jar into
> lib directory where you have your nutch code. There is also a example
> configuration file that comes with the hadoop package that is quite
> explanatory ( conf/capacity-scheduler ). The docs can be found here
> http://hadoop.apache.org/common/docs/r0.19.1/capacity_scheduler.html
>
> The first thing to do is to set the scheduler and define the queues.
> This is done in the hadoop-site.xml file that is used to start the
> jobtracker. This is what mine looks like for defining two queues (plus
> the default).
>
> <property>
> <name>mapred.jobtracker.taskScheduler</name>
> <value>org.apache.hadoop.mapred.CapacityTaskScheduler</value>
> </property>
>
> <property>
> <name>mapred.queue.names</name>
> <value>nutchFetchCycle,nutchIndex,default</value>
> </property>
>
> The other file to add is conf/capacity-scheduler.conf. This is where
> the properties of the queues are defined. Here is some of my scheduler
> conf
>
> <property>
> <name>mapred.capacity-scheduler.queue.nutchFetchCycle.guaranteed-capacity</name>
> <value>50</value>
> <description>Percentage of the number of slots in the cluster that are
> guaranteed to be available for jobs in this queue.
> </description>
> </property>
> <property>
> <name>mapred.capacity-scheduler.queue.nutchIndex.guaranteed-capacity</name>
> <value>50</value>
> <description>Percentage of the number of slots in the cluster that are
> guaranteed to be available for jobs in this queue.
> </description>
> </property>
> <property>
> <name>mapred.capacity-scheduler.queue.default.guaranteed-capacity</name>
> <value>0</value>
> <description>Percentage of the number of slots in the cluster that are
> guaranteed to be available for jobs in this queue.
> </description>
> </property>
>
> You will need to restart your jobtracker in order for these changes
> to be applied. You will be able to see the queues if you visit the web
> interface of the jobtracker.
>
> To use the different queues you will need to set the
> mapred.job.queue.name property. To accomplish this I have two
> directores, nutch-fetch and nutch-index. Each directory has their own
> conf/hadoop-site.xml file with the different queue names ( I also have
> different number of map/reduce tasks ).
>
> nutch-index:
> <property>
> <name>mapred.job.queue.name</name>
> <value>nutchIndex</value>
> </property>
>
> nutch-fetch:
> <property>
> <name>mapred.job.queue.name</name>
> <value>nutchFetchCycle</value>
> </property>
>
>
> When a job is started from one of the directories the job will be
> placed in the corresponding queue and will be able to run
> simultaneously with jobs in the other queue. In order for this to work
> there needs to be at least the capacity for 2 map and 2 reduce tasks
> so each queue will be guaranteed 1 of each ( in this example since
> it's a 50/50 distribution).
>
> Good luck with your concurrent fetches and don't forget to set the
> generate.update.crawldb property to 'true' so that you will generate
> different fetch lists for each instance.
>
> Enjoy,
> Jason
>
>
> On Thu, Dec 17, 2009 at 1:03 PM, Yves Petinot <yv...@snooth.com> wrote:
>
>> Jason,
>>
>> that sounds really good ! ... did you have to upgrade the default version of
>> Hadoop or were you able to get the distro that comes with Nutch (0.19.1 for
>> me , which I assume is the standard) to accept it ? If the latter worked for
>> you, can you take use through your configuration changes ?
>>
>> thanks a bunch ;-)
>>
>> -y
>>
>> J.G.Konrad wrote:
>>
>>> I have integrated the CapacityTaskScheduler into my Nutch 1.0 setup
>>> although it is not for doing concurrent fetching but it could be used
>>> for that purpose. Using the capacity scheduler you set up two
>>> separate queues and allocate half of the available resources
>>> (map/reduce tasks) to each queue. Technically there are three queues
>>> in my setup , two specialized and the 'default'. The capacity
>>> scheduler requires the 'default' queue to be defined although you
>>> don't have to send any jobs to it.
>>>
>>> The only catch is you are not guaranteed symmetrical distribution if
>>> you have multiple machines in the cluster. That may or may not be a
>>> issues depending on your requirements.
>>>
>>> -Jason
>>>
>>>
>>> On Thu, Dec 17, 2009 at 11:00 AM, Yves Petinot <yv...@snooth.com> wrote:
>>>
>>>
>>>>>> Just one comment though I think hadoop will serialize your jobs any how
>>>>>> so
>>>>>> you won't get a parallel execution of your hadoop jobs unless you run
>>>>>> them
>>>>>> from different hardware.
>>>>>>
>>>>>>
>>>> I'm actually wondering if someone on this list has been able to use
>>>> Hadoop's
>>>> Fair Scheduler (or any other scheduler for that matter). This would
>>>> definitely solve this problem (which i'm experiencing too). Is it at all
>>>> possible to change the default scheduler with Nutch 1.0 (and the version
>>>> of
>>>> Hadoop that comes with it) or do we have to wait until the next hadoop
>>>> upgrade ?
>>>>
>>>> hopefully someone on the list can shed some light on this issue,
>>>>
>>>> cheers,
>>>>
>>>> -y
>>>>
>>>>
>>>> MilleBii wrote:
>>>>
>>>>
>>>>> I guess because of the different nutch-site.xml & url filter that you
>>>>> want
>>>>> to use it won't work... but you could try installing nutch twice run the
>>>>> crawl/fetch/parse from those two locations. And joined the segments to
>>>>> recreate a unified searchable index (make sure you put all your segments
>>>>> under the same location).
>>>>>
>>>>> Just one comment though I think hadoop will serialize your jobs any how
>>>>> so
>>>>> you won't get a parallel execution of your hadoop jobs unless you run
>>>>> them
>>>>> from different hardware.
>>>>>
>>>>> 2009/12/16 Christopher Bader <cb...@gmail.com>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Felix,
>>>>>>
>>>>>> I've had trouble running multiple instances. I would be interested in
>>>>>> hearing from anyone who has done it successfully.
>>>>>>
>>>>>> CB
>>>>>>
>>>>>>
>>>>>> On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I would like to run at least two instances of nutch ONLY for crawling
>>>>>>> at
>>>>>>> one time; one for very frequently updated sites and one for other
>>>>>>> sites.
>>>>>>> Will the nutch instances get in trouble when running several
>>>>>>> crawlscripts, especially the nutch confdir variable?
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Felix.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>
>>
Re: Multiple Nutch instances for crawling?
Posted by "J.G.Konrad" <ko...@gmail.com>.
Nope. I do not know how that version of nutch works. I am not familiar
with that project
On Thu, Dec 17, 2009 at 3:39 PM, Felix Zimmermann
<ma...@felix-zimmermann.eu> wrote:
> Hi Jason,
>
> thank you very much for your detailled description, I'll give this a
> try. Do you know how the multiple instances are realised in the
> nutch-gui on http://github.com/101tec/nutch ?
>
> Felix.
>
> Am Donnerstag, den 17.12.2009, 15:11 -0800 schrieb J.G.Konrad:
>> I have upgraded to 0.19.2 but the capacity scheduler feature is
>> available in 0.19.1. You will need to download the hadoop common
>> package ( http://hadoop.apache.org/common/releases.html ) to get the
>> capacity scheduler jar. It is not included in the Nutch releases.
>>
>> After downloading you will want to copy the
>> contrib/capacity-scheduler/hadoop-0.19.x-capacity-scheduler.jar into
>> lib directory where you have your nutch code. There is also a example
>> configuration file that comes with the hadoop package that is quite
>> explanatory ( conf/capacity-scheduler ). The docs can be found here
>> http://hadoop.apache.org/common/docs/r0.19.1/capacity_scheduler.html
>>
>> The first thing to do is to set the scheduler and define the queues.
>> This is done in the hadoop-site.xml file that is used to start the
>> jobtracker. This is what mine looks like for defining two queues (plus
>> the default).
>>
>> <property>
>> <name>mapred.jobtracker.taskScheduler</name>
>> <value>org.apache.hadoop.mapred.CapacityTaskScheduler</value>
>> </property>
>>
>> <property>
>> <name>mapred.queue.names</name>
>> <value>nutchFetchCycle,nutchIndex,default</value>
>> </property>
>>
>> The other file to add is conf/capacity-scheduler.conf. This is where
>> the properties of the queues are defined. Here is some of my scheduler
>> conf
>>
>> <property>
>> <name>mapred.capacity-scheduler.queue.nutchFetchCycle.guaranteed-capacity</name>
>> <value>50</value>
>> <description>Percentage of the number of slots in the cluster that are
>> guaranteed to be available for jobs in this queue.
>> </description>
>> </property>
>> <property>
>> <name>mapred.capacity-scheduler.queue.nutchIndex.guaranteed-capacity</name>
>> <value>50</value>
>> <description>Percentage of the number of slots in the cluster that are
>> guaranteed to be available for jobs in this queue.
>> </description>
>> </property>
>> <property>
>> <name>mapred.capacity-scheduler.queue.default.guaranteed-capacity</name>
>> <value>0</value>
>> <description>Percentage of the number of slots in the cluster that are
>> guaranteed to be available for jobs in this queue.
>> </description>
>> </property>
>>
>> You will need to restart your jobtracker in order for these changes
>> to be applied. You will be able to see the queues if you visit the web
>> interface of the jobtracker.
>>
>> To use the different queues you will need to set the
>> mapred.job.queue.name property. To accomplish this I have two
>> directores, nutch-fetch and nutch-index. Each directory has their own
>> conf/hadoop-site.xml file with the different queue names ( I also have
>> different number of map/reduce tasks ).
>>
>> nutch-index:
>> <property>
>> <name>mapred.job.queue.name</name>
>> <value>nutchIndex</value>
>> </property>
>>
>> nutch-fetch:
>> <property>
>> <name>mapred.job.queue.name</name>
>> <value>nutchFetchCycle</value>
>> </property>
>>
>>
>> When a job is started from one of the directories the job will be
>> placed in the corresponding queue and will be able to run
>> simultaneously with jobs in the other queue. In order for this to work
>> there needs to be at least the capacity for 2 map and 2 reduce tasks
>> so each queue will be guaranteed 1 of each ( in this example since
>> it's a 50/50 distribution).
>>
>> Good luck with your concurrent fetches and don't forget to set the
>> generate.update.crawldb property to 'true' so that you will generate
>> different fetch lists for each instance.
>>
>> Enjoy,
>> Jason
>>
>>
>> On Thu, Dec 17, 2009 at 1:03 PM, Yves Petinot <yv...@snooth.com> wrote:
>> > Jason,
>> >
>> > that sounds really good ! ... did you have to upgrade the default version of
>> > Hadoop or were you able to get the distro that comes with Nutch (0.19.1 for
>> > me , which I assume is the standard) to accept it ? If the latter worked for
>> > you, can you take use through your configuration changes ?
>> >
>> > thanks a bunch ;-)
>> >
>> > -y
>> >
>> > J.G.Konrad wrote:
>> >>
>> >> I have integrated the CapacityTaskScheduler into my Nutch 1.0 setup
>> >> although it is not for doing concurrent fetching but it could be used
>> >> for that purpose. Using the capacity scheduler you set up two
>> >> separate queues and allocate half of the available resources
>> >> (map/reduce tasks) to each queue. Technically there are three queues
>> >> in my setup , two specialized and the 'default'. The capacity
>> >> scheduler requires the 'default' queue to be defined although you
>> >> don't have to send any jobs to it.
>> >>
>> >> The only catch is you are not guaranteed symmetrical distribution if
>> >> you have multiple machines in the cluster. That may or may not be a
>> >> issues depending on your requirements.
>> >>
>> >> -Jason
>> >>
>> >>
>> >> On Thu, Dec 17, 2009 at 11:00 AM, Yves Petinot <yv...@snooth.com> wrote:
>> >>
>> >>>>>
>> >>>>> Just one comment though I think hadoop will serialize your jobs any how
>> >>>>> so
>> >>>>> you won't get a parallel execution of your hadoop jobs unless you run
>> >>>>> them
>> >>>>> from different hardware.
>> >>>>>
>> >>>
>> >>> I'm actually wondering if someone on this list has been able to use
>> >>> Hadoop's
>> >>> Fair Scheduler (or any other scheduler for that matter). This would
>> >>> definitely solve this problem (which i'm experiencing too). Is it at all
>> >>> possible to change the default scheduler with Nutch 1.0 (and the version
>> >>> of
>> >>> Hadoop that comes with it) or do we have to wait until the next hadoop
>> >>> upgrade ?
>> >>>
>> >>> hopefully someone on the list can shed some light on this issue,
>> >>>
>> >>> cheers,
>> >>>
>> >>> -y
>> >>>
>> >>>
>> >>> MilleBii wrote:
>> >>>
>> >>>>
>> >>>> I guess because of the different nutch-site.xml & url filter that you
>> >>>> want
>> >>>> to use it won't work... but you could try installing nutch twice run the
>> >>>> crawl/fetch/parse from those two locations. And joined the segments to
>> >>>> recreate a unified searchable index (make sure you put all your segments
>> >>>> under the same location).
>> >>>>
>> >>>> Just one comment though I think hadoop will serialize your jobs any how
>> >>>> so
>> >>>> you won't get a parallel execution of your hadoop jobs unless you run
>> >>>> them
>> >>>> from different hardware.
>> >>>>
>> >>>> 2009/12/16 Christopher Bader <cb...@gmail.com>
>> >>>>
>> >>>>
>> >>>>
>> >>>>>
>> >>>>> Felix,
>> >>>>>
>> >>>>> I've had trouble running multiple instances. I would be interested in
>> >>>>> hearing from anyone who has done it successfully.
>> >>>>>
>> >>>>> CB
>> >>>>>
>> >>>>>
>> >>>>> On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>> I would like to run at least two instances of nutch ONLY for crawling
>> >>>>>> at
>> >>>>>> one time; one for very frequently updated sites and one for other
>> >>>>>> sites.
>> >>>>>> Will the nutch instances get in trouble when running several
>> >>>>>> crawlscripts, especially the nutch confdir variable?
>> >>>>>>
>> >>>>>> Thanks!
>> >>>>>> Felix.
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >
>> >
>
>
>
Re: Multiple Nutch instances for crawling?
Posted by Felix Zimmermann <ma...@felix-zimmermann.eu>.
Hi Jason,
thank you very much for your detailled description, I'll give this a
try. Do you know how the multiple instances are realised in the
nutch-gui on http://github.com/101tec/nutch ?
Felix.
Am Donnerstag, den 17.12.2009, 15:11 -0800 schrieb J.G.Konrad:
> I have upgraded to 0.19.2 but the capacity scheduler feature is
> available in 0.19.1. You will need to download the hadoop common
> package ( http://hadoop.apache.org/common/releases.html ) to get the
> capacity scheduler jar. It is not included in the Nutch releases.
>
> After downloading you will want to copy the
> contrib/capacity-scheduler/hadoop-0.19.x-capacity-scheduler.jar into
> lib directory where you have your nutch code. There is also a example
> configuration file that comes with the hadoop package that is quite
> explanatory ( conf/capacity-scheduler ). The docs can be found here
> http://hadoop.apache.org/common/docs/r0.19.1/capacity_scheduler.html
>
> The first thing to do is to set the scheduler and define the queues.
> This is done in the hadoop-site.xml file that is used to start the
> jobtracker. This is what mine looks like for defining two queues (plus
> the default).
>
> <property>
> <name>mapred.jobtracker.taskScheduler</name>
> <value>org.apache.hadoop.mapred.CapacityTaskScheduler</value>
> </property>
>
> <property>
> <name>mapred.queue.names</name>
> <value>nutchFetchCycle,nutchIndex,default</value>
> </property>
>
> The other file to add is conf/capacity-scheduler.conf. This is where
> the properties of the queues are defined. Here is some of my scheduler
> conf
>
> <property>
> <name>mapred.capacity-scheduler.queue.nutchFetchCycle.guaranteed-capacity</name>
> <value>50</value>
> <description>Percentage of the number of slots in the cluster that are
> guaranteed to be available for jobs in this queue.
> </description>
> </property>
> <property>
> <name>mapred.capacity-scheduler.queue.nutchIndex.guaranteed-capacity</name>
> <value>50</value>
> <description>Percentage of the number of slots in the cluster that are
> guaranteed to be available for jobs in this queue.
> </description>
> </property>
> <property>
> <name>mapred.capacity-scheduler.queue.default.guaranteed-capacity</name>
> <value>0</value>
> <description>Percentage of the number of slots in the cluster that are
> guaranteed to be available for jobs in this queue.
> </description>
> </property>
>
> You will need to restart your jobtracker in order for these changes
> to be applied. You will be able to see the queues if you visit the web
> interface of the jobtracker.
>
> To use the different queues you will need to set the
> mapred.job.queue.name property. To accomplish this I have two
> directores, nutch-fetch and nutch-index. Each directory has their own
> conf/hadoop-site.xml file with the different queue names ( I also have
> different number of map/reduce tasks ).
>
> nutch-index:
> <property>
> <name>mapred.job.queue.name</name>
> <value>nutchIndex</value>
> </property>
>
> nutch-fetch:
> <property>
> <name>mapred.job.queue.name</name>
> <value>nutchFetchCycle</value>
> </property>
>
>
> When a job is started from one of the directories the job will be
> placed in the corresponding queue and will be able to run
> simultaneously with jobs in the other queue. In order for this to work
> there needs to be at least the capacity for 2 map and 2 reduce tasks
> so each queue will be guaranteed 1 of each ( in this example since
> it's a 50/50 distribution).
>
> Good luck with your concurrent fetches and don't forget to set the
> generate.update.crawldb property to 'true' so that you will generate
> different fetch lists for each instance.
>
> Enjoy,
> Jason
>
>
> On Thu, Dec 17, 2009 at 1:03 PM, Yves Petinot <yv...@snooth.com> wrote:
> > Jason,
> >
> > that sounds really good ! ... did you have to upgrade the default version of
> > Hadoop or were you able to get the distro that comes with Nutch (0.19.1 for
> > me , which I assume is the standard) to accept it ? If the latter worked for
> > you, can you take use through your configuration changes ?
> >
> > thanks a bunch ;-)
> >
> > -y
> >
> > J.G.Konrad wrote:
> >>
> >> I have integrated the CapacityTaskScheduler into my Nutch 1.0 setup
> >> although it is not for doing concurrent fetching but it could be used
> >> for that purpose. Using the capacity scheduler you set up two
> >> separate queues and allocate half of the available resources
> >> (map/reduce tasks) to each queue. Technically there are three queues
> >> in my setup , two specialized and the 'default'. The capacity
> >> scheduler requires the 'default' queue to be defined although you
> >> don't have to send any jobs to it.
> >>
> >> The only catch is you are not guaranteed symmetrical distribution if
> >> you have multiple machines in the cluster. That may or may not be a
> >> issues depending on your requirements.
> >>
> >> -Jason
> >>
> >>
> >> On Thu, Dec 17, 2009 at 11:00 AM, Yves Petinot <yv...@snooth.com> wrote:
> >>
> >>>>>
> >>>>> Just one comment though I think hadoop will serialize your jobs any how
> >>>>> so
> >>>>> you won't get a parallel execution of your hadoop jobs unless you run
> >>>>> them
> >>>>> from different hardware.
> >>>>>
> >>>
> >>> I'm actually wondering if someone on this list has been able to use
> >>> Hadoop's
> >>> Fair Scheduler (or any other scheduler for that matter). This would
> >>> definitely solve this problem (which i'm experiencing too). Is it at all
> >>> possible to change the default scheduler with Nutch 1.0 (and the version
> >>> of
> >>> Hadoop that comes with it) or do we have to wait until the next hadoop
> >>> upgrade ?
> >>>
> >>> hopefully someone on the list can shed some light on this issue,
> >>>
> >>> cheers,
> >>>
> >>> -y
> >>>
> >>>
> >>> MilleBii wrote:
> >>>
> >>>>
> >>>> I guess because of the different nutch-site.xml & url filter that you
> >>>> want
> >>>> to use it won't work... but you could try installing nutch twice run the
> >>>> crawl/fetch/parse from those two locations. And joined the segments to
> >>>> recreate a unified searchable index (make sure you put all your segments
> >>>> under the same location).
> >>>>
> >>>> Just one comment though I think hadoop will serialize your jobs any how
> >>>> so
> >>>> you won't get a parallel execution of your hadoop jobs unless you run
> >>>> them
> >>>> from different hardware.
> >>>>
> >>>> 2009/12/16 Christopher Bader <cb...@gmail.com>
> >>>>
> >>>>
> >>>>
> >>>>>
> >>>>> Felix,
> >>>>>
> >>>>> I've had trouble running multiple instances. I would be interested in
> >>>>> hearing from anyone who has done it successfully.
> >>>>>
> >>>>> CB
> >>>>>
> >>>>>
> >>>>> On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de>
> >>>>> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I would like to run at least two instances of nutch ONLY for crawling
> >>>>>> at
> >>>>>> one time; one for very frequently updated sites and one for other
> >>>>>> sites.
> >>>>>> Will the nutch instances get in trouble when running several
> >>>>>> crawlscripts, especially the nutch confdir variable?
> >>>>>>
> >>>>>> Thanks!
> >>>>>> Felix.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >
> >
Re: Multiple Nutch instances for crawling?
Posted by "J.G.Konrad" <ko...@gmail.com>.
I have upgraded to 0.19.2 but the capacity scheduler feature is
available in 0.19.1. You will need to download the hadoop common
package ( http://hadoop.apache.org/common/releases.html ) to get the
capacity scheduler jar. It is not included in the Nutch releases.
After downloading you will want to copy the
contrib/capacity-scheduler/hadoop-0.19.x-capacity-scheduler.jar into
lib directory where you have your nutch code. There is also a example
configuration file that comes with the hadoop package that is quite
explanatory ( conf/capacity-scheduler ). The docs can be found here
http://hadoop.apache.org/common/docs/r0.19.1/capacity_scheduler.html
The first thing to do is to set the scheduler and define the queues.
This is done in the hadoop-site.xml file that is used to start the
jobtracker. This is what mine looks like for defining two queues (plus
the default).
<property>
<name>mapred.jobtracker.taskScheduler</name>
<value>org.apache.hadoop.mapred.CapacityTaskScheduler</value>
</property>
<property>
<name>mapred.queue.names</name>
<value>nutchFetchCycle,nutchIndex,default</value>
</property>
The other file to add is conf/capacity-scheduler.conf. This is where
the properties of the queues are defined. Here is some of my scheduler
conf
<property>
<name>mapred.capacity-scheduler.queue.nutchFetchCycle.guaranteed-capacity</name>
<value>50</value>
<description>Percentage of the number of slots in the cluster that are
guaranteed to be available for jobs in this queue.
</description>
</property>
<property>
<name>mapred.capacity-scheduler.queue.nutchIndex.guaranteed-capacity</name>
<value>50</value>
<description>Percentage of the number of slots in the cluster that are
guaranteed to be available for jobs in this queue.
</description>
</property>
<property>
<name>mapred.capacity-scheduler.queue.default.guaranteed-capacity</name>
<value>0</value>
<description>Percentage of the number of slots in the cluster that are
guaranteed to be available for jobs in this queue.
</description>
</property>
You will need to restart your jobtracker in order for these changes
to be applied. You will be able to see the queues if you visit the web
interface of the jobtracker.
To use the different queues you will need to set the
mapred.job.queue.name property. To accomplish this I have two
directores, nutch-fetch and nutch-index. Each directory has their own
conf/hadoop-site.xml file with the different queue names ( I also have
different number of map/reduce tasks ).
nutch-index:
<property>
<name>mapred.job.queue.name</name>
<value>nutchIndex</value>
</property>
nutch-fetch:
<property>
<name>mapred.job.queue.name</name>
<value>nutchFetchCycle</value>
</property>
When a job is started from one of the directories the job will be
placed in the corresponding queue and will be able to run
simultaneously with jobs in the other queue. In order for this to work
there needs to be at least the capacity for 2 map and 2 reduce tasks
so each queue will be guaranteed 1 of each ( in this example since
it's a 50/50 distribution).
Good luck with your concurrent fetches and don't forget to set the
generate.update.crawldb property to 'true' so that you will generate
different fetch lists for each instance.
Enjoy,
Jason
On Thu, Dec 17, 2009 at 1:03 PM, Yves Petinot <yv...@snooth.com> wrote:
> Jason,
>
> that sounds really good ! ... did you have to upgrade the default version of
> Hadoop or were you able to get the distro that comes with Nutch (0.19.1 for
> me , which I assume is the standard) to accept it ? If the latter worked for
> you, can you take use through your configuration changes ?
>
> thanks a bunch ;-)
>
> -y
>
> J.G.Konrad wrote:
>>
>> I have integrated the CapacityTaskScheduler into my Nutch 1.0 setup
>> although it is not for doing concurrent fetching but it could be used
>> for that purpose. Using the capacity scheduler you set up two
>> separate queues and allocate half of the available resources
>> (map/reduce tasks) to each queue. Technically there are three queues
>> in my setup , two specialized and the 'default'. The capacity
>> scheduler requires the 'default' queue to be defined although you
>> don't have to send any jobs to it.
>>
>> The only catch is you are not guaranteed symmetrical distribution if
>> you have multiple machines in the cluster. That may or may not be a
>> issues depending on your requirements.
>>
>> -Jason
>>
>>
>> On Thu, Dec 17, 2009 at 11:00 AM, Yves Petinot <yv...@snooth.com> wrote:
>>
>>>>>
>>>>> Just one comment though I think hadoop will serialize your jobs any how
>>>>> so
>>>>> you won't get a parallel execution of your hadoop jobs unless you run
>>>>> them
>>>>> from different hardware.
>>>>>
>>>
>>> I'm actually wondering if someone on this list has been able to use
>>> Hadoop's
>>> Fair Scheduler (or any other scheduler for that matter). This would
>>> definitely solve this problem (which i'm experiencing too). Is it at all
>>> possible to change the default scheduler with Nutch 1.0 (and the version
>>> of
>>> Hadoop that comes with it) or do we have to wait until the next hadoop
>>> upgrade ?
>>>
>>> hopefully someone on the list can shed some light on this issue,
>>>
>>> cheers,
>>>
>>> -y
>>>
>>>
>>> MilleBii wrote:
>>>
>>>>
>>>> I guess because of the different nutch-site.xml & url filter that you
>>>> want
>>>> to use it won't work... but you could try installing nutch twice run the
>>>> crawl/fetch/parse from those two locations. And joined the segments to
>>>> recreate a unified searchable index (make sure you put all your segments
>>>> under the same location).
>>>>
>>>> Just one comment though I think hadoop will serialize your jobs any how
>>>> so
>>>> you won't get a parallel execution of your hadoop jobs unless you run
>>>> them
>>>> from different hardware.
>>>>
>>>> 2009/12/16 Christopher Bader <cb...@gmail.com>
>>>>
>>>>
>>>>
>>>>>
>>>>> Felix,
>>>>>
>>>>> I've had trouble running multiple instances. I would be interested in
>>>>> hearing from anyone who has done it successfully.
>>>>>
>>>>> CB
>>>>>
>>>>>
>>>>> On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I would like to run at least two instances of nutch ONLY for crawling
>>>>>> at
>>>>>> one time; one for very frequently updated sites and one for other
>>>>>> sites.
>>>>>> Will the nutch instances get in trouble when running several
>>>>>> crawlscripts, especially the nutch confdir variable?
>>>>>>
>>>>>> Thanks!
>>>>>> Felix.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>
>
Re: Multiple Nutch instances for crawling?
Posted by Yves Petinot <yv...@snooth.com>.
Jason,
that sounds really good ! ... did you have to upgrade the default
version of Hadoop or were you able to get the distro that comes with
Nutch (0.19.1 for me , which I assume is the standard) to accept it ? If
the latter worked for you, can you take use through your configuration
changes ?
thanks a bunch ;-)
-y
J.G.Konrad wrote:
> I have integrated the CapacityTaskScheduler into my Nutch 1.0 setup
> although it is not for doing concurrent fetching but it could be used
> for that purpose. Using the capacity scheduler you set up two
> separate queues and allocate half of the available resources
> (map/reduce tasks) to each queue. Technically there are three queues
> in my setup , two specialized and the 'default'. The capacity
> scheduler requires the 'default' queue to be defined although you
> don't have to send any jobs to it.
>
> The only catch is you are not guaranteed symmetrical distribution if
> you have multiple machines in the cluster. That may or may not be a
> issues depending on your requirements.
>
> -Jason
>
>
> On Thu, Dec 17, 2009 at 11:00 AM, Yves Petinot <yv...@snooth.com> wrote:
>
>>>> Just one comment though I think hadoop will serialize your jobs any how
>>>> so
>>>> you won't get a parallel execution of your hadoop jobs unless you run
>>>> them
>>>> from different hardware.
>>>>
>> I'm actually wondering if someone on this list has been able to use Hadoop's
>> Fair Scheduler (or any other scheduler for that matter). This would
>> definitely solve this problem (which i'm experiencing too). Is it at all
>> possible to change the default scheduler with Nutch 1.0 (and the version of
>> Hadoop that comes with it) or do we have to wait until the next hadoop
>> upgrade ?
>>
>> hopefully someone on the list can shed some light on this issue,
>>
>> cheers,
>>
>> -y
>>
>>
>> MilleBii wrote:
>>
>>> I guess because of the different nutch-site.xml & url filter that you want
>>> to use it won't work... but you could try installing nutch twice run the
>>> crawl/fetch/parse from those two locations. And joined the segments to
>>> recreate a unified searchable index (make sure you put all your segments
>>> under the same location).
>>>
>>> Just one comment though I think hadoop will serialize your jobs any how so
>>> you won't get a parallel execution of your hadoop jobs unless you run them
>>> from different hardware.
>>>
>>> 2009/12/16 Christopher Bader <cb...@gmail.com>
>>>
>>>
>>>
>>>> Felix,
>>>>
>>>> I've had trouble running multiple instances. I would be interested in
>>>> hearing from anyone who has done it successfully.
>>>>
>>>> CB
>>>>
>>>>
>>>> On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>> Hi,
>>>>>
>>>>> I would like to run at least two instances of nutch ONLY for crawling at
>>>>> one time; one for very frequently updated sites and one for other sites.
>>>>> Will the nutch instances get in trouble when running several
>>>>> crawlscripts, especially the nutch confdir variable?
>>>>>
>>>>> Thanks!
>>>>> Felix.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>>
Re: Multiple Nutch instances for crawling?
Posted by "J.G.Konrad" <ko...@gmail.com>.
I have integrated the CapacityTaskScheduler into my Nutch 1.0 setup
although it is not for doing concurrent fetching but it could be used
for that purpose. Using the capacity scheduler you set up two
separate queues and allocate half of the available resources
(map/reduce tasks) to each queue. Technically there are three queues
in my setup , two specialized and the 'default'. The capacity
scheduler requires the 'default' queue to be defined although you
don't have to send any jobs to it.
The only catch is you are not guaranteed symmetrical distribution if
you have multiple machines in the cluster. That may or may not be a
issues depending on your requirements.
-Jason
On Thu, Dec 17, 2009 at 11:00 AM, Yves Petinot <yv...@snooth.com> wrote:
>>> Just one comment though I think hadoop will serialize your jobs any how
>>> so
>>> you won't get a parallel execution of your hadoop jobs unless you run
>>> them
>>> from different hardware.
>
> I'm actually wondering if someone on this list has been able to use Hadoop's
> Fair Scheduler (or any other scheduler for that matter). This would
> definitely solve this problem (which i'm experiencing too). Is it at all
> possible to change the default scheduler with Nutch 1.0 (and the version of
> Hadoop that comes with it) or do we have to wait until the next hadoop
> upgrade ?
>
> hopefully someone on the list can shed some light on this issue,
>
> cheers,
>
> -y
>
>
> MilleBii wrote:
>>
>> I guess because of the different nutch-site.xml & url filter that you want
>> to use it won't work... but you could try installing nutch twice run the
>> crawl/fetch/parse from those two locations. And joined the segments to
>> recreate a unified searchable index (make sure you put all your segments
>> under the same location).
>>
>> Just one comment though I think hadoop will serialize your jobs any how so
>> you won't get a parallel execution of your hadoop jobs unless you run them
>> from different hardware.
>>
>> 2009/12/16 Christopher Bader <cb...@gmail.com>
>>
>>
>>>
>>> Felix,
>>>
>>> I've had trouble running multiple instances. I would be interested in
>>> hearing from anyone who has done it successfully.
>>>
>>> CB
>>>
>>>
>>> On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de>
>>> wrote:
>>>
>>>
>>>>
>>>> Hi,
>>>>
>>>> I would like to run at least two instances of nutch ONLY for crawling at
>>>> one time; one for very frequently updated sites and one for other sites.
>>>> Will the nutch instances get in trouble when running several
>>>> crawlscripts, especially the nutch confdir variable?
>>>>
>>>> Thanks!
>>>> Felix.
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>>
>>
>>
>
>
Re: Multiple Nutch instances for crawling?
Posted by Yves Petinot <yv...@snooth.com>.
>>Just one comment though I think hadoop will serialize your jobs any how so
>>you won't get a parallel execution of your hadoop jobs unless you run them
>>from different hardware.
I'm actually wondering if someone on this list has been able to use
Hadoop's Fair Scheduler (or any other scheduler for that matter). This
would definitely solve this problem (which i'm experiencing too). Is it
at all possible to change the default scheduler with Nutch 1.0 (and the
version of Hadoop that comes with it) or do we have to wait until the
next hadoop upgrade ?
hopefully someone on the list can shed some light on this issue,
cheers,
-y
MilleBii wrote:
> I guess because of the different nutch-site.xml & url filter that you want
> to use it won't work... but you could try installing nutch twice run the
> crawl/fetch/parse from those two locations. And joined the segments to
> recreate a unified searchable index (make sure you put all your segments
> under the same location).
>
> Just one comment though I think hadoop will serialize your jobs any how so
> you won't get a parallel execution of your hadoop jobs unless you run them
> from different hardware.
>
> 2009/12/16 Christopher Bader <cb...@gmail.com>
>
>
>> Felix,
>>
>> I've had trouble running multiple instances. I would be interested in
>> hearing from anyone who has done it successfully.
>>
>> CB
>>
>>
>> On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de> wrote:
>>
>>
>>> Hi,
>>>
>>> I would like to run at least two instances of nutch ONLY for crawling at
>>> one time; one for very frequently updated sites and one for other sites.
>>> Will the nutch instances get in trouble when running several
>>> crawlscripts, especially the nutch confdir variable?
>>>
>>> Thanks!
>>> Felix.
>>>
>>>
>>>
>>>
>>>
>
>
>
>
Re: Multiple Nutch instances for crawling?
Posted by "J.G.Konrad" <ko...@gmail.com>.
The crawl directory ( crawldb, linkdb, etc. ) can be shared between jobs
although depending on the implementation you may run into locking issues.
For example if there is a 'generate' job running while a 'updatedb' job is
trying to be executed ( on the same crawldb ) the 'updatedb' job will faile
because the 'generate' job has a lock on the crawldb.
On Thu, Dec 17, 2009 at 10:15 PM, Jun Mao <Ju...@symantec.com> wrote:
> Is that still true if I start two jobs( they will not share crawdb,linkdb)
and write index to two different locations?
>
> Thanks,
>
> Jun
>
> -----Original Message-----
> From: MilleBii [mailto:millebii@gmail.com]
> Sent: 2009年12月17日 16:57
> To: nutch-user@lucene.apache.org
> Subject: Re: Multiple Nutch instances for crawling?
>
> I guess because of the different nutch-site.xml & url filter that you want
> to use it won't work... but you could try installing nutch twice run the
> crawl/fetch/parse from those two locations. And joined the segments to
> recreate a unified searchable index (make sure you put all your segments
> under the same location).
>
> Just one comment though I think hadoop will serialize your jobs any how so
> you won't get a parallel execution of your hadoop jobs unless you run them
> from different hardware.
>
> 2009/12/16 Christopher Bader <cb...@gmail.com>
>
>> Felix,
>>
>> I've had trouble running multiple instances. I would be interested in
>> hearing from anyone who has done it successfully.
>>
>> CB
>>
>>
>> On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de>
wrote:
>>
>> > Hi,
>> >
>> > I would like to run at least two instances of nutch ONLY for crawling
at
>> > one time; one for very frequently updated sites and one for other
sites.
>> > Will the nutch instances get in trouble when running several
>> > crawlscripts, especially the nutch confdir variable?
>> >
>> > Thanks!
>> > Felix.
>> >
>> >
>> >
>> >
>>
>
>
>
> --
> -MilleBii-
>
RE: Multiple Nutch instances for crawling?
Posted by Jun Mao <Ju...@symantec.com>.
Is that still true if I start two jobs( they will not share crawdb,linkdb) and write index to two different locations?
Thanks,
Jun
-----Original Message-----
From: MilleBii [mailto:millebii@gmail.com]
Sent: 2009年12月17日 16:57
To: nutch-user@lucene.apache.org
Subject: Re: Multiple Nutch instances for crawling?
I guess because of the different nutch-site.xml & url filter that you want
to use it won't work... but you could try installing nutch twice run the
crawl/fetch/parse from those two locations. And joined the segments to
recreate a unified searchable index (make sure you put all your segments
under the same location).
Just one comment though I think hadoop will serialize your jobs any how so
you won't get a parallel execution of your hadoop jobs unless you run them
from different hardware.
2009/12/16 Christopher Bader <cb...@gmail.com>
> Felix,
>
> I've had trouble running multiple instances. I would be interested in
> hearing from anyone who has done it successfully.
>
> CB
>
>
> On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de> wrote:
>
> > Hi,
> >
> > I would like to run at least two instances of nutch ONLY for crawling at
> > one time; one for very frequently updated sites and one for other sites.
> > Will the nutch instances get in trouble when running several
> > crawlscripts, especially the nutch confdir variable?
> >
> > Thanks!
> > Felix.
> >
> >
> >
> >
>
--
-MilleBii-
Re: Multiple Nutch instances for crawling?
Posted by MilleBii <mi...@gmail.com>.
I guess because of the different nutch-site.xml & url filter that you want
to use it won't work... but you could try installing nutch twice run the
crawl/fetch/parse from those two locations. And joined the segments to
recreate a unified searchable index (make sure you put all your segments
under the same location).
Just one comment though I think hadoop will serialize your jobs any how so
you won't get a parallel execution of your hadoop jobs unless you run them
from different hardware.
2009/12/16 Christopher Bader <cb...@gmail.com>
> Felix,
>
> I've had trouble running multiple instances. I would be interested in
> hearing from anyone who has done it successfully.
>
> CB
>
>
> On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de> wrote:
>
> > Hi,
> >
> > I would like to run at least two instances of nutch ONLY for crawling at
> > one time; one for very frequently updated sites and one for other sites.
> > Will the nutch instances get in trouble when running several
> > crawlscripts, especially the nutch confdir variable?
> >
> > Thanks!
> > Felix.
> >
> >
> >
> >
>
--
-MilleBii-
Re: Multiple Nutch instances for crawling?
Posted by Christopher Bader <cb...@gmail.com>.
Felix,
I've had trouble running multiple instances. I would be interested in
hearing from anyone who has done it successfully.
CB
On Wed, Dec 16, 2009 at 4:26 PM, Felix Zimmermann <fe...@gmx.de> wrote:
> Hi,
>
> I would like to run at least two instances of nutch ONLY for crawling at
> one time; one for very frequently updated sites and one for other sites.
> Will the nutch instances get in trouble when running several
> crawlscripts, especially the nutch confdir variable?
>
> Thanks!
> Felix.
>
>
>
>