You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Anton Potehin <an...@orbita1.ru> on 2005/11/08 14:31:20 UTC
questions
After I looked thru Crawl.java I exploded all tasks for several phases:
1) Inject - here we add web-links into crawlDb
2) Generate segment - here we create data segment
3) Fetching
4) Parse segment
5) Update crawlDb - here the information is added from segment
into crawlDb
6) Phase 2 - 5 is repeated several times
7) Link db
I can't understand how the clusterization is performed. What phases may
be performed parallel on several machines and how jobs may be separated
for several machines.
What is performed at 7th phase?
Re: questions
Posted by Stefan Groschupf <sg...@media-style.com>.
> Does it mean that every job at every phase may be separated for
> several
> machines (for example: "generate" or every rest phases may be
> performed
> parallel on several machines)?
>
Yes.
> Give us URL for presentation on wiki please?
>
... nutch.org ... wiki ?!
> -----Original Message-----
> From: Stefan Groschupf [mailto:sg@media-style.com]
> Sent: Tuesday, November 08, 2005 4:47 PM
> To: nutch-dev@lucene.apache.org
> Subject: Re: questions
>
> clustering is done until searching and only the first 200 hists are
> clustered.
> parsing is normally done u ntil fetching.
> map reduce split all jobs into several tasks and reduce the results
> together.
> you will find some presentation slides in the wiki,
> HTH
> Stefan
> Am 08.11.2005 um 14:31 schrieb Anton Potehin:
>
>> After I looked thru Crawl.java I exploded all tasks for several
>> phases:
>>
>> 1) Inject - here we add web-links into crawlDb
>>
>> 2) Generate segment - here we create data segment
>>
>> 3) Fetching
>>
>> 4) Parse segment
>>
>> 5) Update crawlDb - here the information is added from segment
>> into crawlDb
>>
>> 6) Phase 2 - 5 is repeated several times
>>
>> 7) Link db
>>
>>
>>
>> I can't understand how the clusterization is performed. What phases
>> may
>> be performed parallel on several machines and how jobs may be
>> separated
>> for several machines.
>>
>> What is performed at 7th phase?
>>
>>
>>
>
>
>
>
RE: questions
Posted by an...@orbita1.ru.
Does it mean that every job at every phase may be separated for several
machines (for example: "generate" or every rest phases may be performed
parallel on several machines)?
Give us URL for presentation on wiki please?
-----Original Message-----
From: Stefan Groschupf [mailto:sg@media-style.com]
Sent: Tuesday, November 08, 2005 4:47 PM
To: nutch-dev@lucene.apache.org
Subject: Re: questions
clustering is done until searching and only the first 200 hists are
clustered.
parsing is normally done u ntil fetching.
map reduce split all jobs into several tasks and reduce the results
together.
you will find some presentation slides in the wiki,
HTH
Stefan
Am 08.11.2005 um 14:31 schrieb Anton Potehin:
> After I looked thru Crawl.java I exploded all tasks for several
> phases:
>
> 1) Inject - here we add web-links into crawlDb
>
> 2) Generate segment - here we create data segment
>
> 3) Fetching
>
> 4) Parse segment
>
> 5) Update crawlDb - here the information is added from segment
> into crawlDb
>
> 6) Phase 2 - 5 is repeated several times
>
> 7) Link db
>
>
>
> I can't understand how the clusterization is performed. What phases
> may
> be performed parallel on several machines and how jobs may be
> separated
> for several machines.
>
> What is performed at 7th phase?
>
>
>
Re: questions
Posted by Stefan Groschupf <sg...@media-style.com>.
clustering is done until searching and only the first 200 hists are
clustered.
parsing is normally done u ntil fetching.
map reduce split all jobs into several tasks and reduce the results
together.
you will find some presentation slides in the wiki,
HTH
Stefan
Am 08.11.2005 um 14:31 schrieb Anton Potehin:
> After I looked thru Crawl.java I exploded all tasks for several
> phases:
>
> 1) Inject - here we add web-links into crawlDb
>
> 2) Generate segment - here we create data segment
>
> 3) Fetching
>
> 4) Parse segment
>
> 5) Update crawlDb - here the information is added from segment
> into crawlDb
>
> 6) Phase 2 - 5 is repeated several times
>
> 7) Link db
>
>
>
> I can't understand how the clusterization is performed. What phases
> may
> be performed parallel on several machines and how jobs may be
> separated
> for several machines.
>
> What is performed at 7th phase?
>
>
>