You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Joe Zhang <sm...@gmail.com> on 2012/11/29 21:52:08 UTC
The "topN" parameter in nutch crawl
Dear list,
This parameter is causing me some confusion. To me, there are at 3 possible
meanings for "topN":
1. The branching factor at a given node
2. *"*the maximum number of pages that will be retrieved at each level up
to the depth" (from the wiki), which seems to refer to the total of
branching factors at any given level
3. The size of the entire frontier/queue
To me, (1) makes the most sense, and (3) is the easiest to implement
programming-wise.
If (2) is the actual implementation in nutch, it means the effective
branching factor would be lower at deeper levels, correct?
In this sense, in order to conduct a "comprehensive" crawl, if we have to
trade off between "depth" and "topN", we should probably favor larger
"topN"? In other words, "-depth 5 -topN 1000" would make more sense than
"-depth 10 -topN 100" for a comprehensive crawl, correct?
Thanks!
RE: The "topN" parameter in nutch crawl
Posted by Markus Jelsma <ma...@openindex.io>.
Yes :)
-----Original message-----
> From:Joe Zhang <sm...@gmail.com>
> Sent: Thu 29-Nov-2012 23:55
> To: user@nutch.apache.org
> Subject: Re: The "topN" parameter in nutch crawl
>
> Thanks!
>
> so the best way to think about "-depth 5 -topN 100" would be "5 batches of
> 100 ULRs", correct?
>
> On Thu, Nov 29, 2012 at 3:42 PM, Markus Jelsma
> <ma...@openindex.io>wrote:
>
> >
> > -----Original message-----
> > > From:Joe Zhang <sm...@gmail.com>
> > > Sent: Thu 29-Nov-2012 23:33
> > > To: user@nutch.apache.org
> > > Subject: Re: The "topN" parameter in nutch crawl
> > >
> > > I'm not sure I completely understand.
> > >
> > > Typically when we think about the crawling problem as one of graph
> > > traversal, the # of nodes visited would some exponential function.
> > >
> > > Are you saying that this is not true, and if we specify somehting like
> > > "-depth 5 -topN 100", we'll at most visit 500 nodes?
> >
> > Yes. Nutch generates fetch lists from the CrawlDB which is nothing more
> > than a sorted list of URL (score, than alphabetically). It just picks the
> > first eligible URL in the sorted list. You really should take a good look
> > at the Generator code, it'll answer most of your questions.
> >
> >
> >
> > >
> > > On Thu, Nov 29, 2012 at 3:03 PM, Markus Jelsma
> > > <ma...@openindex.io>wrote:
> > >
> > > > Nutch does neither. If scoring is used the records to fetch are
> > ordered by
> > > > score and if there is no score it's simply sorted alphabetically. With
> > some
> > > > tuning to a scoring filter you can do whatever you want but in the end
> > > > everything is going to be crawled (if there are enough resources).
> > > >
> > > > What are you trying to do? If you're not going to process many
> > millions of
> > > > records it doesn't really matter because all records will be fetched
> > within
> > > > a reasonable amount of time.
> > > >
> > > > -----Original message-----
> > > > > From:Joe Zhang <sm...@gmail.com>
> > > > > Sent: Thu 29-Nov-2012 22:45
> > > > > To: user@nutch.apache.org
> > > > > Subject: Re: The "topN" parameter in nutch crawl
> > > > >
> > > > > How would you characterize the crawling algorithm? Depth-first,
> > > > > breath-first, or some heuristic-based?
> > > > >
> > > > > On Thu, Nov 29, 2012 at 2:10 PM, Markus Jelsma
> > > > > <ma...@openindex.io>wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > None of all three. the topN-parameter simply means that the
> > generator
> > > > will
> > > > > > select up to N records to fetch for each time it is invoked. It's
> > best
> > > > to
> > > > > > forget the notion of depth in crawling, it has little meaning in
> > most
> > > > > > cases. Usually one will just continously crawl until there are no
> > more
> > > > > > records to fetch.
> > > > > >
> > > > > > We continously invoke the crawler and tell it to do something. If
> > > > there's
> > > > > > nothing to do (but that never happens) we just invoke it again the
> > next
> > > > > > time.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > >
> > > > > > -----Original message-----
> > > > > > > From:Joe Zhang <sm...@gmail.com>
> > > > > > > Sent: Thu 29-Nov-2012 21:58
> > > > > > > To: user <us...@nutch.apache.org>
> > > > > > > Subject: The "topN" parameter in nutch crawl
> > > > > > >
> > > > > > > Dear list,
> > > > > > >
> > > > > > > This parameter is causing me some confusion. To me, there are at
> > 3
> > > > > > possible
> > > > > > > meanings for "topN":
> > > > > > >
> > > > > > > 1. The branching factor at a given node
> > > > > > > 2. *"*the maximum number of pages that will be retrieved at each
> > > > level up
> > > > > > > to the depth" (from the wiki), which seems to refer to the total
> > of
> > > > > > > branching factors at any given level
> > > > > > > 3. The size of the entire frontier/queue
> > > > > > >
> > > > > > > To me, (1) makes the most sense, and (3) is the easiest to
> > implement
> > > > > > > programming-wise.
> > > > > > >
> > > > > > > If (2) is the actual implementation in nutch, it means the
> > effective
> > > > > > > branching factor would be lower at deeper levels, correct?
> > > > > > >
> > > > > > > In this sense, in order to conduct a "comprehensive" crawl, if we
> > > > have to
> > > > > > > trade off between "depth" and "topN", we should probably favor
> > larger
> > > > > > > "topN"? In other words, "-depth 5 -topN 1000" would make more
> > sense
> > > > than
> > > > > > > "-depth 10 -topN 100" for a comprehensive crawl, correct?
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: The "topN" parameter in nutch crawl
Posted by Joe Zhang <sm...@gmail.com>.
Thanks!
so the best way to think about "-depth 5 -topN 100" would be "5 batches of
100 ULRs", correct?
On Thu, Nov 29, 2012 at 3:42 PM, Markus Jelsma
<ma...@openindex.io>wrote:
>
> -----Original message-----
> > From:Joe Zhang <sm...@gmail.com>
> > Sent: Thu 29-Nov-2012 23:33
> > To: user@nutch.apache.org
> > Subject: Re: The "topN" parameter in nutch crawl
> >
> > I'm not sure I completely understand.
> >
> > Typically when we think about the crawling problem as one of graph
> > traversal, the # of nodes visited would some exponential function.
> >
> > Are you saying that this is not true, and if we specify somehting like
> > "-depth 5 -topN 100", we'll at most visit 500 nodes?
>
> Yes. Nutch generates fetch lists from the CrawlDB which is nothing more
> than a sorted list of URL (score, than alphabetically). It just picks the
> first eligible URL in the sorted list. You really should take a good look
> at the Generator code, it'll answer most of your questions.
>
>
>
> >
> > On Thu, Nov 29, 2012 at 3:03 PM, Markus Jelsma
> > <ma...@openindex.io>wrote:
> >
> > > Nutch does neither. If scoring is used the records to fetch are
> ordered by
> > > score and if there is no score it's simply sorted alphabetically. With
> some
> > > tuning to a scoring filter you can do whatever you want but in the end
> > > everything is going to be crawled (if there are enough resources).
> > >
> > > What are you trying to do? If you're not going to process many
> millions of
> > > records it doesn't really matter because all records will be fetched
> within
> > > a reasonable amount of time.
> > >
> > > -----Original message-----
> > > > From:Joe Zhang <sm...@gmail.com>
> > > > Sent: Thu 29-Nov-2012 22:45
> > > > To: user@nutch.apache.org
> > > > Subject: Re: The "topN" parameter in nutch crawl
> > > >
> > > > How would you characterize the crawling algorithm? Depth-first,
> > > > breath-first, or some heuristic-based?
> > > >
> > > > On Thu, Nov 29, 2012 at 2:10 PM, Markus Jelsma
> > > > <ma...@openindex.io>wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > None of all three. the topN-parameter simply means that the
> generator
> > > will
> > > > > select up to N records to fetch for each time it is invoked. It's
> best
> > > to
> > > > > forget the notion of depth in crawling, it has little meaning in
> most
> > > > > cases. Usually one will just continously crawl until there are no
> more
> > > > > records to fetch.
> > > > >
> > > > > We continously invoke the crawler and tell it to do something. If
> > > there's
> > > > > nothing to do (but that never happens) we just invoke it again the
> next
> > > > > time.
> > > > >
> > > > > Cheers,
> > > > >
> > > > >
> > > > > -----Original message-----
> > > > > > From:Joe Zhang <sm...@gmail.com>
> > > > > > Sent: Thu 29-Nov-2012 21:58
> > > > > > To: user <us...@nutch.apache.org>
> > > > > > Subject: The "topN" parameter in nutch crawl
> > > > > >
> > > > > > Dear list,
> > > > > >
> > > > > > This parameter is causing me some confusion. To me, there are at
> 3
> > > > > possible
> > > > > > meanings for "topN":
> > > > > >
> > > > > > 1. The branching factor at a given node
> > > > > > 2. *"*the maximum number of pages that will be retrieved at each
> > > level up
> > > > > > to the depth" (from the wiki), which seems to refer to the total
> of
> > > > > > branching factors at any given level
> > > > > > 3. The size of the entire frontier/queue
> > > > > >
> > > > > > To me, (1) makes the most sense, and (3) is the easiest to
> implement
> > > > > > programming-wise.
> > > > > >
> > > > > > If (2) is the actual implementation in nutch, it means the
> effective
> > > > > > branching factor would be lower at deeper levels, correct?
> > > > > >
> > > > > > In this sense, in order to conduct a "comprehensive" crawl, if we
> > > have to
> > > > > > trade off between "depth" and "topN", we should probably favor
> larger
> > > > > > "topN"? In other words, "-depth 5 -topN 1000" would make more
> sense
> > > than
> > > > > > "-depth 10 -topN 100" for a comprehensive crawl, correct?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > >
> > > >
> > >
> >
>
RE: The "topN" parameter in nutch crawl
Posted by Markus Jelsma <ma...@openindex.io>.
-----Original message-----
> From:Joe Zhang <sm...@gmail.com>
> Sent: Thu 29-Nov-2012 23:33
> To: user@nutch.apache.org
> Subject: Re: The "topN" parameter in nutch crawl
>
> I'm not sure I completely understand.
>
> Typically when we think about the crawling problem as one of graph
> traversal, the # of nodes visited would some exponential function.
>
> Are you saying that this is not true, and if we specify somehting like
> "-depth 5 -topN 100", we'll at most visit 500 nodes?
Yes. Nutch generates fetch lists from the CrawlDB which is nothing more than a sorted list of URL (score, than alphabetically). It just picks the first eligible URL in the sorted list. You really should take a good look at the Generator code, it'll answer most of your questions.
>
> On Thu, Nov 29, 2012 at 3:03 PM, Markus Jelsma
> <ma...@openindex.io>wrote:
>
> > Nutch does neither. If scoring is used the records to fetch are ordered by
> > score and if there is no score it's simply sorted alphabetically. With some
> > tuning to a scoring filter you can do whatever you want but in the end
> > everything is going to be crawled (if there are enough resources).
> >
> > What are you trying to do? If you're not going to process many millions of
> > records it doesn't really matter because all records will be fetched within
> > a reasonable amount of time.
> >
> > -----Original message-----
> > > From:Joe Zhang <sm...@gmail.com>
> > > Sent: Thu 29-Nov-2012 22:45
> > > To: user@nutch.apache.org
> > > Subject: Re: The "topN" parameter in nutch crawl
> > >
> > > How would you characterize the crawling algorithm? Depth-first,
> > > breath-first, or some heuristic-based?
> > >
> > > On Thu, Nov 29, 2012 at 2:10 PM, Markus Jelsma
> > > <ma...@openindex.io>wrote:
> > >
> > > > Hi,
> > > >
> > > > None of all three. the topN-parameter simply means that the generator
> > will
> > > > select up to N records to fetch for each time it is invoked. It's best
> > to
> > > > forget the notion of depth in crawling, it has little meaning in most
> > > > cases. Usually one will just continously crawl until there are no more
> > > > records to fetch.
> > > >
> > > > We continously invoke the crawler and tell it to do something. If
> > there's
> > > > nothing to do (but that never happens) we just invoke it again the next
> > > > time.
> > > >
> > > > Cheers,
> > > >
> > > >
> > > > -----Original message-----
> > > > > From:Joe Zhang <sm...@gmail.com>
> > > > > Sent: Thu 29-Nov-2012 21:58
> > > > > To: user <us...@nutch.apache.org>
> > > > > Subject: The "topN" parameter in nutch crawl
> > > > >
> > > > > Dear list,
> > > > >
> > > > > This parameter is causing me some confusion. To me, there are at 3
> > > > possible
> > > > > meanings for "topN":
> > > > >
> > > > > 1. The branching factor at a given node
> > > > > 2. *"*the maximum number of pages that will be retrieved at each
> > level up
> > > > > to the depth" (from the wiki), which seems to refer to the total of
> > > > > branching factors at any given level
> > > > > 3. The size of the entire frontier/queue
> > > > >
> > > > > To me, (1) makes the most sense, and (3) is the easiest to implement
> > > > > programming-wise.
> > > > >
> > > > > If (2) is the actual implementation in nutch, it means the effective
> > > > > branching factor would be lower at deeper levels, correct?
> > > > >
> > > > > In this sense, in order to conduct a "comprehensive" crawl, if we
> > have to
> > > > > trade off between "depth" and "topN", we should probably favor larger
> > > > > "topN"? In other words, "-depth 5 -topN 1000" would make more sense
> > than
> > > > > "-depth 10 -topN 100" for a comprehensive crawl, correct?
> > > > >
> > > > > Thanks!
> > > > >
> > > >
> > >
> >
>
Re: The "topN" parameter in nutch crawl
Posted by Joe Zhang <sm...@gmail.com>.
I'm not sure I completely understand.
Typically when we think about the crawling problem as one of graph
traversal, the # of nodes visited would some exponential function.
Are you saying that this is not true, and if we specify somehting like
"-depth 5 -topN 100", we'll at most visit 500 nodes?
On Thu, Nov 29, 2012 at 3:03 PM, Markus Jelsma
<ma...@openindex.io>wrote:
> Nutch does neither. If scoring is used the records to fetch are ordered by
> score and if there is no score it's simply sorted alphabetically. With some
> tuning to a scoring filter you can do whatever you want but in the end
> everything is going to be crawled (if there are enough resources).
>
> What are you trying to do? If you're not going to process many millions of
> records it doesn't really matter because all records will be fetched within
> a reasonable amount of time.
>
> -----Original message-----
> > From:Joe Zhang <sm...@gmail.com>
> > Sent: Thu 29-Nov-2012 22:45
> > To: user@nutch.apache.org
> > Subject: Re: The "topN" parameter in nutch crawl
> >
> > How would you characterize the crawling algorithm? Depth-first,
> > breath-first, or some heuristic-based?
> >
> > On Thu, Nov 29, 2012 at 2:10 PM, Markus Jelsma
> > <ma...@openindex.io>wrote:
> >
> > > Hi,
> > >
> > > None of all three. the topN-parameter simply means that the generator
> will
> > > select up to N records to fetch for each time it is invoked. It's best
> to
> > > forget the notion of depth in crawling, it has little meaning in most
> > > cases. Usually one will just continously crawl until there are no more
> > > records to fetch.
> > >
> > > We continously invoke the crawler and tell it to do something. If
> there's
> > > nothing to do (but that never happens) we just invoke it again the next
> > > time.
> > >
> > > Cheers,
> > >
> > >
> > > -----Original message-----
> > > > From:Joe Zhang <sm...@gmail.com>
> > > > Sent: Thu 29-Nov-2012 21:58
> > > > To: user <us...@nutch.apache.org>
> > > > Subject: The "topN" parameter in nutch crawl
> > > >
> > > > Dear list,
> > > >
> > > > This parameter is causing me some confusion. To me, there are at 3
> > > possible
> > > > meanings for "topN":
> > > >
> > > > 1. The branching factor at a given node
> > > > 2. *"*the maximum number of pages that will be retrieved at each
> level up
> > > > to the depth" (from the wiki), which seems to refer to the total of
> > > > branching factors at any given level
> > > > 3. The size of the entire frontier/queue
> > > >
> > > > To me, (1) makes the most sense, and (3) is the easiest to implement
> > > > programming-wise.
> > > >
> > > > If (2) is the actual implementation in nutch, it means the effective
> > > > branching factor would be lower at deeper levels, correct?
> > > >
> > > > In this sense, in order to conduct a "comprehensive" crawl, if we
> have to
> > > > trade off between "depth" and "topN", we should probably favor larger
> > > > "topN"? In other words, "-depth 5 -topN 1000" would make more sense
> than
> > > > "-depth 10 -topN 100" for a comprehensive crawl, correct?
> > > >
> > > > Thanks!
> > > >
> > >
> >
>
RE: The "topN" parameter in nutch crawl
Posted by Markus Jelsma <ma...@openindex.io>.
Nutch does neither. If scoring is used the records to fetch are ordered by score and if there is no score it's simply sorted alphabetically. With some tuning to a scoring filter you can do whatever you want but in the end everything is going to be crawled (if there are enough resources).
What are you trying to do? If you're not going to process many millions of records it doesn't really matter because all records will be fetched within a reasonable amount of time.
-----Original message-----
> From:Joe Zhang <sm...@gmail.com>
> Sent: Thu 29-Nov-2012 22:45
> To: user@nutch.apache.org
> Subject: Re: The "topN" parameter in nutch crawl
>
> How would you characterize the crawling algorithm? Depth-first,
> breath-first, or some heuristic-based?
>
> On Thu, Nov 29, 2012 at 2:10 PM, Markus Jelsma
> <ma...@openindex.io>wrote:
>
> > Hi,
> >
> > None of all three. the topN-parameter simply means that the generator will
> > select up to N records to fetch for each time it is invoked. It's best to
> > forget the notion of depth in crawling, it has little meaning in most
> > cases. Usually one will just continously crawl until there are no more
> > records to fetch.
> >
> > We continously invoke the crawler and tell it to do something. If there's
> > nothing to do (but that never happens) we just invoke it again the next
> > time.
> >
> > Cheers,
> >
> >
> > -----Original message-----
> > > From:Joe Zhang <sm...@gmail.com>
> > > Sent: Thu 29-Nov-2012 21:58
> > > To: user <us...@nutch.apache.org>
> > > Subject: The "topN" parameter in nutch crawl
> > >
> > > Dear list,
> > >
> > > This parameter is causing me some confusion. To me, there are at 3
> > possible
> > > meanings for "topN":
> > >
> > > 1. The branching factor at a given node
> > > 2. *"*the maximum number of pages that will be retrieved at each level up
> > > to the depth" (from the wiki), which seems to refer to the total of
> > > branching factors at any given level
> > > 3. The size of the entire frontier/queue
> > >
> > > To me, (1) makes the most sense, and (3) is the easiest to implement
> > > programming-wise.
> > >
> > > If (2) is the actual implementation in nutch, it means the effective
> > > branching factor would be lower at deeper levels, correct?
> > >
> > > In this sense, in order to conduct a "comprehensive" crawl, if we have to
> > > trade off between "depth" and "topN", we should probably favor larger
> > > "topN"? In other words, "-depth 5 -topN 1000" would make more sense than
> > > "-depth 10 -topN 100" for a comprehensive crawl, correct?
> > >
> > > Thanks!
> > >
> >
>
Re: The "topN" parameter in nutch crawl
Posted by Joe Zhang <sm...@gmail.com>.
How would you characterize the crawling algorithm? Depth-first,
breath-first, or some heuristic-based?
On Thu, Nov 29, 2012 at 2:10 PM, Markus Jelsma
<ma...@openindex.io>wrote:
> Hi,
>
> None of all three. the topN-parameter simply means that the generator will
> select up to N records to fetch for each time it is invoked. It's best to
> forget the notion of depth in crawling, it has little meaning in most
> cases. Usually one will just continously crawl until there are no more
> records to fetch.
>
> We continously invoke the crawler and tell it to do something. If there's
> nothing to do (but that never happens) we just invoke it again the next
> time.
>
> Cheers,
>
>
> -----Original message-----
> > From:Joe Zhang <sm...@gmail.com>
> > Sent: Thu 29-Nov-2012 21:58
> > To: user <us...@nutch.apache.org>
> > Subject: The "topN" parameter in nutch crawl
> >
> > Dear list,
> >
> > This parameter is causing me some confusion. To me, there are at 3
> possible
> > meanings for "topN":
> >
> > 1. The branching factor at a given node
> > 2. *"*the maximum number of pages that will be retrieved at each level up
> > to the depth" (from the wiki), which seems to refer to the total of
> > branching factors at any given level
> > 3. The size of the entire frontier/queue
> >
> > To me, (1) makes the most sense, and (3) is the easiest to implement
> > programming-wise.
> >
> > If (2) is the actual implementation in nutch, it means the effective
> > branching factor would be lower at deeper levels, correct?
> >
> > In this sense, in order to conduct a "comprehensive" crawl, if we have to
> > trade off between "depth" and "topN", we should probably favor larger
> > "topN"? In other words, "-depth 5 -topN 1000" would make more sense than
> > "-depth 10 -topN 100" for a comprehensive crawl, correct?
> >
> > Thanks!
> >
>
RE: The "topN" parameter in nutch crawl
Posted by Markus Jelsma <ma...@openindex.io>.
Hi,
None of all three. the topN-parameter simply means that the generator will select up to N records to fetch for each time it is invoked. It's best to forget the notion of depth in crawling, it has little meaning in most cases. Usually one will just continously crawl until there are no more records to fetch.
We continously invoke the crawler and tell it to do something. If there's nothing to do (but that never happens) we just invoke it again the next time.
Cheers,
-----Original message-----
> From:Joe Zhang <sm...@gmail.com>
> Sent: Thu 29-Nov-2012 21:58
> To: user <us...@nutch.apache.org>
> Subject: The "topN" parameter in nutch crawl
>
> Dear list,
>
> This parameter is causing me some confusion. To me, there are at 3 possible
> meanings for "topN":
>
> 1. The branching factor at a given node
> 2. *"*the maximum number of pages that will be retrieved at each level up
> to the depth" (from the wiki), which seems to refer to the total of
> branching factors at any given level
> 3. The size of the entire frontier/queue
>
> To me, (1) makes the most sense, and (3) is the easiest to implement
> programming-wise.
>
> If (2) is the actual implementation in nutch, it means the effective
> branching factor would be lower at deeper levels, correct?
>
> In this sense, in order to conduct a "comprehensive" crawl, if we have to
> trade off between "depth" and "topN", we should probably favor larger
> "topN"? In other words, "-depth 5 -topN 1000" would make more sense than
> "-depth 10 -topN 100" for a comprehensive crawl, correct?
>
> Thanks!
>