You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Marcel T <md...@hotmail.com> on 2008/11/17 06:49:26 UTC

alternation of topN

Hi, there,
 
In the example of nutch crawling, there is a topN parameter. This parameter sets limit on the total crawled pages in each depth level. This limit is on all urls.  Is there any way I set a limit on each page. In other words, for example the limit is 5, which means at most 5 more linked will be added into the queue from each url.
 
Thanks.

RE: alternation of topN

Posted by Marcel T <md...@hotmail.com>.

thanks, John!> Date: Sun, 16 Nov 2008 22:07:44 -0800> From: jlogan@nxar.com> To: nutch-user@lucene.apache.org> Subject: Re: alternation of topN> > Marcel,> > set the following property in nutch-site.xml...> > <property>> <name>db.max.outlinks.per.page</name>> <value>5</value>> <description>The maximum number of outlinks that we'll process for a page.> If this value is nonnegative (>=0), at most db.max.outlinks.per.page > outlinks> will be processed for a page; otherwise, all outlinks will be processed.> </description>> </property>> > John> > Marcel T wrote:> > Hi, there,> > > > In the example of nutch crawling, there is a topN parameter. This parameter sets limit on the total crawled pages in each depth level. This limit is on all urls. Is there any way I set a limit on each page. In other words, for example the limit is 5, which means at most 5 more linked will be added into the queue from each url.> > > > Thanks.> > >

Re: alternation of topN

Posted by John Logan <jl...@nxar.com>.

Marcel,

set the following property in nutch-site.xml...

<property>
  <name>db.max.outlinks.per.page</name>
  <value>5</value>
  <description>The maximum number of outlinks that we'll process for a page.
  If this value is nonnegative (>=0), at most db.max.outlinks.per.page 
outlinks
  will be processed for a page; otherwise, all outlinks will be processed.
  </description>
</property>

John

Marcel T wrote:
> Hi, there,
>  
> In the example of nutch crawling, there is a topN parameter. This parameter sets limit on the total crawled pages in each depth level. This limit is on all urls.  Is there any way I set a limit on each page. In other words, for example the limit is 5, which means at most 5 more linked will be added into the queue from each url.
>  
> Thanks.
>