You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Patricio Galeas <pg...@yahoo.de> on 2010/04/07 14:20:56 UTC

crawling without topN

Hi all,

I ran a web-crawl using a domain filter for a specific country.
-----------regex-urlfilter.txt--------------
# URLs ending with the domain .al
+^http://(.*(\.al|\.al/.*)$)

# skip anything else
-.
----------------------------------------------

Even though I didn't set the topN parameter, all new generated segments have almost the same size.
Is it a normal behavior?

Thanks
Pato

__________________________________________________
Do You Yahoo!?
Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz gegen Massenmails. 
http://mail.yahoo.com

Re: crawling without topN

Posted by whereIstand help <he...@whereistand.com>.

Please take me off this list.

On Wed, Apr 7, 2010 at 8:20 AM, Patricio Galeas <pg...@yahoo.de> wrote:

> Hi all,
>
> I ran a web-crawl using a domain filter for a specific country.
> -----------regex-urlfilter.txt--------------
> # URLs ending with the domain .al
> +^http://(.*(\.al|\.al/.*)$)
>
> # skip anything else
> -.
> ----------------------------------------------
>
> Even though I didn't set the topN parameter, all new generated segments
> have almost the same size.
> Is it a normal behavior?
>
> Thanks
> Pato
>
> __________________________________________________
> Do You Yahoo!?
> Sie sind Spam leid? Yahoo! Mail verfügt über einen herausragenden Schutz
> gegen Massenmails.
> http://mail.yahoo.com
>