You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Chee Wu <ch...@gmail.com> on 2006/12/31 03:47:11 UTC

Re: how to crawl Specified type files?

just take a look at  config file like "crawl-urlfilter.txt"...
nutch can use "regular expressions" to filtrate the URLs.
On 12/31/06, fangky@gzedu.gov.cn <fa...@gzedu.gov.cn> wrote:
> hi
>
> I want to know whether nutch can be set to crawl specified type files and specified name files?
>
> for example: If I crawl a website that contains many document files , and I want nutch only crawl pdf and doc files but not html files,how to do?
>
> and another question is can I want nutch only to crawl specified name files like index.htm or so ?
>
> thanks in advance
>
>