You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Dean Del Ponte <de...@gmail.com> on 2012/01/18 20:06:39 UTC
How to exclude a specific URL from crawling
I'm new to Nutch. Currently using version 1.4.
I have some URLs which I would like excluded from the crawl.
What's the best way to do this?
Thanks!
Dean
Re: How to exclude a specific URL from crawling
Posted by Lewis John Mcgibbney <le...@gmail.com>.
Well this might be the file that you edit if you are using the
urlfilter-automaton plugin for urlfiltering.
Markus was indicating that you may wish to begin by looking at
urlfilter-regex and subsequently regex-urlfilter.txt, I would go as far to
say that this is the most commonly implemented urlfiler for most crawling
that people seem to be doing with Nutch.
You will get some great functionality out of urlfilter-regex by learning a
bit about regular expression matching.
hth
On Wed, Jan 18, 2012 at 9:07 PM, Dean Del Ponte <de...@gmail.com>wrote:
> Thanks! So, just to confirm, the file I will modify
> is automaton-urlfilter.txt?
>
> - Dean
>
> On Wed, Jan 18, 2012 at 2:51 PM, Markus Jelsma
> <ma...@openindex.io>wrote:
>
> > Please check the URL filters that Nutch provides. You can use the regex
> > filter
> > to filter out specific patterns that match your urls.
> >
> > > I'm new to Nutch. Currently using version 1.4.
> > >
> > > I have some URLs which I would like excluded from the crawl.
> > >
> > > What's the best way to do this?
> > >
> > > Thanks!
> > >
> > > Dean
> >
>
--
*Lewis*
Re: How to exclude a specific URL from crawling
Posted by Dean Del Ponte <de...@gmail.com>.
Thanks! So, just to confirm, the file I will modify
is automaton-urlfilter.txt?
- Dean
On Wed, Jan 18, 2012 at 2:51 PM, Markus Jelsma
<ma...@openindex.io>wrote:
> Please check the URL filters that Nutch provides. You can use the regex
> filter
> to filter out specific patterns that match your urls.
>
> > I'm new to Nutch. Currently using version 1.4.
> >
> > I have some URLs which I would like excluded from the crawl.
> >
> > What's the best way to do this?
> >
> > Thanks!
> >
> > Dean
>
Re: How to exclude a specific URL from crawling
Posted by Markus Jelsma <ma...@openindex.io>.
Please check the URL filters that Nutch provides. You can use the regex filter
to filter out specific patterns that match your urls.
> I'm new to Nutch. Currently using version 1.4.
>
> I have some URLs which I would like excluded from the crawl.
>
> What's the best way to do this?
>
> Thanks!
>
> Dean