You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Os Tyler <ot...@ur.com> on 2013/07/31 00:26:58 UTC

regex-urlfilter test shows negative, but URL still crawled

I have an entry in regex-urlfilter.txt designed to prevent crawling of urls that are part of our UPS search app.

# skip URLs from the UPS search app
-\?ups=
-index.php/ups\?aa

When I test the urls, it appears that regex-urlfilter should exclude them, for example:
echo "http://redacted.com/index.php/ups?aa" | /usr/local/apache-nutch/bin/nutch org/apache/nutch/net/URLFilterChecker -filterName org.apache.nutch.urlfilter.regex.RegexURLFilter

Checking URLFilter org.apache.nutch.urlfilter.regex.RegexURLFilter
-http://redacted.com/index.php/ups?aa

But when I run 'crawl', it does not skip these urls.

Thanks for any help in showing me what I'm missing here.


Re: regex-urlfilter test shows negative, but URL still crawled

Posted by Sebastian Nagel <wa...@googlemail.com>.
Hi,

in the most simple case, you don't change the configuration including URL filters
while a crawl is running. It would be waste of CPU time to filter and normalize
every step - it's enough to do this when new URLs are found (during inject
and when outlinks are extracted during parse).

If rules may change at any time you have to filter (and normalize) in other
steps as well.

Sebastian

On 08/02/2013 12:20 AM, Os Tyler wrote:
> Answering my own question here, please correct me if I'm wrong.
> 
> In order for the entries in regex-urlfilter.txt to be relevant to your crawl and indexing, you need to manually edit 'bin/crawl' and remove -noFilter from the 'nutch generate' command.
> 
> Additionally, you need to edit the portion that calls 'nutch solrindex' and add '-filter' to the solrindex call.
> 
> ________________________________
> From: Os Tyler
> Sent: Tuesday, July 30, 2013 3:26 PM
> To: user@nutch.apache.org
> Subject: regex-urlfilter test shows negative, but URL still crawled
> 
> I have an entry in regex-urlfilter.txt designed to prevent crawling of urls that are part of our UPS search app.
> 
> # skip URLs from the UPS search app
> -\?ups=
> -index.php/ups\?aa
> 
> When I test the urls, it appears that regex-urlfilter should exclude them, for example:
> echo "http://redacted.com/index.php/ups?aa" | /usr/local/apache-nutch/bin/nutch org/apache/nutch/net/URLFilterChecker -filterName org.apache.nutch.urlfilter.regex.RegexURLFilter
> 
> Checking URLFilter org.apache.nutch.urlfilter.regex.RegexURLFilter
> -http://redacted.com/index.php/ups?aa
> 
> But when I run 'crawl', it does not skip these urls.
> 
> Thanks for any help in showing me what I'm missing here.
> 
> 


RE: regex-urlfilter test shows negative, but URL still crawled

Posted by Os Tyler <ot...@ur.com>.
Answering my own question here, please correct me if I'm wrong.

In order for the entries in regex-urlfilter.txt to be relevant to your crawl and indexing, you need to manually edit 'bin/crawl' and remove -noFilter from the 'nutch generate' command.

Additionally, you need to edit the portion that calls 'nutch solrindex' and add '-filter' to the solrindex call.

________________________________
From: Os Tyler
Sent: Tuesday, July 30, 2013 3:26 PM
To: user@nutch.apache.org
Subject: regex-urlfilter test shows negative, but URL still crawled

I have an entry in regex-urlfilter.txt designed to prevent crawling of urls that are part of our UPS search app.

# skip URLs from the UPS search app
-\?ups=
-index.php/ups\?aa

When I test the urls, it appears that regex-urlfilter should exclude them, for example:
echo "http://redacted.com/index.php/ups?aa" | /usr/local/apache-nutch/bin/nutch org/apache/nutch/net/URLFilterChecker -filterName org.apache.nutch.urlfilter.regex.RegexURLFilter

Checking URLFilter org.apache.nutch.urlfilter.regex.RegexURLFilter
-http://redacted.com/index.php/ups?aa

But when I run 'crawl', it does not skip these urls.

Thanks for any help in showing me what I'm missing here.