You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Lyndon Maydwell <ma...@gmail.com> on 2008/03/12 16:35:07 UTC

Re: How To Fetch for '?' URLs

> # skip URLs containing certain characters as probable queries, etc.
> -[*!@=]

http://www.giantfood.com/corporate/company_press_display.htm?press_id=380

has '=' in it.

Nutch bases the inclusion / exclusion upon the first regex that matches the url.

Re: How To Fetch for '?' URLs

Posted by Nick Tkach <nt...@peapod.com>.

Oh, duh! :)  Can't believe I missed that.  Yes, that does the trick.  I 
just had to take '=' out of the special characters list.  Thanks!

Lyndon Maydwell wrote:
>> # skip URLs containing certain characters as probable queries, etc.
>> -[*!@=]
> 
> http://www.giantfood.com/corporate/company_press_display.htm?press_id=380
> 
> has '=' in it.
> 
> Nutch bases the inclusion / exclusion upon the first regex that matches the url.