You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@nutch.apache.org by Chris Stephens <ch...@liveoakinteractive.com> on 2006/08/30 21:22:13 UTC

Follow urls with GET/Query String?

How do I get Nutch to follow URLs that contain a query string such as 
?blah=something at the end of the url?  Nutch seems to ignore these and 
I didn't find any configuration option to enable this.  Does a plugin or 
some such exist to facilitate following these types of links?

Re: Follow urls with GET/Query String?

Posted by Chris Stephens <ch...@liveoakinteractive.com>.

To answer my own question, I now realize there is an entry in 
crawl-urlfilter.txt to ignore query strings by default.  I commented 
that out and it works now.

Chris Stephens wrote:
> How do I get Nutch to follow URLs that contain a query string such as 
> ?blah=something at the end of the url?  Nutch seems to ignore these 
> and I didn't find any configuration option to enable this.  Does a 
> plugin or some such exist to facilitate following these types of links?
>
>
>
>