You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@nutch.apache.org by Dean Elwood <de...@gmail.com> on 2005/11/06 22:37:23 UTC

Not crawling specific pages

Hi guys, first post to this list, and a new Nutch user so please be gentle 
;-)

I would like to first of all congratulate all involved in this project. I've 
been playing with Nutch all weekend and have been extremely impressed with 
it so far. I'm experimenting by indexing one of my own websites (some 30k 
pages), an index which I hope will ultimately become the main medium for 
searching the site by our users. The current search mechanism is a simple 
text type system (PostNuke) and far from ideal.

One of the issues I've come across is having "high ranking" pages which are 
actually error pages. This is largely due to site structure (really, the 
error page needs to contain an HTTP header error like a 404). Regardless of 
the "real" problem I do need to find a way around it.

The error page is ".error_page".

As this is for an internal site search engine I've used the example given 
for intranet use in the tutorial.

Here's the relevant extract from my crawl-urlfilter.txt file:-

# Site to crawl
+^http://([a-z0-9]*\.)*mysite.org/

# ignore error pages
-^http://www.mysite.org/view/.error_page

As you can see, I took a "guess" that I could simply use the minus sign as a 
means of ignoring the page that I want excluded.

This doesn't seem to work. Any guidance would be greatly appreciated.

Thanks,

Dean


Re: Not crawling specific pages

Posted by EM <em...@cpuedge.com>.
np, you are welcome

Dean Elwood wrote:

> Ah, thanks EM - so basically we need to escape the dots....... 
> something that didn't even occur to me -many thanks!
>
> Dean
>
> ----- Original Message ----- From: "EM" <em...@cpuedge.com>
> To: <nu...@lucene.apache.org>
> Sent: Sunday, November 06, 2005 9:43 PM
> Subject: Re: Not crawling specific pages
>
>
>>
>>>
>>> Here's the relevant extract from my crawl-urlfilter.txt file:-
>>>
>>> # Site to crawl
>>> +^http://([a-z0-9]*\.)*mysite.org/
>>>
>>> # ignore error pages
>>> -^http://www.mysite.org/view/.error_page
>>>
>>> As you can see, I took a "guess" that I could simply use the minus 
>>> sign as a means of ignoring the page that I want excluded.
>>>
>>> This doesn't seem to work. Any guidance would be greatly appreciated.
>>>
>>>
>> any dot in the url, has to be substituted with "\." without the quotes.
>> Just putting a dot in the expression will match any character.
>>
>> For more, google for "regex"
>>
>> Hope this helps,
>> EM
>>
>


Re: Not crawling specific pages

Posted by Dean Elwood <de...@gmail.com>.
Ah, thanks EM - so basically we need to escape the dots....... something 
that didn't even occur to me -many thanks!

Dean

----- Original Message ----- 
From: "EM" <em...@cpuedge.com>
To: <nu...@lucene.apache.org>
Sent: Sunday, November 06, 2005 9:43 PM
Subject: Re: Not crawling specific pages


>
>>
>> Here's the relevant extract from my crawl-urlfilter.txt file:-
>>
>> # Site to crawl
>> +^http://([a-z0-9]*\.)*mysite.org/
>>
>> # ignore error pages
>> -^http://www.mysite.org/view/.error_page
>>
>> As you can see, I took a "guess" that I could simply use the minus sign 
>> as a means of ignoring the page that I want excluded.
>>
>> This doesn't seem to work. Any guidance would be greatly appreciated.
>>
>>
> any dot in the url, has to be substituted with "\." without the quotes.
> Just putting a dot in the expression will match any character.
>
> For more, google for "regex"
>
> Hope this helps,
> EM
> 


Re: Not crawling specific pages

Posted by EM <em...@cpuedge.com>.
>
> Here's the relevant extract from my crawl-urlfilter.txt file:-
>
> # Site to crawl
> +^http://([a-z0-9]*\.)*mysite.org/
>
> # ignore error pages
> -^http://www.mysite.org/view/.error_page
>
> As you can see, I took a "guess" that I could simply use the minus 
> sign as a means of ignoring the page that I want excluded.
>
> This doesn't seem to work. Any guidance would be greatly appreciated.
>
>
any dot in the url, has to be substituted with "\." without the quotes.
Just putting a dot in the expression will match any character.

For more, google for "regex"

Hope this helps,
EM