Infinite Loop for parent directory urls?

[qu...@webmail.co.za]

Hi Guys

Just noticed something (nutch 0.6) - when extracting urls
from pages referencing the parent directory eg:
The page is http://www.xyz.com/corporate/aboutus.asp and
the link on that page is <a href="../more.asp">Example</a>
I get tons of files being referenced that include other
pages for example http://www.xyz.com/corporate/../more.asp
- now this should theoretically be a 404 not found error
however Ive found 1 or 2 sites that have a custom 404 page
which nutch doesnt pick up as page not found - us there
anyway that I can set somewhere in nutch's config that if
certain words in the title or on the page exist (eg. ERROR
404 or PAGE NOT FOUND or ERROR) that nutch will not index?

I hope you understand the problem?

Thanks!
_____________________________________________________________________
For super low premiums, click here http://www.dialdirect.co.za/quote

Re: Infinite Loop for parent directory urls?

[Doug Cutting <cu...@nutch.org>]

quovadis@webmail.co.za wrote:
> Would anyone be able to help me with this or provide a
> sample of how to do this kind of thing?

> On Mon, 18 Apr 2005 10:38:11 -0700
>  Doug Cutting <cu...@nutch.org> wrote:
>>You could implement a html filter plugin that looks for
>>these strings in the title and, when they're found,
>>throws a ParseException to abort the page.

It would look something like the following (untested) code:

public class TitleRegexFilter implements HtmlParseFilter {
   private Perl5Pattern pattern;
   private Perl5Matcher matcher;

   public TitleRegexFilter() {
     String regex = NutchConf.get().get("html.exclude.title.regex", "");
     this.pattern = new Perl5Compiler().compile(regex);
     this.matcher = new Perl5Matcher();
   }

   public synchronized Parse filter(Content content, Parse parse,
                                    DocumentFragment doc)
     throws ParseException {
       String title = parse.getData().getTitle();
       if (matcher.contains(pattern, title)) {
         throw new ParseException("Title rejected:" + title);
       }
     }
   }
}

You'd need to put this under src/plugin/parse-title-regex-filter, with a 
build.xml and a plugin.xml, add a value for html.exclude.title.regex in 
the config file, etc.  Does this make sense?

Doug

Re: Infinite Loop for parent directory urls?

[qu...@webmail.co.za]

Would anyone be able to help me with this or provide a
sample of how to do this kind of thing?

Thanks

On Mon, 18 Apr 2005 10:38:11 -0700
 Doug Cutting <cu...@nutch.org> wrote:
> quovadis@webmail.co.za wrote:
> > I get tons of files being referenced that include other
> > pages for example
> http://www.xyz.com/corporate/../more.asp
> > - now this should theoretically be a 404 not found
> error
> > however Ive found 1 or 2 sites that have a custom 404
> page
> > which nutch doesnt pick up as page not found - us there
> > anyway that I can set somewhere in nutch's config that
> if
> > certain words in the title or on the page exist (eg.
> ERROR
> > 404 or PAGE NOT FOUND or ERROR) that nutch will not
> index?
> 
> You could implement a html filter plugin that looks for
> these strings in the title and, when they're found,
> throws a ParseException to abort the page.
> 
> Doug

_____________________________________________________________________
For super low premiums, click here http://www.dialdirect.co.za/quote

Re: Infinite Loop for parent directory urls?

[Doug Cutting <cu...@nutch.org>]

quovadis@webmail.co.za wrote:
> I get tons of files being referenced that include other
> pages for example http://www.xyz.com/corporate/../more.asp
> - now this should theoretically be a 404 not found error
> however Ive found 1 or 2 sites that have a custom 404 page
> which nutch doesnt pick up as page not found - us there
> anyway that I can set somewhere in nutch's config that if
> certain words in the title or on the page exist (eg. ERROR
> 404 or PAGE NOT FOUND or ERROR) that nutch will not index?

You could implement a html filter plugin that looks for these strings in 
the title and, when they're found, throws a ParseException to abort the 
page.

Doug