Are there known problems with spaces (%20) in urls with nutch?

[Marek Bachmann <m....@uni-kassel.de>]

Hello again,

I am looking for the reason of a crawler trap on a site.

The base page should be:

http://www.uni-kassel.de/uni/studium/studienorganisation/ 
studienverlauf/workshopprogramm/

(Don't know why they had build this space in)

But nutch generates more thousands of pages in that kind:

http://www.uni-kassel.de/uni/studium/studienorganisation/ 
studienverlauf/workshopprogramm/nbjmup+qsfttfAvoj.lbttfm/typo3/ext/nbjmup+qsfttfAvoj.lbttfm/typo3/ext/nbjmup+qsft

This iteration grows longer and longer until the Url is too long.

I inspected the source code of the page but can't find any suspicious 
links, but some href with javascript inside.

First I thought this is just a bug in the cms and not my problem, but I 
noticed that nutch hasn't this problem with other cms sites where there 
is no space in the uri.

My only actual question to you is: Are there known problems with spaces 
in URI?

BTW: Is there an effective way to handle crawler traps?

Thank you very much

Re: Are there known problems with spaces (%20) in urls with nutch?

[Markus Jelsma <ma...@openindex.io>]

On Monday 17 October 2011 16:04:54 Marek Bachmann wrote:
> Hello again,
> 
> I am looking for the reason of a crawler trap on a site.
> 
> The base page should be:
> 
> http://www.uni-kassel.de/uni/studium/studienorganisation/
> studienverlauf/workshopprogramm/
> 
> (Don't know why they had build this space in)
> 
> But nutch generates more thousands of pages in that kind:
> 
> http://www.uni-kassel.de/uni/studium/studienorganisation/
> studienverlauf/workshopprogramm/nbjmup+qsfttfAvoj.lbttfm/typo3/ext/nbjmup+q
> sfttfAvoj.lbttfm/typo3/ext/nbjmup+qsft
> 
> This iteration grows longer and longer until the Url is too long.

Most likely an issue with relative URL's without base href.

> 
> I inspected the source code of the page but can't find any suspicious
> links, but some href with javascript inside.
> 
> First I thought this is just a bug in the cms and not my problem, but I
> noticed that nutch hasn't this problem with other cms sites where there
> is no space in the uri.
> 
> My only actual question to you is: Are there known problems with spaces
> in URI?

I've not seen clear issues myself yet. It usually just works. There's a issue 
however to encode it properly.

You can use regex filters to get rid of these bad url's e.g.
-([a-zA-Z-_.0-9]+/[a-zA-Z-_.0-9]+)/(.*)\1

> 
> BTW: Is there an effective way to handle crawler traps?

No. Using LinkRank may not work either because low scoring url's can still be 
generated when having very large fetch lists.


> 
> Thank you very much

-- 
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350