Url validator rejected url because of 2 dots

[Mustafa Sertac Turkel <se...@agmlab.com>]

hi all,

I have a seedlist file. The file includes a url something like this:

http://www.example.com/example-example..-16067h.htm

Url validator plugin reject this kind of url because of ..  .I had a look RFC 2396 and w3c standarts. There is no constraint about .. except these /../ and /.. kind of statements.

to try this I prepared a local system which is included folder hierarchy like this

-dir1 
---example1.html
---dir2
------example2.html

I set up a apache server and closed url-validator plugin. And i added this http://localhost/dir1/dir2/../example1.html in my seedlist : 

As I expected, fetched example1.html .

So http://localhost/dir1/dir2/../example1.html kind of urls handled automaticly.

I think, 

Thus http://localhost/dir1/dir2/../example1.html kind of url handle automaticly. So http://example.com/exa..mple.html kind of urls should not be reject. 

Or should? Is there any point that I missed?. What do you think about this topic.


Thank you.

Best Regards.   

Re: Url validator rejected url because of 2 dots

[Sebastian Nagel <wa...@googlemail.com>]

Hi,

> Url validator plugin reject this kind of url because of ..  .
> I had a look RFC 2396 and w3c standarts. There is no constraint
> about .. except these /../ and /.. kind of statements.

Also Unix systems accept files containing two dots "abc..xyz.txt".
urlfilter-validator should be relaxed to allow such path names.
But paths containing "/../" or "/.." in final position should
be still rejected.

Can you open a Jira to fix this?

> So http://localhost/dir1/dir2/../example1.html kind of urls handled automaticly.

Of course, the URL is valid and the server resolves the path.
But the point is: if even the most trivial URL variants are accepted,
the struggle against duplicates will be lost before it begins.
When operating Nutch, such URLs will harm!
And urlfilter-validator checks whether the configured URL
normalizers work appropriately.

Thanks,
Sebastian

On 04/04/2014 02:59 PM, Mustafa Sertac Turkel wrote:
> hi all,
> 
> I have a seedlist file. The file includes a url something like this:
> 
> http://www.example.com/example-example..-16067h.htm
> 
> Url validator plugin reject this kind of url because of ..  .I had a look RFC 2396 and w3c standarts. There is no constraint about .. except these /../ and /.. kind of statements.
> 
> to try this I prepared a local system which is included folder hierarchy like this
> 
> -dir1 
> ---example1.html
> ---dir2
> ------example2.html
> 
> I set up a apache server and closed url-validator plugin. And i added this http://localhost/dir1/dir2/../example1.html in my seedlist : 
> 
> As I expected, fetched example1.html .
> 
> So http://localhost/dir1/dir2/../example1.html kind of urls handled automaticly.
> 
> I think, 
> 
> Thus http://localhost/dir1/dir2/../example1.html kind of url handle automaticly. So http://example.com/exa..mple.html kind of urls should not be reject. 
> 
> Or should? Is there any point that I missed?. What do you think about this topic.
> 
> 
> Thank you.
> 
> Best Regards.   
>