You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Matthias Jaekle <ja...@eventax.de> on 2005/03/18 19:29:57 UTC

Bug? Non-public urls are crawled

Hi,

For the domains www.tik24.de there is a dns entry 127.0.0.1.

I think nutch should realize that and ignore such domains, if this won't 
be a problem for intranet crawling.

Matthias


Re: Bug? Non-public urls are crawled

Posted by Sami Siren <s....@sonera.inet.fi>.
If ip filtering is not done a malicious user, controlling a single 
domain could feed links to crawl your internal servers (if they're 
accessible from host runnning fetcher). That is why I think this issue 
should be addressed somehow at some point.

--
  Sami Siren

Stefan Groschupf wrote:
> Hi,
> this a curious dns entry.
> To check the ip until fetch-list generation could be an performance 
> problem, to check ip until crawling would be the only chance i see.
> Anyway since this is a problem of domain registration we should think 
> about just inform denic instead of writing a filter.
> 
> 
> 
> Stefan
> 
> 
> Am 18.03.2005 um 20:02 schrieb Sami Siren:
> 
>> Hi,
>>
>> I agree,  at least in theory current behaviour might expose some 
>> unwanted content if the search results were public.
>>
>> Could you please submit this to jira?
>>
>> -- 
>>  Sami Siren
>>
>>
>> Matthias Jaekle wrote:
>>
>>> For the domains www.tik24.de there is a dns entry 127.0.0.1.
>>> I think nutch should realize that and ignore such domains, if this 
>>> won't be a problem for intranet crawling.
>>
>>
>>
> ---------------------------------------------------------------
> company:        http://www.media-style.com
> forum:        http://www.text-mining.org
> blog:            http://www.find23.net
> 
> 


Re: Bug? Non-public urls are crawled

Posted by Stefan Groschupf <sg...@media-style.com>.
Hi,
this a curious dns entry.
To check the ip until fetch-list generation could be an performance 
problem, to check ip until crawling would be the only chance i see.
Anyway since this is a problem of domain registration we should think 
about just inform denic instead of writing a filter.



Stefan


Am 18.03.2005 um 20:02 schrieb Sami Siren:

> Hi,
>
> I agree,  at least in theory current behaviour might expose some 
> unwanted content if the search results were public.
>
> Could you please submit this to jira?
>
> --
>  Sami Siren
>
>
> Matthias Jaekle wrote:
>> For the domains www.tik24.de there is a dns entry 127.0.0.1.
>> I think nutch should realize that and ignore such domains, if this 
>> won't be a problem for intranet crawling.
>
>
---------------------------------------------------------------
company:		http://www.media-style.com
forum:		http://www.text-mining.org
blog:			http://www.find23.net


Re: Bug? Non-public urls are crawled

Posted by Sami Siren <s....@sonera.inet.fi>.
Hi,

I agree,  at least in theory current behaviour might expose some 
unwanted content if the search results were public.

Could you please submit this to jira?

--
  Sami Siren


Matthias Jaekle wrote:
> For the domains www.tik24.de there is a dns entry 127.0.0.1.
> 
> I think nutch should realize that and ignore such domains, if this won't 
> be a problem for intranet crawling.