You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by Matthias Jaekle <ja...@eventax.de> on 2005/03/18 19:29:57 UTC
Bug? Non-public urls are crawled
Hi,
For the domains www.tik24.de there is a dns entry 127.0.0.1.
I think nutch should realize that and ignore such domains, if this won't
be a problem for intranet crawling.
Matthias
Re: Bug? Non-public urls are crawled
Posted by Sami Siren <s....@sonera.inet.fi>.
If ip filtering is not done a malicious user, controlling a single
domain could feed links to crawl your internal servers (if they're
accessible from host runnning fetcher). That is why I think this issue
should be addressed somehow at some point.
--
Sami Siren
Stefan Groschupf wrote:
> Hi,
> this a curious dns entry.
> To check the ip until fetch-list generation could be an performance
> problem, to check ip until crawling would be the only chance i see.
> Anyway since this is a problem of domain registration we should think
> about just inform denic instead of writing a filter.
>
>
>
> Stefan
>
>
> Am 18.03.2005 um 20:02 schrieb Sami Siren:
>
>> Hi,
>>
>> I agree, at least in theory current behaviour might expose some
>> unwanted content if the search results were public.
>>
>> Could you please submit this to jira?
>>
>> --
>> Sami Siren
>>
>>
>> Matthias Jaekle wrote:
>>
>>> For the domains www.tik24.de there is a dns entry 127.0.0.1.
>>> I think nutch should realize that and ignore such domains, if this
>>> won't be a problem for intranet crawling.
>>
>>
>>
> ---------------------------------------------------------------
> company: http://www.media-style.com
> forum: http://www.text-mining.org
> blog: http://www.find23.net
>
>
Re: Bug? Non-public urls are crawled
Posted by Stefan Groschupf <sg...@media-style.com>.
Hi,
this a curious dns entry.
To check the ip until fetch-list generation could be an performance
problem, to check ip until crawling would be the only chance i see.
Anyway since this is a problem of domain registration we should think
about just inform denic instead of writing a filter.
Stefan
Am 18.03.2005 um 20:02 schrieb Sami Siren:
> Hi,
>
> I agree, at least in theory current behaviour might expose some
> unwanted content if the search results were public.
>
> Could you please submit this to jira?
>
> --
> Sami Siren
>
>
> Matthias Jaekle wrote:
>> For the domains www.tik24.de there is a dns entry 127.0.0.1.
>> I think nutch should realize that and ignore such domains, if this
>> won't be a problem for intranet crawling.
>
>
---------------------------------------------------------------
company: http://www.media-style.com
forum: http://www.text-mining.org
blog: http://www.find23.net
Re: Bug? Non-public urls are crawled
Posted by Sami Siren <s....@sonera.inet.fi>.
Hi,
I agree, at least in theory current behaviour might expose some
unwanted content if the search results were public.
Could you please submit this to jira?
--
Sami Siren
Matthias Jaekle wrote:
> For the domains www.tik24.de there is a dns entry 127.0.0.1.
>
> I think nutch should realize that and ignore such domains, if this won't
> be a problem for intranet crawling.