You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by Stefan Groschupf <sg...@media-style.com> on 2006/07/21 01:21:38 UTC

log when blocked by robots.txt

Hi Developers,
another thing in the discussion to be more polite.
I suggest that we log a message in case an requested URL was blocked  
by a robots.txt.
Optimal would be if we only log this message in case the current used  
agent name is only blocked and it is not a general blocking of all  
agents.

Should I create a patch?

Stefan

Re: log when blocked by robots.txt

Posted by Piotr Kosiorowski <pk...@gmail.com>.

I think I would log in both situations but different message.
+1
P.

On 7/21/06, Stefan Groschupf <sg...@media-style.com> wrote:
> Hi Developers,
> another thing in the discussion to be more polite.
> I suggest that we log a message in case an requested URL was blocked
> by a robots.txt.
> Optimal would be if we only log this message in case the current used
> agent name is only blocked and it is not a general blocking of all
> agents.
>
> Should I create a patch?
>
> Stefan
>
>