You are viewing a plain text version of this content. The canonical link for it is here.

Posted to agent@nutch.apache.org by eGrants Help Desk <ra...@state.pa.us> on 2005/10/03 16:05:20 UTC

Nutch Ignoring Robots.txt

The following has apparently been ignoring our robots.txt page:
NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)


The file can be viewed at http://egrants.ed.state.pa.us/robots.txt

Can you shed light as to why, or possible error on our end?

Thanks,
Lynn

Re: Nutch Ignoring Robots.txt

Posted by Doug Cutting <cu...@nutch.org>.

eGrants Help Desk wrote:
> The following has apparently been ignoring our robots.txt page:
> NutchCVS/0.05 (Nutch; http://www.nutch.org/docs/en/bot.html; nutch-agent@lists.sourceforge.net)

What IP address are these requests coming from?

Thanks,

Doug