You are viewing a plain text version of this content. The canonical link for it is here.

Posted to agent@nutch.apache.org by Fuad Efendi <fu...@efendi.ca> on 2005/10/09 01:38:34 UTC

Should not be visited.

For Webmasters:

The Robots Exclusion Protocol is a method that allows Web site
administrators to indicate to visiting robots which parts of their site
should not be visited by the robot. 

Please note, "Should Not".
http://www.robotstxt.org/wc/exclusion.html#robotstxt

It's just a useful information in Robots.Txt file, such as "do not go
there: it is a Shopping Card (Chat, Forum, etc.)".