You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@tomcat.apache.org by Rahul Akolkar <ra...@gmail.com> on 2005/07/14 20:21:51 UTC

Re: [OT] robots.txt

Marked OT.

On 7/14/05, Eric J. Pastoor <ep...@vt.edu> wrote:
> The msn bot has been hammering one of my webapps for about 2 weeks now. I
> have a calendar style transaction page that is not password protected.
> Essentially this calendar goes on forever. This means that the msn bot is
> basically going on forever.  It is making my log files insanely large.  Is
> there a way to create a robots.txt file for basically one servlet instead of
> everything?

In theory, yes, by using the disallow directive appropriately. For
example, this snippet recommends all bots stay away from your CGI
scripts.

User-agent: *
Disallow: /cgi-bin/

and this one recommends that badbot stays away from the GoodServlet

User-agent: badbot
Disallow: /servlets/GoodServlet

-Rahul

---------------------------------------------------------------------
To unsubscribe, e-mail: tomcat-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: tomcat-user-help@jakarta.apache.org