You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Mário Gamito <ga...@gmail.com> on 2008/10/24 21:29:43 UTC
[users@httpd] Stoping bots from sucking files
Hi,
I have this site that has a directory with some files.
A few weeks ago, two web bots started sucking those files at an impressive rate.
I've put a .htaccess file like this in the directory:
AuthType Basic
AuthName files
AuthUserFile /home/foobar/public_html/files/.htpasswd
require valid-user
My question is if this is enough to stop the bots from sucking the
files or do I have to make Rewrite Rules for the bots and put them in
the .htaccess file ?
It seems odd to ask this here, because I should be able to see it for
myself, but I have the site on a hosting service and I don't have
access to the logs.
Only at the beginning of the next month.
Warm Regards,
Mário Gamito
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Stoping bots from sucking files
Posted by Joseph S D Yao <js...@tux.org>.
On Fri, Oct 24, 2008 at 08:29:43PM +0100, Mário Gamito wrote:
> Hi,
>
> I have this site that has a directory with some files.
> A few weeks ago, two web bots started sucking those files at an impressive rate.
Use a robots.txt file in your home directory.
http://www.robotstxt.org/
http://en.wikipedia.org/wiki/Robots.txt
If they ignore it, then use iptables to block them. That takes the
strain off httpd.
--
/*********************************************************************\
**
** Joe Yao jsdy@tux.org - Joseph S. D. Yao
**
\*********************************************************************/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org