You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Jack Stone <an...@hotmail.com> on 2005/03/23 19:27:25 UTC

[users@httpd] Is it possible to redirect mass downloaders?

Hello:

For quite a while, I have been doing almost hand-to-hand combat with people 
using those mass downloaders like HTTRACK. Since the site in question is an 
online Tech Mag and meant to be read online, most sessions seen are where 
intent is to ignore the robots.text and download the entire website which is 
a huge amount of content. Of course, this hogs bandwidth and slows down 
things for all the other readers.

When alerted to a session via one of those utilities, they are stopped via a 
deny of IP from the website.

My Question:
Is there a way to set a redirect to send any requests specifically from 
HTTRACK and others to a dead end so they suck air so I don't have to do this 
manually?

Appreciate any help.

Best regards, Jack

_________________________________________________________________
Don’t just search. Find. Check out the new MSN Search! 
http://search.msn.click-url.com/go/onm00200636ave/direct/01/


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] Is it possible to redirect mass downloaders?

Posted by Jack Stone <an...@hotmail.com>.
>From: Joshua Slive <js...@gmail.com>
>Reply-To: Joshua Slive <js...@gmail.com>
>To: users@httpd.apache.org
>Subject: Re: [users@httpd] Is it possible to redirect mass downloaders?
>Date: Wed, 23 Mar 2005 13:49:45 -0500
>
>On Wed, 23 Mar 2005 12:27:25 -0600, Jack Stone <an...@hotmail.com> 
>wrote:
> > Hello:
> >
> > For quite a while, I have been doing almost hand-to-hand combat with 
>people
> > using those mass downloaders like HTTRACK. Since the site in question is 
>an
> > online Tech Mag and meant to be read online, most sessions seen are 
>where
> > intent is to ignore the robots.text and download the entire website 
>which is
> > a huge amount of content. Of course, this hogs bandwidth and slows down
> > things for all the other readers.
> >
> > When alerted to a session via one of those utilities, they are stopped 
>via a
> > deny of IP from the website.
> >
> > My Question:
> > Is there a way to set a redirect to send any requests specifically from
> > HTTRACK and others to a dead end so they suck air so I don't have to do 
>this
> > manually?
>
>RewriteEngine On
>RewriteCond %{HTTP_REFERER} HTTRACK
>RewriteRule .* /path/to/nodownloadersallowed.html
>
>Many people use something similar to block all those
>email-address-harvesting robots.
>
>Joshua.
>

Joshua, thank you so much from the tip and syntax.

You are right about several more of them, webreaper is another.

Would this work to combine more than one offender?
RewriteEngine On
RewriteCond %{HTTP_REFERER} HTTRACK WebReaper
RewriteRule .* /path/to/nodownloadersallowed.html

Again, many thanks. This will be a very large help!

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today - it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/


---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] Is it possible to redirect mass downloaders?

Posted by Joshua Slive <js...@gmail.com>.
On Wed, 23 Mar 2005 12:27:25 -0600, Jack Stone <an...@hotmail.com> wrote:
> Hello:
> 
> For quite a while, I have been doing almost hand-to-hand combat with people
> using those mass downloaders like HTTRACK. Since the site in question is an
> online Tech Mag and meant to be read online, most sessions seen are where
> intent is to ignore the robots.text and download the entire website which is
> a huge amount of content. Of course, this hogs bandwidth and slows down
> things for all the other readers.
> 
> When alerted to a session via one of those utilities, they are stopped via a
> deny of IP from the website.
> 
> My Question:
> Is there a way to set a redirect to send any requests specifically from
> HTTRACK and others to a dead end so they suck air so I don't have to do this
> manually?

RewriteEngine On
RewriteCond %{HTTP_REFERER} HTTRACK
RewriteRule .* /path/to/nodownloadersallowed.html

Many people use something similar to block all those
email-address-harvesting robots.

Joshua.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org