You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Grant Peel <gp...@thenetnow.com> on 2008/03/30 18:15:54 UTC
[users@httpd] Logging Denied Referrers
Hi all,
As mentioned in previous emails, I am trying to deny access via SetEnvIf statements in my httpd.conf (to block smap bots / email harvesters etc.
I have two questions:
1. My server has several hundred VirtualHost directives. When the SetEnvIf statements are just placed in the main server config, will the block those referrers from the VirtualHosts as well?
2. How can I set logging to see if SetEnvIf statements are doing what I intended them to do?
-Grant
Re: [users@httpd] Logging Denied Referrers
Posted by Grant Peel <gp...@thenetnow.com>.
All,
I misspoke below, I am talking about User Agents not referrers!
----- Original Message -----
From: Grant Peel
To: users@httpd.apache.org
Sent: Sunday, March 30, 2008 12:15 PM
Subject: [users@httpd] Logging Denied Referrers
Hi all,
As mentioned in previous emails, I am trying to deny access via SetEnvIf statements in my httpd.conf (to block smap bots / email harvesters etc.
I have two questions:
1. My server has several hundred VirtualHost directives. When the SetEnvIf statements are just placed in the main server config, will the block those referrers from the VirtualHosts as well?
2. How can I set logging to see if SetEnvIf statements are doing what I intended them to do?
-Grant
Re: [users@httpd] Logging Denied Referrers
Posted by Joshua Slive <jo...@slive.ca>.
On Mon, Mar 31, 2008 at 8:21 AM, Grant Peel <gp...@thenetnow.com> wrote:
> Just for my own amusement, could they be combined like this (since I am
> realy only worried about the virtual hosts document roots only),
>
>
> <Directory "/home/*/www">
> Options Indexes Includes FollowSymLinks
> AllowOverride Options All
>
>
> Order Allow,Deny
> Allow from all
> Deny from env=block_bad_bots
> </Directory>
Yes, that would have the same effect.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging Denied Referrers
Posted by Grant Peel <gp...@thenetnow.com>.
----- Original Message -----
From: "Joshua Slive" <jo...@slive.ca>
To: <us...@httpd.apache.org>; "Grant Peel" <gp...@thenetnow.com>
Sent: Monday, March 31, 2008 7:47 AM
Subject: Re: [users@httpd] Logging Denied Referrers
> On Sun, Mar 30, 2008 at 8:32 PM, Grant Peel <gp...@thenetnow.com> wrote:
>> Joshua,
>>
>> So, cand these be combined into one container somehow?
>>
>>
>> <Directory "/">
>> Order Allow,Deny
>> Allow from all
>> Deny from env=block_bad_bots
>> </Directory>
>>
>> <Directory "/home/*/www">
>> Options Indexes Includes FollowSymLinks
>> # FollowSymLinks MultiViews
>> AllowOverride Options All
>> Order allow,deny
>> Allow from all
>>
>> </Directory>
>
> Just delete the "Order allow,deny" and "Allow from all" from the
> second section. They will negate the effect of the first section if
> they are left there.
>
> Joshua.
>
Thanks Josh, that seems to be working.
Just for my own amusement, could they be combined like this (since I am
realy only worried about the virtual hosts document roots only),
<Directory "/home/*/www">
Options Indexes Includes FollowSymLinks
AllowOverride Options All
Order Allow,Deny
Allow from all
Deny from env=block_bad_bots
</Directory>
-Grant
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging Denied Referrers
Posted by Joshua Slive <jo...@slive.ca>.
On Sun, Mar 30, 2008 at 8:32 PM, Grant Peel <gp...@thenetnow.com> wrote:
> Joshua,
>
> So, cand these be combined into one container somehow?
>
>
> <Directory "/">
> Order Allow,Deny
> Allow from all
> Deny from env=block_bad_bots
> </Directory>
>
> <Directory "/home/*/www">
> Options Indexes Includes FollowSymLinks
> # FollowSymLinks MultiViews
> AllowOverride Options All
> Order allow,deny
> Allow from all
>
> </Directory>
Just delete the "Order allow,deny" and "Allow from all" from the
second section. They will negate the effect of the first section if
they are left there.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging Denied Referrers
Posted by Grant Peel <gp...@thenetnow.com>.
Joshua,
So, cand these be combined into one container somehow?
<Directory "/">
Order Allow,Deny
Allow from all
Deny from env=block_bad_bots
</Directory>
<Directory "/home/*/www">
Options Indexes Includes FollowSymLinks
# FollowSymLinks MultiViews
AllowOverride Options All
Order allow,deny
Allow from all
</Directory>
-Grant
----- Original Message -----
From: "Joshua Slive" <jo...@slive.ca>
To: <us...@httpd.apache.org>; "Grant Peel" <gp...@thenetnow.com>
Sent: Sunday, March 30, 2008 5:43 PM
Subject: Re: [users@httpd] Logging Denied Referrers
> On Sun, Mar 30, 2008 at 3:32 PM, Grant Peel <gp...@thenetnow.com> wrote:
>> Thanks Joshua,
>>
>> So, If I am reading that docs correctly, the bit I have to deny certain
>> USer
>> Agents should go near the top of my httpd.conf?
>>
>> SetEnvIfNoCase User-Agent "^IDBot" block_bad_bots
>> SetEnvIfNoCase User-Agent "^$" block_bad_bots
>> SetEnvIfNoCase User-Agent "Java/1.4.1_04" block_bad_bots
>>
>> <Directory "/">
>> Order Allow,Deny
>> Allow from all
>> Deny from env=block_bad_bots
>> </Directory>
>
> It doesn't matter where in httpd.conf that it goes. But this will only
> work if you have NO OTHER Order/Allow/Deny statements in httpd.conf.
> Since <Directory /> is essentially the first thing evaluated
> (regardless of its location) it will get overridden by any other
> access control directives. This isn't a problem unless you have some
> generic
> <Directory /var/web>
> Order Allow,Deny
> Allow from all
> </Directory>
> sections which will turn off your anti-robot rules.
>
> Joshua.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging Denied Referrers
Posted by Joshua Slive <jo...@slive.ca>.
On Sun, Mar 30, 2008 at 3:32 PM, Grant Peel <gp...@thenetnow.com> wrote:
> Thanks Joshua,
>
> So, If I am reading that docs correctly, the bit I have to deny certain USer
> Agents should go near the top of my httpd.conf?
>
> SetEnvIfNoCase User-Agent "^IDBot" block_bad_bots
> SetEnvIfNoCase User-Agent "^$" block_bad_bots
> SetEnvIfNoCase User-Agent "Java/1.4.1_04" block_bad_bots
>
> <Directory "/">
> Order Allow,Deny
> Allow from all
> Deny from env=block_bad_bots
> </Directory>
It doesn't matter where in httpd.conf that it goes. But this will only
work if you have NO OTHER Order/Allow/Deny statements in httpd.conf.
Since <Directory /> is essentially the first thing evaluated
(regardless of its location) it will get overridden by any other
access control directives. This isn't a problem unless you have some
generic
<Directory /var/web>
Order Allow,Deny
Allow from all
</Directory>
sections which will turn off your anti-robot rules.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging Denied Referrers
Posted by Grant Peel <gp...@thenetnow.com>.
Thanks Joshua,
So, If I am reading that docs correctly, the bit I have to deny certain USer
Agents should go near the top of my httpd.conf?
SetEnvIfNoCase User-Agent "^IDBot" block_bad_bots
SetEnvIfNoCase User-Agent "^$" block_bad_bots
SetEnvIfNoCase User-Agent "Java/1.4.1_04" block_bad_bots
<Directory "/">
Order Allow,Deny
Allow from all
Deny from env=block_bad_bots
</Directory>
-Grant
----- Original Message -----
From: "Joshua Slive" <jo...@slive.ca>
To: <us...@httpd.apache.org>; "Grant Peel" <gp...@thenetnow.com>
Sent: Sunday, March 30, 2008 12:53 PM
Subject: Re: [users@httpd] Logging Denied Referrers
> On Sun, Mar 30, 2008 at 12:15 PM, Grant Peel <gp...@thenetnow.com> wrote:
>>
>>
>> Hi all,
>>
>> As mentioned in previous emails, I am trying to deny access via SetEnvIf
>> statements in my httpd.conf (to block smap bots / email harvesters etc.
>>
>> I have two questions:
>>
>> 1. My server has several hundred VirtualHost directives. When the
>> SetEnvIf
>> statements are just placed in the main server config, will the block
>> those
>> referrers from the VirtualHosts as well?
>
> Yes, if placed in a <Directory> or <Location> section that is not
> overridden inside the <VirtualHost>s. See:
> http://httpd.apache.org/docs/2.2/sections.html#mergin
>
>>
>> 2. How can I set logging to see if SetEnvIf statements are doing what I
>> intended them to do?
>
> For your access log, use a log format that reports the user agent
> (such as the combined log format). Then you can just make sure the
> response code is 403 for the appropriate user agents.
>
> Joshua.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Logging Denied Referrers
Posted by Joshua Slive <jo...@slive.ca>.
On Sun, Mar 30, 2008 at 12:15 PM, Grant Peel <gp...@thenetnow.com> wrote:
>
>
> Hi all,
>
> As mentioned in previous emails, I am trying to deny access via SetEnvIf
> statements in my httpd.conf (to block smap bots / email harvesters etc.
>
> I have two questions:
>
> 1. My server has several hundred VirtualHost directives. When the SetEnvIf
> statements are just placed in the main server config, will the block those
> referrers from the VirtualHosts as well?
Yes, if placed in a <Directory> or <Location> section that is not
overridden inside the <VirtualHost>s. See:
http://httpd.apache.org/docs/2.2/sections.html#mergin
>
> 2. How can I set logging to see if SetEnvIf statements are doing what I
> intended them to do?
For your access log, use a log format that reports the user agent
(such as the combined log format). Then you can just make sure the
response code is 403 for the appropriate user agents.
Joshua.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org