You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by c....@itassistance.ch on 2004/01/28 13:40:19 UTC
Re[2]: Search Bot
Shawn wrote:
> Hi, I have been trying to figure out a way to limit the massive amount
> of bandwidth that search bots (Googlebot/2.1) consume daily from my
> website. My problem is that I am running Apache::ASP and about 90% of
> the site is dynamic content, links such as product.htm?id=100. The
> dynamic content gets changed quite a bit so I dont want to use any
> caching for regular users, but it would be fine for the bots to use a
> cached copy for a month or so. The solution I came up with is manually
> modifying the headers to keeping sending back 304 HTTP_NOT_MODIFIED for
> a month before allowing new content to be served up to only search bots
> and not to regular web browsers. Can anyone tell me if there are some
> problems you for see with doing something like this? I have only tested
> this on a dev server and was just wondering if anyone else had this
> problem or any suggestions they might have.
Set up Apache to respond to a 'If-Modified-Since somedate' request with 304
response containing the whole page.
To be honest I havent't set it up, as I don't know the way to do that in
httpd.conf over all host as a default way to respind. The only way I did it is
via the request header negotiation in mod_perl.
Best Regards, Christian
--
Reporting bugs: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html