Weird apache bug, or system related.

[Thierry Zoller <su...@sniff-em.com>]

Hello Group,
(My) Apache (Apache/1.3.24) starts sending 404 for files which exist.

Case robots.txt :
Some spiders get 200; come back 2 minutes later and get 404 for the same file, totaly random.


Here are some excerpts from the log :

Some of the 404 :
monster.ucr.edu - - [12/Jul/2002:10:49:42 -0400] "GET /robots.txt HTTP/1.0" 404 1430 "-" "larbin_2.6.2 (larbin2.6.2@unspecified.mail)"
203-109-168-241.aus.ihugultra.com.au - - [12/Jul/2002:11:53:28 -0400] "GET /robots.txt HTTP/1.1" 404 1489 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; MSIECrawler)"
us-135.picsearch.com - - [12/Jul/2002:13:29:59 -0400] "GET /robots.txt HTTP/1.0" 404 1423 "-" "psbot/0.1 (+http://www.picsearch.com/bot.html)"
209.249.66.34 - - [13/Jul/2002:05:20:07 -0400] "GET /robots.txt HTTP/1.1" 404 1491 "-" "Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)"
209.249.66.47 - - [13/Jul/2002:12:11:20 -0400] "GET /robots.txt HTTP/1.1" 404 1483 "-" "Mozilla/4.0 compatible ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com)"

Some of the 200 :
crawl5.googlebot.com - - [08/Jun/2002:10:53:35 -0400] "GET /robots.txt HTTP/1.0" 200 40 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
66.28.250.171 - - [08/Jun/2002:11:00:06 -0400] "GET /robots.txt HTTP/1.0" 200 40 "-" "ia_archiver"
195.246.203.13 - - [08/Jun/2002:11:01:38 -0400] "GET /robots.txt HTTP/1.0" 200 40 "-" "Mozilla/4.6 [en] (X11; I; AIX 4.3)"
host213-123-17-229.in-addr.btopenworld.com - - [08/Jun/2002:11:03:00 -0400] "GET /robots.txt HTTP/1.1" 200 40 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; BTOW V9.0; MSIECrawler)"


The file is defintely there. 
-> Actively use Mod_rewrite (NOT rewriting TXT accesese), 
and Mod_gzip (latest avaiable) (txt compression enabled).

No other modules I am aware of.


My question : 
Is anybody else experiencing these issues ? 
If so do you have Mod:gzip installed ?

Regards,
Thierry Zoller



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org