You are viewing a plain text version of this content. The canonical link for it is here.

Posted to agent@nutch.apache.org by ji...@jidanni.org on 2008/01/01 22:32:42 UTC

Wild Chinese robot

Didn't I block this with http://jidanni.org/robots.txt ?:
124.115.4.226 - - [01/Jan/2008:02:13:46 -0800] "GET /geo/antipodes/images/tai_par_arg.png HTTP/1.1" 200 4773 "http://image.soso.com" "Mozilla/4.0 (compatible; MSIE 6.0)"

Re: Wild Chinese robot

Posted by Ken Krugler <kk...@transpac.com>.

>Didn't I block this with http://jidanni.org/robots.txt ?:
>124.115.4.226 - - [01/Jan/2008:02:13:46 -0800] "GET 
>/geo/antipodes/images/tai_par_arg.png HTTP/1.1" 200 4773 
>"http://image.soso.com" "Mozilla/4.0 (compatible; MSIE 6.0)"

Is there some connection here with Nutch that I'm not seeing?

Thanks,

-- Ken

PS - From our experience, there are a number of China-based bots that 
don't obey robots.txt. We wind up having to block them via their IP 
address.
-- 
Ken Krugler
Krugle, Inc.
+1 530-210-6378
"If you can't find it, you can't fix it"