You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@nutch.apache.org by Andrzej Bialecki <ab...@getopt.org> on 2006/05/13 01:57:09 UTC

Experiment on crawler behaviour

Hi,

I found this article pretty interesting:

    http://drunkmenworkhere.org/219

Could we come up with some codified rules, reverse-engineered from the 
bots' behavior?

-- 
Best regards,
Andrzej Bialecki     <><
 ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com