OT: watch out for alexa.com robot

[Marc Slemko <ma...@znep.com>]

It is offtopic but somewhat relevant to a lot of people...

Just a FYI: alexa.com's robot is pathologically stupid.  It pounds on
servers as hard as it possibly can; over the past few days, it has been
pounding on one site I've been doing some stuff on at a rate of ~2 million
hits per day.  I don't know where they get away thinking that sort of
thing is acceptable, but be warned...

Obviously, they are no longer welcome at that site.

User-Agent: ia_archiver
Coming from: various addresses in 206.132.186.0/24

Re: OT: watch out for alexa.com robot

[Ian Kallen <sp...@salon.com>]

The other really silly thing about Alexa's bot is its rechecking behavior.
It just can't seem to accept somethings going away and generates repeated
404's.  I configured one my boxes that had been besieged by this to just
403 ia_archiver, haven't analyzed efficacy of this yet though.

Today, Marc Slemko <ma...@znep.com> frothed and gesticulated about OT:...:
> Just a FYI: alexa.com's robot is pathologically stupid.  It pounds on
> servers as hard as it possibly can; over the past few days, it has been
> pounding on one site I've been doing some stuff on at a rate of ~2 million
> hits per day.  I don't know where they get away thinking that sort of
> thing is acceptable, but be warned...
> 
> Obviously, they are no longer welcome at that site.
> 
> User-Agent: ia_archiver
> Coming from: various addresses in 206.132.186.0/24

--
Salon Internet 				http://www.salon.com/
  Manager, Software and Systems "Livin' La Vida Unix!"
Ian Kallen <id...@salon.com> / AIM: iankallen / Fax: (415) 354-3326