You are viewing a plain text version of this content. The canonical link for it is here.
Posted to embperl@perl.apache.org by Axel Beckert - ecos gmbh <be...@ecos.de> on 2002/03/13 20:45:37 UTC
Re: PBrowse, DSurf et al
Hi!
On Wed, Mar 13, 2002 at 02:08:35PM -0500, Neil Gunton wrote:
> There are some User-Agents that keep hitting my site, and they're
> driving me up the wall. [...] I have blocked these agents (using
> the BlockAgent script in the O'Reilly mod_perl book),
There is a more easier way, which doesn't need mod_perl. I would use
something like the following:
BrowserMatchNoCase "(PBrowse|[DPR]Surf15a)" is_a_bot
<Limit>
[...]
Deny from env=is_a_bot
</Limit>
> The requests come from a large number of IP addresses (though some
> IP's are used over a period of weeks), so blocking by IP is
> impractical.
Try to find out (using whois or nslookup), if the IP belongs to some
ISP. If yes, then complain to abuse@<isp>: This often helps.
If they're not belonging to ISPs, it sounds like they used the same
technic as used for DDoS attacks: Using root kits to spread the
clients over a big number of hosts. In this case the repsonsible
administrator will be glad, if you inform him about the compromised
systems.
That's at least my experience and solution with unfriendly crawlers
and skript kiddies. (Although I mainly had to fight search engines,
which were indexing pages, we had in robots.txt.)
Regards, Axel Beckert
--
-------------------------------------------------------------
Axel Beckert ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: beckert@ecos.de Voice: +49 6133 926530
WWW: http://www.ecos.de/ Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: PBrowse, DSurf et al
Posted by Neil Gunton <ne...@nilspace.com>.
Axel Beckert - ecos gmbh wrote:
> RSurf seems to be from home.com while PSurf seems to come from
> qwest.net, Optonline.net and Roadrunner.Net, according to
> http://www.clearwaterbeachcam.com/d--skinner/spiders.html and/or
> http://www.psychedelix.com/agents.html
>
> They also seem to submit (empty data) to guestbooks like at
> http://www.donotenter.com/guestbook/gbook.html.
>
> The only thing I found on PBrowse was
> http://members.aol.com/pbtips/. But this doesn't seem to be a web crawler.
Thanks again, you pretty much turned up what I did. The entries on
donotenter.com are intriguing. I wonder why the guy hasn't removed them.
Obviously these agents are not browsers. One of the worst offenders is
DSurf, for which the entries on the sites you give above just say "user
agent". The others that say "home.com" may just be computers that have
been compromised by some trojan or virus. Obviously not an "official"
browser, given the behavior. Otherwise, not a lot to go on from the Web,
as you can see. And PBrowse does seem to be something to do with an
object browser for Delphi or something similar. Not the culprit involved
here, I think.
Still searching.
Thanks much!
-Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: PBrowse, DSurf et al
Posted by Axel Beckert - ecos gmbh <be...@ecos.de>.
Hi!
On Wed, Mar 13, 2002 at 03:04:27PM -0500, Neil Gunton wrote:
> > Try to find out (using whois or nslookup), if the IP belongs to some
> > ISP. If yes, then complain to abuse@<isp>: This often helps.
>
> Many times the IP address comes back as unresolvable.
whois should also give you the owner of the IP space, the address
belongs to, so this also helps, if you get no DNS entry. Example:
43/0 abe@sycorax:pts/4 20:54 [~/quotes] > whois 134.96.7.7
[No name] (NS-RZ) NS.RZ.UNI-SB.DE 134.96.7.7
University of the Saarland (NET-UNISB-LAN) UNISB-LAN
134.96.0.0 - 134.96.255.255
> Meantime - any clues as to identity/sources of these rogue tools are
> still most welcome...
What I found out about those web clients using Google:
RSurf seems to be from home.com while PSurf seems to come from
qwest.net, Optonline.net and Roadrunner.Net, according to
http://www.clearwaterbeachcam.com/d--skinner/spiders.html and/or
http://www.psychedelix.com/agents.html
They also seem to submit (empty data) to guestbooks like at
http://www.donotenter.com/guestbook/gbook.html.
The only thing I found on PBrowse was
http://members.aol.com/pbtips/. But this doesn't seem to be a web crawler.
HTH.
Regards, Axel Beckert
--
-------------------------------------------------------------
Axel Beckert ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: beckert@ecos.de Voice: +49 6133 926530
WWW: http://www.ecos.de/ Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: PBrowse, DSurf et al
Posted by Neil Gunton <ne...@nilspace.com>.
Axel Beckert - ecos gmbh wrote:
> There is a more easier way, which doesn't need mod_perl. I would use
> something like the following:
>
> BrowserMatchNoCase "(PBrowse|[DPR]Surf15a)" is_a_bot
> <Limit>
> [...]
> Deny from env=is_a_bot
> </Limit>
Huh, I hadn't seen mod_setenvif before. I'll play with that - Thanks!
But, my main point is really not so much how to block, but rather WHAT
are the tools and/or WHO are these people... I would just like to know
what is doing this, and how it seems to come from so many different
sources...
> Try to find out (using whois or nslookup), if the IP belongs to some
> ISP. If yes, then complain to abuse@<isp>: This often helps.
Many times the IP address comes back as unresolvable. I guess a nice
solution might be a module or script that automatically resolves bad
requests and then sends an email to the admin at the ISP concerned (max
one a day), telling them about the abuse. Yet another Nice Little
Project that I don't have time to do.
Thanks again... but if anyone has any information about the tools/people
that actually spawn these requests, that would be even more useful.
Eventually, the spambots will become smarter and start using the same
User-Agent strings as Netscape and IE (dunno why they don't do that
already, to be honest), at which point we are left with behavioral
solutions (e.g. frequency of requests and other patterns), which are
much harder to detect, let alone prevent (without potentially blocking
valid users).
Meantime - any clues as to identity/sources of these rogue tools are
still most welcome...
-Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org