You are viewing a plain text version of this content. The canonical link for it is here.
Posted to embperl@perl.apache.org by Neil Gunton <ne...@nilspace.com> on 2002/03/13 20:08:35 UTC
Off-topic: Re: PBrowse, DSurf et al
There are some User-Agents that keep hitting my site, and they're
driving me up the wall. They seem to be spambots of some kind, since
they always come in looking for the same page and, if allowed, only
traverse links which have words like 'guestbook', 'message', 'post',
etc. They ignore robots.txt. I have long since taken all email addresses
off my websites, but I continue to get hit very frequently.
I have blocked these agents (using the BlockAgent script in the O'Reilly
mod_perl book), but they continue to hit at regular intervals. I feel I
have to log the "403 Forbidden" messages, and the sheer number of these
is amazing, over 1200 just over the last 10 days. I can't find any clues
on the internet about them, when I search all I get is a bunch of web
server statistics pages from other sites that have also obviously been
hit. The main user agents are:
PBrowse 1.4b
DSurf15a 01
PSurf15a VA
RSurf15a 41 (and other variants of .Surf)
XVBNSNBV (and other random strings of capital letters)
Sorry to bother you all, I was just wondering if anyone had a clue as to
who these people are. The requests come from a large number of IP
addresses (though some IP's are used over a period of weeks), so
blocking by IP is impractical. They always come straight in looking for
the same page, which makes me think that at some point that URL must
have gotten into some database which is being passed around. Or, is this
all one organization/person? Any ideas?
Thanks in advance for any clues...
-Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: PBrowse, DSurf et al
Posted by Neil Gunton <ne...@nilspace.com>.
Axel Beckert - ecos gmbh wrote:
> RSurf seems to be from home.com while PSurf seems to come from
> qwest.net, Optonline.net and Roadrunner.Net, according to
> http://www.clearwaterbeachcam.com/d--skinner/spiders.html and/or
> http://www.psychedelix.com/agents.html
>
> They also seem to submit (empty data) to guestbooks like at
> http://www.donotenter.com/guestbook/gbook.html.
>
> The only thing I found on PBrowse was
> http://members.aol.com/pbtips/. But this doesn't seem to be a web crawler.
Thanks again, you pretty much turned up what I did. The entries on
donotenter.com are intriguing. I wonder why the guy hasn't removed them.
Obviously these agents are not browsers. One of the worst offenders is
DSurf, for which the entries on the sites you give above just say "user
agent". The others that say "home.com" may just be computers that have
been compromised by some trojan or virus. Obviously not an "official"
browser, given the behavior. Otherwise, not a lot to go on from the Web,
as you can see. And PBrowse does seem to be something to do with an
object browser for Delphi or something similar. Not the culprit involved
here, I think.
Still searching.
Thanks much!
-Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: PBrowse, DSurf et al
Posted by Axel Beckert - ecos gmbh <be...@ecos.de>.
Hi!
On Wed, Mar 13, 2002 at 03:04:27PM -0500, Neil Gunton wrote:
> > Try to find out (using whois or nslookup), if the IP belongs to some
> > ISP. If yes, then complain to abuse@<isp>: This often helps.
>
> Many times the IP address comes back as unresolvable.
whois should also give you the owner of the IP space, the address
belongs to, so this also helps, if you get no DNS entry. Example:
43/0 abe@sycorax:pts/4 20:54 [~/quotes] > whois 134.96.7.7
[No name] (NS-RZ) NS.RZ.UNI-SB.DE 134.96.7.7
University of the Saarland (NET-UNISB-LAN) UNISB-LAN
134.96.0.0 - 134.96.255.255
> Meantime - any clues as to identity/sources of these rogue tools are
> still most welcome...
What I found out about those web clients using Google:
RSurf seems to be from home.com while PSurf seems to come from
qwest.net, Optonline.net and Roadrunner.Net, according to
http://www.clearwaterbeachcam.com/d--skinner/spiders.html and/or
http://www.psychedelix.com/agents.html
They also seem to submit (empty data) to guestbooks like at
http://www.donotenter.com/guestbook/gbook.html.
The only thing I found on PBrowse was
http://members.aol.com/pbtips/. But this doesn't seem to be a web crawler.
HTH.
Regards, Axel Beckert
--
-------------------------------------------------------------
Axel Beckert ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: beckert@ecos.de Voice: +49 6133 926530
WWW: http://www.ecos.de/ Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: PBrowse, DSurf et al
Posted by Neil Gunton <ne...@nilspace.com>.
Axel Beckert - ecos gmbh wrote:
> There is a more easier way, which doesn't need mod_perl. I would use
> something like the following:
>
> BrowserMatchNoCase "(PBrowse|[DPR]Surf15a)" is_a_bot
> <Limit>
> [...]
> Deny from env=is_a_bot
> </Limit>
Huh, I hadn't seen mod_setenvif before. I'll play with that - Thanks!
But, my main point is really not so much how to block, but rather WHAT
are the tools and/or WHO are these people... I would just like to know
what is doing this, and how it seems to come from so many different
sources...
> Try to find out (using whois or nslookup), if the IP belongs to some
> ISP. If yes, then complain to abuse@<isp>: This often helps.
Many times the IP address comes back as unresolvable. I guess a nice
solution might be a module or script that automatically resolves bad
requests and then sends an email to the admin at the ISP concerned (max
one a day), telling them about the abuse. Yet another Nice Little
Project that I don't have time to do.
Thanks again... but if anyone has any information about the tools/people
that actually spawn these requests, that would be even more useful.
Eventually, the spambots will become smarter and start using the same
User-Agent strings as Netscape and IE (dunno why they don't do that
already, to be honest), at which point we are left with behavioral
solutions (e.g. frequency of requests and other patterns), which are
much harder to detect, let alone prevent (without potentially blocking
valid users).
Meantime - any clues as to identity/sources of these rogue tools are
still most welcome...
-Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: PBrowse, DSurf et al
Posted by Axel Beckert - ecos gmbh <be...@ecos.de>.
Hi!
On Wed, Mar 13, 2002 at 02:08:35PM -0500, Neil Gunton wrote:
> There are some User-Agents that keep hitting my site, and they're
> driving me up the wall. [...] I have blocked these agents (using
> the BlockAgent script in the O'Reilly mod_perl book),
There is a more easier way, which doesn't need mod_perl. I would use
something like the following:
BrowserMatchNoCase "(PBrowse|[DPR]Surf15a)" is_a_bot
<Limit>
[...]
Deny from env=is_a_bot
</Limit>
> The requests come from a large number of IP addresses (though some
> IP's are used over a period of weeks), so blocking by IP is
> impractical.
Try to find out (using whois or nslookup), if the IP belongs to some
ISP. If yes, then complain to abuse@<isp>: This often helps.
If they're not belonging to ISPs, it sounds like they used the same
technic as used for DDoS attacks: Using root kits to spread the
clients over a big number of hosts. In this case the repsonsible
administrator will be glad, if you inform him about the compromised
systems.
That's at least my experience and solution with unfriendly crawlers
and skript kiddies. (Although I mainly had to fight search engines,
which were indexing pages, we had in robots.txt.)
Regards, Axel Beckert
--
-------------------------------------------------------------
Axel Beckert ecos electronic communication services gmbh
Internetconnect * Webserver/-design/-datenbanken * Consulting
Post: Tulpenstrasse 5 D-55276 Dienheim b. Mainz
E-Mail: beckert@ecos.de Voice: +49 6133 926530
WWW: http://www.ecos.de/ Fax: +49 6133 925152
-------------------------------------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org
Re: Off-topic: Re: PBrowse, DSurf et al
Posted by Neil Gunton <ne...@nilspace.com>.
Leeland Heins wrote:
>
> A little searching on Google shows that those user agent strings
> show up in significant numbers all over the web, in people's
> user agent logs and guest books.
>
> My guess is that they are somehow searching for guestbooks
> because those often have people's email addresses in them and
> are scanning them to build spam lists. I've seen spammers do
> that before.
I think you're absolutely right. At this point I am mostly interested in
knowing exactly how these programs are being run. Are they used
intentionally on a given computer, or is this some kind of
trojan/virus/worm that executes without the user's knowledge, and then
forwards the harvested email addresses to a server elsewhere? If so,
then we might be able to disable the thing altogether by getting the ISP
of the destination server to take it down.
Also: If I can nail down the particular virus/worm/trojan, then I can
give the sysadmin at the ISP concerned much more useful information,
e.g. "Hi, it appears that xxx.xxx.xxx.xxx has been hacked by XXXXX worm.
Please inform the user and get them to fix it, by going to XXXXX.com".
This, as I said before, could even potentially be automated in a small
script. Potentially nice little open source project.
Thanks again, and once again apologies for bringing this up in the
Embperl list.
Any other info much appreciated.
-Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: embperl-unsubscribe@perl.apache.org
For additional commands, e-mail: embperl-help@perl.apache.org