You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Ryan Thompson <ry...@sasknow.com> on 2004/09/09 23:28:07 UTC
Re: [SURBL-Discuss] Start an IP list to block?
Chris Santerre wrote to SURBL Discussion list (E-mail):
> OK, this isn't the first time we've had this discussion, but Raymond
> and I felt this should be made public again. He ran thru some tests of
> 1500+ domains and found the following data. Looks like they maybe send
> from zombies, and never their hosts. IPs are similar across the board.
>
> So is there a way to use the IP info in a good way? Could SA or SURBL
> do a quick ping of the URL and match against a URL? This would allow
> us to simply list 1 IP instead of all these domains.
>
> (I'm well aware of virtual hosts! So only the filthiest of spammers
> would be put on this IP list. Then their IP better boot them or anyone
> hosted on that box would feel the rath of SURBL.)
I talked to Raymond about this, too... and, basically, here are my
big thoughts:
We need to find the correlation of IP addresses to hostnames. See
http://whois.sc/ ; I can, with some help, duplicate what they're doing
in a way that will help us fight spam.
Then, for 219.254.32.111, we could see that there are, say, 200 sites
hosted at that IP, and, after some hand checking, identify that all of
them belong to spammers.
However, for all we know *so far*, 219.254.32.111 could be a HA cluster
of a few dozen machines, and, while there may be 200 pill spammers on
that cluster, there may be 20,000 other legit sites.
With our current data, we can't make either determination. But, using
forward zone data, we can do forward lookups, and track them in a database.
Then, do forward lookups on SURBL data to get the IPs of spammers, and
(algorithmically!) find correlations.
The programming effort to implement this would not be trivial, not to
mention processing power and bandwidth, to do the initial run. The
datasets (.com!) are huge. After that, we just have to periodically
sample for new, removed, and changed domains, at which point the
processing will be reduced.
Still, there's no way I have time or money to do this alone, given my
current commitments. I *wish* I could spend my whole day fighting spam.
I'd need a fair amount of real help. It'd be good to make happen,
though, considering we could then *proactively* list domains (or IPs)
with a high degree of confidence and little or no collateral damage.
(Because we can *measure* collateral damage if we know which other
domains are hosted on a particular IP). And there would be many many
other statistical benefits we could gain.
- Ryan
--
Ryan Thompson <ry...@sasknow.com>
SaskNow Technologies - http://www.sasknow.com
901-1st Avenue North - Saskatoon, SK - S7K 1Y4
Tel: 306-664-3600 Fax: 306-244-7037 Saskatoon
Toll-Free: 877-727-5669 (877-SASKNOW) North America
Re: [SURBL-Discuss] Start an IP list to block?
Posted by Jeff Chan <je...@surbl.org>.
On Thursday, September 9, 2004, 2:49:39 PM, System Dan Mahoney wrote:
> On Thu, 9 Sep 2004, Jeff Chan wrote:
>> On Thursday, September 9, 2004, 2:28:07 PM, Ryan Thompson wrote:
>>> However, for all we know *so far*, 219.254.32.111 could be a HA cluster
>>> of a few dozen machines, and, while there may be 200 pill spammers on
>>> that cluster, there may be 20,000 other legit sites.
>>
>>> With our current data, we can't make either determination. But, using
>>> forward zone data, we can do forward lookups, and track them in a database.
>>> Then, do forward lookups on SURBL data to get the IPs of spammers, and
>>> (algorithmically!) find correlations.
>>
>>> The programming effort to implement this would not be trivial, not to
>>> mention processing power and bandwidth, to do the initial run. The
>>> datasets (.com!) are huge. After that, we just have to periodically
>>> sample for new, removed, and changed domains, at which point the
>>> processing will be reduced.
>>
>> .com is so large and rapidly changing as to be practically
>> unknowable. That's what I mean by "can't".
>>
>> By the time you have all of .com fully cataloged, it will have
>> changed significantly.
>>
>> Really the only ones who could collectively determine how spammy
>> a particular virtual host IP is are the domain registrars working
>> together and pooling all their registration data then resolving
>> every hostname and building a database of all the resolved IPs
>> mapped back into all of their domain names.
> That's not how DNS works.
> -Dan
Exactly my point. It is not reverse DNS.
It would be a separate, extremely large database of all DNS
information and all registration information. That would be the
only way to know all the domains that use a given IP address,
unless the hosting providers would give us all the information
about their virtual hosting accounts, which seems unlikely.
Jeff C.
Re: [SURBL-Discuss] Start an IP list to block?
Posted by "Dan Mahoney, System Admin" <da...@prime.gushi.org>.
On Thu, 9 Sep 2004, Jeff Chan wrote:
> On Thursday, September 9, 2004, 2:28:07 PM, Ryan Thompson wrote:
>> However, for all we know *so far*, 219.254.32.111 could be a HA cluster
>> of a few dozen machines, and, while there may be 200 pill spammers on
>> that cluster, there may be 20,000 other legit sites.
>
>> With our current data, we can't make either determination. But, using
>> forward zone data, we can do forward lookups, and track them in a database.
>> Then, do forward lookups on SURBL data to get the IPs of spammers, and
>> (algorithmically!) find correlations.
>
>> The programming effort to implement this would not be trivial, not to
>> mention processing power and bandwidth, to do the initial run. The
>> datasets (.com!) are huge. After that, we just have to periodically
>> sample for new, removed, and changed domains, at which point the
>> processing will be reduced.
>
> .com is so large and rapidly changing as to be practically
> unknowable. That's what I mean by "can't".
>
> By the time you have all of .com fully cataloged, it will have
> changed significantly.
>
> Really the only ones who could collectively determine how spammy
> a particular virtual host IP is are the domain registrars working
> together and pooling all their registration data then resolving
> every hostname and building a database of all the resolved IPs
> mapped back into all of their domain names.
That's not how DNS works.
-Dan
>
> If you can't see all the good guy domains on a virtual hosting
> IP, then you can't see who else you would block.
>
> Jeff C.
>
--
"There were some sensible, rational, and intelligent things to say. It's
just that it was the last thing I expected to hear from the damn
kangaroo."
-Saravit, December 27, 1997
--------Dan Mahoney--------
Techie, Sysadmin, WebGeek
Gushi on efnet/undernet IRC
ICQ: 13735144 AIM: LarpGM
Site: http://www.gushi.org
---------------------------
Re: [SURBL-Discuss] Start an IP list to block?
Posted by Kai Schaetzl <ma...@conactive.com>.
Ryan Thompson wrote on Thu, 9 Sep 2004 15:56:19 -0600 (CST):
> IIRC, .com is up to about 25M domains, and it's way, way higher than the
> other gTLDs (and light years beyond ccTLDs).
It's not really light years. You will have to add at least the same amount
for all other TLDs. And how do you want to get all of these data? Not all
TLDs are available for zone transfers.
> As I mentioned, the base problem has already been solved by whois.sc,
And they also show the "base problem": they only list a fraction of what
is really on that IP. I checked it with some of ours.
> Oh, and, we can *also* use this data to safely determine domain age for
> newly registered domains. Since the most spammy domains are less than a
> week old, we'll start to have useful information for *that* within about
> a week. :-)
You can get this information at the time you need it, anyway. You don't
need to mine the rootservers for that.
> That's *exactly* what I'm suggesting, and the registrars already pool
> their data. They're called TLD zone files, and (almost) anyone can
> download them.
No, you have to pay for them.
> We *can*, Jeff. We can. That was the whole point of my message.
I think it's a massive effort and doesn't have much to do with spam
fighting. You only want to use it in the end for spam fighting. The
cost:benefit ratio of it is probably quite bad.
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
IE-Center: http://ie5.de & http://msie.winware.org
Re: [SURBL-Discuss] Start an IP list to block?
Posted by Ryan Thompson <ry...@sasknow.com>.
Jeff Chan wrote to SURBL Discussion list and Spamassassin-Talk (E-mail):
> .com is so large and rapidly changing as to be practically
> unknowable. That's what I mean by "can't".
IIRC, .com is up to about 25M domains, and it's way, way higher than the
other gTLDs (and light years beyond ccTLDs).
> By the time you have all of .com fully cataloged, it will have
> changed significantly.
25M queries isn't that hard, and it can be trivially distributed to make
for a more responsive system. Even 250M isn't out of reach.
As I mentioned, the base problem has already been solved by whois.sc,
and probably others. We just need to adapt it to be useful in fighting
spam.
Oh, and, we can *also* use this data to safely determine domain age for
newly registered domains. Since the most spammy domains are less than a
week old, we'll start to have useful information for *that* within about
a week. :-)
> Really the only ones who could collectively determine how spammy a
> particular virtual host IP is are the domain registrars working
> together and pooling all their registration data then resolving every
> hostname and building a database of all the resolved IPs mapped back
> into all of their domain names.
That's *exactly* what I'm suggesting, and the registrars already pool
their data. They're called TLD zone files, and (almost) anyone can
download them.
> If you can't see all the good guy domains on a virtual hosting
> IP, then you can't see who else you would block.
We *can*, Jeff. We can. That was the whole point of my message.
- Ryan
--
Ryan Thompson <ry...@sasknow.com>
SaskNow Technologies - http://www.sasknow.com
901-1st Avenue North - Saskatoon, SK - S7K 1Y4
Tel: 306-664-3600 Fax: 306-244-7037 Saskatoon
Toll-Free: 877-727-5669 (877-SASKNOW) North America
Re: [SURBL-Discuss] Start an IP list to block?
Posted by Jeff Chan <je...@surbl.org>.
On Thursday, September 9, 2004, 2:28:07 PM, Ryan Thompson wrote:
> However, for all we know *so far*, 219.254.32.111 could be a HA cluster
> of a few dozen machines, and, while there may be 200 pill spammers on
> that cluster, there may be 20,000 other legit sites.
> With our current data, we can't make either determination. But, using
> forward zone data, we can do forward lookups, and track them in a database.
> Then, do forward lookups on SURBL data to get the IPs of spammers, and
> (algorithmically!) find correlations.
> The programming effort to implement this would not be trivial, not to
> mention processing power and bandwidth, to do the initial run. The
> datasets (.com!) are huge. After that, we just have to periodically
> sample for new, removed, and changed domains, at which point the
> processing will be reduced.
.com is so large and rapidly changing as to be practically
unknowable. That's what I mean by "can't".
By the time you have all of .com fully cataloged, it will have
changed significantly.
Really the only ones who could collectively determine how spammy
a particular virtual host IP is are the domain registrars working
together and pooling all their registration data then resolving
every hostname and building a database of all the resolved IPs
mapped back into all of their domain names.
If you can't see all the good guy domains on a virtual hosting
IP, then you can't see who else you would block.
Jeff C.
Re: [SURBL-Discuss] Start an IP list to block?
Posted by Jeff Chan <je...@surbl.org>.
On Thursday, September 9, 2004, 3:11:44 PM, Jeff Chan wrote:
> On Thursday, September 9, 2004, 3:05:30 PM, Bill Landry wrote:
>> ----- Original Message -----
>> From: "Ryan Thompson" <ry...@sasknow.com>
>>> We need to find the correlation of IP addresses to hostnames. See
>>> http://whois.sc/ ; I can, with some help, duplicate what they're doing
>>> in a way that will help us fight spam.
>> Uh oh, whois.sc is listed in WS... :-o
>> Bill
> Sigh... WS folks care to explain?
> Jeff C.
BTW I'm not seeing it on SURBLs now....
Jeff C.
Re: [SURBL-Discuss] Start an IP list to block?
Posted by Jeff Chan <je...@surbl.org>.
On Thursday, September 9, 2004, 3:05:30 PM, Bill Landry wrote:
> ----- Original Message -----
> From: "Ryan Thompson" <ry...@sasknow.com>
>> We need to find the correlation of IP addresses to hostnames. See
>> http://whois.sc/ ; I can, with some help, duplicate what they're doing
>> in a way that will help us fight spam.
> Uh oh, whois.sc is listed in WS... :-o
> Bill
Sigh... WS folks care to explain?
Jeff C.
Re: [SURBL-Discuss] Start an IP list to block?
Posted by Bill Landry <bi...@pointshare.com>.
----- Original Message -----
From: "Ryan Thompson" <ry...@sasknow.com>
> We need to find the correlation of IP addresses to hostnames. See
> http://whois.sc/ ; I can, with some help, duplicate what they're doing
> in a way that will help us fight spam.
Uh oh, whois.sc is listed in WS... :-o
Bill