You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by Karl Chen <qu...@NOSPAM.quarl.org> on 2006/12/21 01:56:14 UTC

domain registration age

Hi, I have the idea that a URI with a domain that was recently
registered would be a good indicator of spam.  Has this been
discussed before, and would it be hard to implement?  

Various optimizations are possible over just doing a 'whois' each
time, e.g. caching, creating a global url blocklist (with auto
expiry), etc.

For an unrelated project I've been using the following regexp to
parse registration date from whois records:

 /(?:(?:Creation|created|registration|registered) (?:date|on)):\s*(.*)/i

-- 
Karl 2006-12-20 16:51


Re: domain registration age

Posted by Matthias Leisi <ma...@leisi.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


[This is getting a bit offtopic for dev@sa - any better place to continue?]

Karl Chen wrote:
>>>>>> On 2006-12-21 15:07 PST, Matthias Leisi writes:
> 
>     Matthias> However I'm not convinced any more that domain age
>     Matthias> alone will really be useful; it may make more sense
>     Matthias> to record the history of domains used, ie to have
>     Matthias> something like "date first seen", and not [only]
>     Matthias> "date registered". The two data points combined may
>     Matthias> be highly effective.
> 
> That is a good idea.  What do you think about a global database
> mapping domain => "date first seen"?  Would the size of the
> database be a problem?

I have no good educated guess on how many domains are actually found in
e-mails. But when remembering the discussion and issues around "domain
tasting" it would seem that the number of short-lived domains registered
by spammers is in the hundreds of thousands each month. Add to this the
millions of good-faith domains in all the cc and gTLDs.

Even if such a database would regularly pruned of domains not "seen"
over (let's say) four weeks, it would still contain several hundred
thousand entries. For ongoing updates and pruning, I would assume on
average five database operations (select, insert, update, delete) per
day (many more on heavily used domains, but a "long tail" distribution
can be expected).

Then there are practical issues: the DNS setup for such a system would
need to be distributed -- but there must be a feedback from the
distributed nameservers to a more-or-less central facility, and this
must happen in near-realtime. So it becomes a lot more complex than
regular DNSWL/DNSBL operations.

Additionally, it would again be easy to flood such a scheme with bogus
requests and is open to be gamed by spammers since it would effectively
be a read/write system as opposed to DNSWL/DNSBL which are read-only in
regular usage and do not make "call-outs" in response to queries made.

An alternative would be to not run this on a global / shared scale
(which is feasible when *not* doing the whois/cache dance, see below).
Then it would be a matter of an SA plugin which keeps the history of
domains seen in a local store (file or, most likely preferrable,
database-driven); however this would reduce the accuracy and reliability
of such a scheme (risk of false positives), especially on smaller systems.


> If we only need to check for valid registration, we could do
> a DNS SOA lookup instead of whois.

ACK. There are interesting differences between nameservers announced
through whois and through walking the DNS tree, but in order to
determine the "freshness", SOA lookups should be enough.

Despite the concerns above, I still believe a "domain establishedness
test" would be a powerful addition to the anti-spam toolset. Getting it
right is non-trivial, though.

- -- Matthias

- --
http://www.dnswl.org - Protect against false positives
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFFi6PSxbHw2nyi/okRAhqjAKCiPKnlk5qOvRIbo/M9/ioQ+RguzQCgy3il
NYn3+Y7mj+Eqfv/3xOxIMao=
=fMjD
-----END PGP SIGNATURE-----

Re: domain registration age

Posted by Matthias Leisi <ma...@leisi.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Karl Chen wrote:
> Hi, I have the idea that a URI with a domain that was recently
> registered would be a good indicator of spam.  Has this been
> discussed before, and would it be hard to implement?  
> 
> Various optimizations are possible over just doing a 'whois' each
> time, e.g. caching, creating a global url blocklist (with auto
> expiry), etc.

Some time ago (woot, over a year? oh my.) I wrote an experimental "DNS
Age List" using a custom DNS server in Perl, doing whois searches
w/caching, see [1].

However I'm not convinced any more that domain age alone will really be
useful; it may make more sense to record the history of domains used, ie
to have something like "date first seen", and not [only] "date
registered". The two data points combined may be highly effective.

Further, going through whois is pretty inefficient, even when using
heavy caching. It would be easy for such a service to be DoSed, at least
to have the access to the whois servers revoked for inappropriate usage,
by just about regular traffic.

> For an unrelated project I've been using the following regexp to
> parse registration date from whois records:
> 
>  /(?:(?:Creation|created|registration|registered) (?:date|on)):\s*(.*)/i

Unfortunately this is not enough given the wide variety of whois
formats, as can be seen by the list in WhoisProxy.pm in the .tar.gz at
[1], and that list is far from complete.

I continued with that thing a bit after the blog posting and the list
got longer and longer. In a variant I wrote in Java the corresponding
properties file has 84 entries (77 if we account for those with
identical format), and even then did I encounter new variants each day...

- -- Matthias

[1]
http://matthias.leisi.net/archives/129-New-version-of-Domain-Age-DNS-Server.html
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFFixPLxbHw2nyi/okRAqnOAKCuff0JXz5pivbKDLQgeT0dYCyeHwCfaH0U
8l4kTBhvbjgNQcCuypasUxw=
=gSLw
-----END PGP SIGNATURE-----