You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by "David F. Skoll" <df...@roaringpenguin.com> on 2014/06/09 20:38:50 UTC
Domain ages (was Re: SPAM from a registrar)
On Mon, 09 Jun 2014 14:24:19 -0400
Patrick Domack <pa...@patrickdk.com> wrote:
> That could be easily done. Only issue is, if you trust the
> distributed lookups to have accurate infomation.
> I suppose we could build in a trust system, where if enough
> distributed clients upload the same info, it could be trusted.
There's a company that offers a domain-age-like service:
https://www.farsightsecurity.com/Services/NOD/
Their approach is interesting (they receive a huge volume of DNS
traffic and keep track of domain lookups that are "newly seen".)
Their price for practical volumes of lookups, unfortunately, is
ridiculously expensive, which has prevented us from pursuing this
any further.
Regards,
David.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Patrick Domack <pa...@patrickdk.com>.
Quoting John Hardin <jh...@impsec.org>:
> On Mon, 9 Jun 2014, Kevin A. McGrail wrote:
>
>> On 6/9/2014 2:51 PM, John Hardin wrote:
>>> On Mon, 9 Jun 2014, Kevin A. McGrail wrote:
>>>
>>>> So there is merit in building a distributed look-up system using SA.
>>>
>>> Distributed lookup of *what*, though? Can you clarify that part of your
>>> idea? Are you referring to distributed whois queries for a domain name, to
>>> determine its age?
>>
>> Yes. Because whois data is hard to get and many whois servers
>> limit lookups, distributing and sharing the lookup load to
>> determine age of domains IMO has merit.
>
> Ah, I think there's still two different assumptions occurring in
> this discussion: newly-seen (David and Patrick) vs. newly-registered
> (me and Kevin)...
>
> Maybe we need to clarify that first.
I'm doing age based on registered, and doing whois lookups based on
newly seen (there is no point in doing a new lookup if I already did
one last year, I know the domain exists).
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 6/9/2014 3:33 PM, John Hardin wrote:
> On Mon, 9 Jun 2014, Kevin A. McGrail wrote:
>
>> On 6/9/2014 2:51 PM, John Hardin wrote:
>>> On Mon, 9 Jun 2014, Kevin A. McGrail wrote:
>>>
>>> > So there is merit in building a distributed look-up system using SA.
>>>
>>> Distributed lookup of *what*, though? Can you clarify that part of
>>> your
>>> idea? Are you referring to distributed whois queries for a domain
>>> name, to
>>> determine its age?
>>
>> Yes. Because whois data is hard to get and many whois servers limit
>> lookups, distributing and sharing the lookup load to determine age of
>> domains IMO has merit.
>
> Ah, I think there's still two different assumptions occurring in this
> discussion: newly-seen (David and Patrick) vs. newly-registered (me
> and Kevin)...
>
> Maybe we need to clarify that first.
Good clarification. The spam I envision stopping is spammers using
things like stolen credit cards or trial accounts to register domains
that they then spam and then disappear quite quickly.
So this builds a database of domain whois data (initial discussions
focused on the creation date) using distributed SA nodes to build the data.
And I chose to discuss it here because I get more ideas than I have time
and resources to implement.
Regards,
KAM
Re: Domain ages (was Re: SPAM from a registrar)
Posted by John Hardin <jh...@impsec.org>.
On Mon, 9 Jun 2014, Kevin A. McGrail wrote:
> On 6/9/2014 2:51 PM, John Hardin wrote:
>> On Mon, 9 Jun 2014, Kevin A. McGrail wrote:
>>
>> > So there is merit in building a distributed look-up system using SA.
>>
>> Distributed lookup of *what*, though? Can you clarify that part of your
>> idea? Are you referring to distributed whois queries for a domain name, to
>> determine its age?
>
> Yes. Because whois data is hard to get and many whois servers limit lookups,
> distributing and sharing the lookup load to determine age of domains IMO has
> merit.
Ah, I think there's still two different assumptions occurring in this
discussion: newly-seen (David and Patrick) vs. newly-registered (me and
Kevin)...
Maybe we need to clarify that first.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
You can't reason a person out of a position if he didn't use
reason to get there in the first place. -- Kristopher, at Marko's
-----------------------------------------------------------------------
739 days since the first successful private support mission to ISS (SpaceX)
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 6/9/2014 2:51 PM, John Hardin wrote:
> On Mon, 9 Jun 2014, Kevin A. McGrail wrote:
>
>> So there is merit in building a distributed look-up system using SA.
>
> Distributed lookup of *what*, though? Can you clarify that part of
> your idea? Are you referring to distributed whois queries for a domain
> name, to determine its age?
Yes. Because whois data is hard to get and many whois servers limit
lookups, distributing and sharing the lookup load to determine age of
domains IMO has merit.
RE: Domain ages (was Re: SPAM from a registrar)
Posted by John Hardin <jh...@impsec.org>.
On Mon, 9 Jun 2014, David Jones wrote:
> If SEM was able to detect newly registered domains more quickly then
> that would solve the problem.
Oh, agreed.
The problem is, a registrar feed of registration changes costs a lot, and
this is a free project.
That's why I suggested trying to develop relationships with registrars,
to maybe get them onboard with providing this data for free for this
purpose.
It's possible that the Apache name could provide cachet to get registars
onboard to provide rsync'able data feeds of domain names registered in the
last N days. It might be possible/better to get them to provide the data
to URIBL.org (to act as an aggregator) with a license to provide the data
free via DNS (i.e. non-bulk access) and at a nominal fee for rsync access
(which URIBL already charges for the data they collect).
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
You can't reason a person out of a position if he didn't use
reason to get there in the first place. -- Kristopher, at Marko's
-----------------------------------------------------------------------
739 days since the first successful private support mission to ISS (SpaceX)
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Axb <ax...@gmail.com>.
On 06/09/2014 09:38 PM, Kevin A. McGrail wrote:
> That is the crux of the issue, yes. So how do you identify new domains
> if the registrars/registries won't give you the data? That's the problem
> my idea solves by monitoring newly seen domains with the idea being that
> spammers are not going to buy domains and sit on them before using them.
You get the TLD zone files... and depending on your budget you get them
once/24hrs or hourly diffs (if you can affford a house in The Hamptons,
you can afford the diffs .-)
Some TLDs won't handout zone, period.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 6/9/2014 3:31 PM, David Jones wrote:
> If SEM was able to detect newly registered domains more quickly then that would solve the problem.
That is the crux of the issue, yes. So how do you identify new domains
if the registrars/registries won't give you the data? That's the problem
my idea solves by monitoring newly seen domains with the idea being that
spammers are not going to buy domains and sit on them before using them.
Regards,
KAM
RE: Domain ages (was Re: SPAM from a registrar)
Posted by David Jones <dj...@ena.com>.
If SEM was able to detect newly registered domains more quickly then that would solve the problem.
________________________________________
From: John Hardin <jh...@impsec.org>
Sent: Monday, June 09, 2014 2:24 PM
To: users@spamassassin.apache.org
Subject: Re: Domain ages (was Re: SPAM from a registrar)
On Mon, 9 Jun 2014, David F. Skoll wrote:
> On Mon, 9 Jun 2014 11:51:21 -0700 (PDT)
> John Hardin <jh...@impsec.org> wrote:
>
>>> So there is merit in building a distributed look-up system using SA.
>
>> Distributed lookup of *what*, though? Can you clarify that part of
>> your idea? Are you referring to distributed whois queries for a
>> domain name, to determine its age?
>
> The clever part is that once lots of sites begin using this in their
> SA setups, we'll very quickly build up quite an accurate database of
> newly-seen domains that's completely independent of any registrar for
> a data source.
Ah, ok, that's where I was confused. The proposal is for a distributed
network gathering newly-SEEN domain names, rather than newly-REGISTERED
domain names.
Thanks for the clarification. I was focusing on the latter.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
You can't reason a person out of a position if he didn't use
reason to get there in the first place. -- Kristopher, at Marko's
-----------------------------------------------------------------------
739 days since the first successful private support mission to ISS (SpaceX)
Re: Domain ages (was Re: SPAM from a registrar)
Posted by John Hardin <jh...@impsec.org>.
On Mon, 9 Jun 2014, David F. Skoll wrote:
> On Mon, 9 Jun 2014 11:51:21 -0700 (PDT)
> John Hardin <jh...@impsec.org> wrote:
>
>>> So there is merit in building a distributed look-up system using SA.
>
>> Distributed lookup of *what*, though? Can you clarify that part of
>> your idea? Are you referring to distributed whois queries for a
>> domain name, to determine its age?
>
> The clever part is that once lots of sites begin using this in their
> SA setups, we'll very quickly build up quite an accurate database of
> newly-seen domains that's completely independent of any registrar for
> a data source.
Ah, ok, that's where I was confused. The proposal is for a distributed
network gathering newly-SEEN domain names, rather than newly-REGISTERED
domain names.
Thanks for the clarification. I was focusing on the latter.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
You can't reason a person out of a position if he didn't use
reason to get there in the first place. -- Kristopher, at Marko's
-----------------------------------------------------------------------
739 days since the first successful private support mission to ISS (SpaceX)
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Mon, 9 Jun 2014 22:31:55 +0200
Matthias Leisi <ma...@leisi.net> wrote:
> *But*, again: which domains would be queried for such a list?
I think MAIL FROM domain.
Regards,
David.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Matthias Leisi <ma...@leisi.net>.
On Mon, Jun 9, 2014 at 9:11 PM, David F. Skoll <df...@roaringpenguin.com>
wrote:
> The clever part is that once lots of sites begin using this in their
> SA setups, we'll very quickly build up quite an accurate database of
> newly-seen domains that's completely independent of any registrar for
> a data source.
>
dnswl.org (and many other DNSxLs) already have some of that data as part of
their parsing/handling of DNS logs. For
Furthermore, you can ignore all but the first few hundred lookups before you
> enter the TXT record in the database; this will make it more expensive
> for spammers to poison the data. Or you could not enter a record in the
> database until it has been looked up from 100 different IP addresses... I
> can think of a few other countermeasures.
>
> So.... who's volunteering to do this? :)
>
We had some plans to publish such data. However since it is not really
clear what domains to look for, we did not pursue that a lot further. We
have at least a "primary domain" for each DNSWL record, but at least
historically we were not strict in what type of domain to put there (we
like to use the domain name that most closely links the IPs to the
administratively responsible owner, which is admittedly somewhat vague).
Based on the useage data we gather, we can pretty accurately extract a
"last seen" date for a particular domain (or, it's associated IPs to be
exact).
*But*, again: which domains would be queried for such a list?
-- Matthias
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 6/9/2014 3:11 PM, David F. Skoll wrote:
> On Mon, 9 Jun 2014 11:51:21 -0700 (PDT)
> John Hardin <jh...@impsec.org> wrote:
>
>>> So there is merit in building a distributed look-up system using SA.
>> Distributed lookup of *what*, though? Can you clarify that part of
>> your idea? Are you referring to distributed whois queries for a
>> domain name, to determine its age?
> Well, here's how it could be done. Imagine someone runs a DNS zone
> for "newdomain.example.net". You want to see if "example.org" is a new
> domain, so you look up a TXT record for example.org.newdomain.example.net.
>
> The DNS software that serves the zone newdomain.example.net runs
> the following pseudo-code when "example.org" is looked up:
>
> IF example.org is in my database
> THEN
> return the TXT record associated with example.org
> update the last-looked-up time for example.org
> ELSE
> generate a TXT record of the form YYYYMMDDHHMMSS corresponding to current time (UTC)
> insert it in the database
> return it
> ENDIF
>
> A background job will periodically clean out domains that haven't been
> queried in a long time.
>
> The clever part is that once lots of sites begin using this in their
> SA setups, we'll very quickly build up quite an accurate database of
> newly-seen domains that's completely independent of any registrar for
> a data source.
>
> Yes, spammers can poison it by specifically looking up a domain,
> waiting a couple of days, and then spamming. But I think most won't bother
> (witness how effective greylisting still is.)
>
> Furthermore, you can ignore all but the first few hundred lookups before you
> enter the TXT record in the database; this will make it more expensive
> for spammers to poison the data. Or you could not enter a record in the
> database until it has been looked up from 100 different IP addresses... I
> can think of a few other countermeasures.
>
> So.... who's volunteering to do this? :)
Thank you for elegantly writing my idea out though there is a bit more
to it.
So yes, effectively it's a system that can leverage registry provided
creation data and it can build it's own as domains are seen in emails
using SA installations as nodes to spread the whois load around so as to
get around whois server bans,
And I think as someone pointed out, it needs to also report the
registrar seen at the time the record is created. I'm not sure about
DNS servers but we could try logging that as well. Perhaps the whole
whois record could be stored and parsed later.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Mon, 9 Jun 2014 22:44:22 +0200
Matthias Leisi <ma...@leisi.net> wrote:
> I still have an experimental DNS server (written in Perl) lying
> around that this more-or-less what is described here. The overall
> system would need a bit more thought, though.
Attached is a hacky proof-of-concept script that stores state in
Berkeley DB. You query "something.com.da.example.com" and get back a
TXT record with the UNIX time in seconds or an A record with the UNIX
time encoded in the A record. (This time is the time in seconds since
Jan 1 1970 00:00 UTC when the domain was first queried.)
The script only handles com, net and org top-level domains. It also
only looks at the domain label just before com, net and org so that
"foo.com" and "sub.foo.com" are both treated as "foo.com"
It sets the TTL of returned records to 14 days, so if you put this behind
a caching name server like "unbound", it might even work OK under
reasonably heavy load.
Regards,
David.
=======================================================================
#!/usr/bin/perl
use strict;
use warnings;
use DB_File;
use Net::DNS::Nameserver;
my %hash;
# Replace this with the path of your DB
my $handle = tie %hash, 'DB_File', 'domain-age.db';
# Adjust settings below as needed...
my $ns = new Net::DNS::Nameserver(
LocalAddr => ['127.0.0.1'],
LocalPort => '5354',
ReplyHandler => \&handler,
Verbose => 0,
Truncate => 0,
);
$ns->main_loop();
exit(1);
sub chunk_to_addr
{
my ($chunk) = @_;
my $d = $chunk & 255; $chunk /= 256;
my $c = $chunk & 255; $chunk /= 256;
my $b = $chunk & 255; $chunk /= 256;
my $a = $chunk & 255;
return "$a.$b.$c.$d";
}
sub handler
{
my ($qname, $qclass, $qtype, $peerhost, $query, $conn) = @_;
my (@ans, @auth, @add);
# Adjust qname regex as needed
if ($qname !~ /([^.]+)\.(com|net|org)\.da\.example\.com$/i) {
return ('REFUSED', \@ans, \@auth, \@add, {aa => 1 });
}
if ($qtype ne 'TXT' && $qtype ne 'A') {
return ('NXDOMAIN', \@ans, \@auth, \@add, {aa => 1 });
}
my $chunk = lc("$1.$2");
if (!exists($hash{$chunk})) {
$hash{$chunk} = time();
# FIXME: Maybe don't sync too often? Keep track and only
# sync every 10 seconds?
$handle->sync();
}
if ($qtype eq 'TXT') {
push(@ans, new Net::DNS::RR(name => $qname,
ttl => 86400 * 14,
type => 'TXT',
txtdata => $hash{$chunk}));
} elsif ($qtype = 'A') {
push(@ans, new Net::DNS::RR(name => $qname,
ttl => 86400 * 14,
type => 'A',
address => chunk_to_addr($hash{$chunk})));
}
return ('NOERROR', \@ans, \@auth, \@add, {aa => 1 });
}
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Matthias Leisi <ma...@leisi.net>.
On Mon, Jun 9, 2014 at 9:11 PM, David F. Skoll <df...@roaringpenguin.com>
wrote:
> The DNS software that serves the zone newdomain.example.net runs
> the following pseudo-code when "example.org" is looked up:
> [..]
So.... who's volunteering to do this? :)
>
*raises hand*
I still have an experimental DNS server (written in Perl) lying around that
this more-or-less what is described here. The overall system would need a
bit more thought, though.
* Distributed over n nodes. Given that data can have pretty long TTL, it
does not need a lot of nodes, but still the distributed nature brings some
challenges.
* Definition of the granularity of data - should a "first seen" date be
returned, or an age (in days?)
* Querying whois servers is not practical at that scale.
* How would the queries be sent to the nodes? Domain-based BL-type queries?
* Would the SA project take on some operational responsibilities?
* The dnswl.org project can sponsor resources and take on some operational
aspects, but we would welcome some support.
-- Matthias
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Patrick Domack <pa...@patrickdk.com>.
Quoting "David F. Skoll" <df...@roaringpenguin.com>:
> On Mon, 09 Jun 2014 15:24:29 -0400
> Patrick Domack <pa...@patrickdk.com> wrote:
>
>> The point was, I have already done this, and have it in production.
>> I did this cause this subject keeps coming up from time to time, and
>> I was personally interested to see the results of it.
>
> Interesting. If you don't mind my asking... how much data do you
> collect? How many lookups/day?
>
> I was thinking a system that gets lookups from thousands or more SA
> installations would get a pretty good overview of new domains. A local
> installation would necessarily see a limited subset.
>
>> And I do agree with Rob McEwen on many points. And I would be
>> hisentant to outright block. But so far, and I doubt much in real
>> usage, and haven't found any yet, any issues with blocking <1day
>> outright.
>
> Or even just holding the mail for a day or so and then re-analyzing it.
Yes, I did use a greylisting type method also.
I am distributing it from 5 mailsystems right now. Doing around 10k
new domains added to the db each day.
I guess what would need to be hammered out, is, the exact info wanted.
We know age, and registrar. Though doing the registrar isn't so
simple, as the same for just ENOM changes between tld, and even within
a single tld (likely from the mergers they had).
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Mon, 09 Jun 2014 15:24:29 -0400
Patrick Domack <pa...@patrickdk.com> wrote:
> The point was, I have already done this, and have it in production.
> I did this cause this subject keeps coming up from time to time, and
> I was personally interested to see the results of it.
Interesting. If you don't mind my asking... how much data do you
collect? How many lookups/day?
I was thinking a system that gets lookups from thousands or more SA
installations would get a pretty good overview of new domains. A local
installation would necessarily see a limited subset.
> And I do agree with Rob McEwen on many points. And I would be
> hisentant to outright block. But so far, and I doubt much in real
> usage, and haven't found any yet, any issues with blocking <1day
> outright.
Or even just holding the mail for a day or so and then re-analyzing it.
Regards,
David.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Patrick Domack <pa...@patrickdk.com>.
Quoting Matthias Leisi <ma...@leisi.net>:
> On Mon, Jun 9, 2014 at 11:31 PM, Richard Doyle <li...@islandnetworks.com>
> wrote:
>
>
>> A caching whois client (jwhois, for example) can significantly reduce
>> the volume of queries.
>>
>
> You will need to query potentially hundreds or thousands of domains *per
> day* - mostly throw away domains from spammers.
>
> 1) What are the typical rate limits on public whois servers?
> 2) How to protect against attackers sending random non-existant domain
> names your way, thus ensuring you hit rate limites early?
> 3) How to parse the myriads of formats sent by whois servers?
> 4) How do you handle TLDs which do not publish registration dates, like eg
> .de? (At least they did not last time I checked.)
>
> Whois is not a feasible data source.
>
> -- Matthias
1) I dunno, but I am doing around 15k lookups a day, from a single ip,
without getting limited/blocked
2) This is hard, and I don't know, currently the postfix reject
unknown sender helps solve this for me, but won't for dns based lookups
3) This, while annoying, is solved in my code, not too hard
4) These I just don't bother doing lookups for, there is no solution,
other than to let them bypass this system, or rate them via seen
before method.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Richard Doyle <li...@islandnetworks.com>.
On 06/09/2014 02:42 PM, Matthias Leisi wrote:
>
> On Mon, Jun 9, 2014 at 11:31 PM, Richard Doyle
> <listsubs@islandnetworks.com <ma...@islandnetworks.com>> wrote:
>
>
> A caching whois client (jwhois, for example) can significantly reduce
> the volume of queries.
>
>
> You will need to query potentially hundreds or thousands of domains
> *per day* - mostly throw away domains from spammers.
>
> 1) What are the typical rate limits on public whois servers?
Apparently higher than my usage (cached names aren't rechecked)
> 2) How to protect against attackers sending random non-existant domain
> names your way, thus ensuring you hit rate limites early?
Sender verification
> 3) How to parse the myriads of formats sent by whois servers?
Don't try (see 4)
> 4) How do you handle TLDs which do not publish registration dates,
> like eg .de? (At least they did not last time I checked.)
I only check .com, .net and .org
>
> Whois is not a feasible data source.
Whois certainly has limited usefulness, but is a "feasible data source"
within those limits
>
> -- Matthias
>
-Richard
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Matthias Leisi <ma...@leisi.net>.
On Mon, Jun 9, 2014 at 11:31 PM, Richard Doyle <li...@islandnetworks.com>
wrote:
> A caching whois client (jwhois, for example) can significantly reduce
> the volume of queries.
>
You will need to query potentially hundreds or thousands of domains *per
day* - mostly throw away domains from spammers.
1) What are the typical rate limits on public whois servers?
2) How to protect against attackers sending random non-existant domain
names your way, thus ensuring you hit rate limites early?
3) How to parse the myriads of formats sent by whois servers?
4) How do you handle TLDs which do not publish registration dates, like eg
.de? (At least they did not last time I checked.)
Whois is not a feasible data source.
-- Matthias
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Richard Doyle <li...@islandnetworks.com>.
On 06/09/2014 12:29 PM, Kevin A. McGrail wrote:
> On 6/9/2014 3:24 PM, Patrick Domack wrote:
>> The point was, I have already done this, and have it in production. I
>> did this cause this subject keeps coming up from time to time, and I
>> was personally interested to see the results of it.
>>
>> And I do agree with Rob McEwen on many points. And I would be
>> hisentant to outright block. But so far, and I doubt much in real
>> usage, and haven't found any yet, any issues with blocking <1day
>> outright.
>>
>> But then the only way to be completely sure of that, will be time.
>
> My conjecture is that many people have built this for lower volume.
> But you can't be doing much volume or your IP gets blocked from whois
> servers. The twist I want to do is bring more data back centralized
> from SA installations such as whois data where it can only be done in
> a distributed manner.
>
> regards,
> KAM
>
A caching whois client (jwhois, for example) can significantly reduce
the volume of queries.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 6/9/2014 3:24 PM, Patrick Domack wrote:
> The point was, I have already done this, and have it in production. I
> did this cause this subject keeps coming up from time to time, and I
> was personally interested to see the results of it.
>
> And I do agree with Rob McEwen on many points. And I would be
> hisentant to outright block. But so far, and I doubt much in real
> usage, and haven't found any yet, any issues with blocking <1day
> outright.
>
> But then the only way to be completely sure of that, will be time.
My conjecture is that many people have built this for lower volume. But
you can't be doing much volume or your IP gets blocked from whois
servers. The twist I want to do is bring more data back centralized
from SA installations such as whois data where it can only be done in a
distributed manner.
regards,
KAM
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Patrick Domack <pa...@patrickdk.com>.
Quoting "David F. Skoll" <df...@roaringpenguin.com>:
> On Mon, 9 Jun 2014 11:51:21 -0700 (PDT)
> John Hardin <jh...@impsec.org> wrote:
>
>> > So there is merit in building a distributed look-up system using SA.
>
>> Distributed lookup of *what*, though? Can you clarify that part of
>> your idea? Are you referring to distributed whois queries for a
>> domain name, to determine its age?
>
> Well, here's how it could be done. Imagine someone runs a DNS zone
> for "newdomain.example.net". You want to see if "example.org" is a new
> domain, so you look up a TXT record for example.org.newdomain.example.net.
>
> The DNS software that serves the zone newdomain.example.net runs
> the following pseudo-code when "example.org" is looked up:
>
> IF example.org is in my database
> THEN
> return the TXT record associated with example.org
> update the last-looked-up time for example.org
> ELSE
> generate a TXT record of the form YYYYMMDDHHMMSS corresponding to
> current time (UTC)
> insert it in the database
> return it
> ENDIF
>
> A background job will periodically clean out domains that haven't been
> queried in a long time.
>
> The clever part is that once lots of sites begin using this in their
> SA setups, we'll very quickly build up quite an accurate database of
> newly-seen domains that's completely independent of any registrar for
> a data source.
>
> Yes, spammers can poison it by specifically looking up a domain,
> waiting a couple of days, and then spamming. But I think most won't bother
> (witness how effective greylisting still is.)
>
> Furthermore, you can ignore all but the first few hundred lookups before you
> enter the TXT record in the database; this will make it more expensive
> for spammers to poison the data. Or you could not enter a record in the
> database until it has been looked up from 100 different IP addresses... I
> can think of a few other countermeasures.
>
> So.... who's volunteering to do this? :)
>
> Regards,
>
> David.
The point was, I have already done this, and have it in production. I
did this cause this subject keeps coming up from time to time, and I
was personally interested to see the results of it.
And I do agree with Rob McEwen on many points. And I would be
hisentant to outright block. But so far, and I doubt much in real
usage, and haven't found any yet, any issues with blocking <1day
outright.
But then the only way to be completely sure of that, will be time.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Mon, 9 Jun 2014 11:51:21 -0700 (PDT)
John Hardin <jh...@impsec.org> wrote:
> > So there is merit in building a distributed look-up system using SA.
> Distributed lookup of *what*, though? Can you clarify that part of
> your idea? Are you referring to distributed whois queries for a
> domain name, to determine its age?
Well, here's how it could be done. Imagine someone runs a DNS zone
for "newdomain.example.net". You want to see if "example.org" is a new
domain, so you look up a TXT record for example.org.newdomain.example.net.
The DNS software that serves the zone newdomain.example.net runs
the following pseudo-code when "example.org" is looked up:
IF example.org is in my database
THEN
return the TXT record associated with example.org
update the last-looked-up time for example.org
ELSE
generate a TXT record of the form YYYYMMDDHHMMSS corresponding to current time (UTC)
insert it in the database
return it
ENDIF
A background job will periodically clean out domains that haven't been
queried in a long time.
The clever part is that once lots of sites begin using this in their
SA setups, we'll very quickly build up quite an accurate database of
newly-seen domains that's completely independent of any registrar for
a data source.
Yes, spammers can poison it by specifically looking up a domain,
waiting a couple of days, and then spamming. But I think most won't bother
(witness how effective greylisting still is.)
Furthermore, you can ignore all but the first few hundred lookups before you
enter the TXT record in the database; this will make it more expensive
for spammers to poison the data. Or you could not enter a record in the
database until it has been looked up from 100 different IP addresses... I
can think of a few other countermeasures.
So.... who's volunteering to do this? :)
Regards,
David.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by John Hardin <jh...@impsec.org>.
On Mon, 9 Jun 2014, Kevin A. McGrail wrote:
> So there is merit in building a distributed look-up system using SA.
Distributed lookup of *what*, though? Can you clarify that part of your
idea? Are you referring to distributed whois queries for a domain name, to
determine its age?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Gun Control laws aren't enacted to control guns, they are enacted
to control people: catholics (1500s), japanese peasants (1600s),
blacks (1860s), italian immigrants (1911), armenians (1911),
the irish (1920s), jews (1930s), blacks (1960s), the poor (always)
-----------------------------------------------------------------------
739 days since the first successful private support mission to ISS (SpaceX)
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 6/9/2014 4:25 PM, Matthias Leisi wrote:
>
>
> On Mon, Jun 9, 2014 at 8:43 PM, Kevin A. McGrail <KMcGrail@pccc.com
> <ma...@pccc.com>> wrote:
>
> I think the core issue is that age of domains is a good indicator
> of spam. So there is merit in building a distributed look-up
> system using SA.
>
> I have more ideas than resources, of course...
>
>
> I repeat my question: which domain? HELO, MAIL FROM, From:, ...?
I envision it for potentially any and all domains in the email.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Axb <ax...@gmail.com>.
On 06/09/2014 10:32 PM, Patrick Domack wrote:
>
> Quoting Matthias Leisi <ma...@leisi.net>:
>
>> On Mon, Jun 9, 2014 at 8:43 PM, Kevin A. McGrail <KM...@pccc.com>
>> wrote:
>>
>>
>>> I think the core issue is that age of domains is a good indicator of
>>> spam.
>>> So there is merit in building a distributed look-up system using SA.
>>>
>>> I have more ideas than resources, of course...
>>>
>>
>> I repeat my question: which domain? HELO, MAIL FROM, From:, ...?
>>
>> -- Matthias
>
> HELO hasn't matched anything in my tests.
>
> MAIL FROM has matched many, though the helo's are always a different domain
>
> From I have only started doing yesterday, and not sure exactly how I
> will track them. Likely just wait a few days, and check my ham/spam
> folders and compare what rules where hit.
LOTS of the recent .us & .me will match sender/ptr/A/HELO
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Patrick Domack <pa...@patrickdk.com>.
Quoting Matthias Leisi <ma...@leisi.net>:
> On Mon, Jun 9, 2014 at 8:43 PM, Kevin A. McGrail <KM...@pccc.com> wrote:
>
>
>> I think the core issue is that age of domains is a good indicator of spam.
>> So there is merit in building a distributed look-up system using SA.
>>
>> I have more ideas than resources, of course...
>>
>
> I repeat my question: which domain? HELO, MAIL FROM, From:, ...?
>
> -- Matthias
HELO hasn't matched anything in my tests.
MAIL FROM has matched many, though the helo's are always a different domain
From I have only started doing yesterday, and not sure exactly how I
will track them. Likely just wait a few days, and check my ham/spam
folders and compare what rules where hit.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Matthias Leisi <ma...@leisi.net>.
On Mon, Jun 9, 2014 at 8:43 PM, Kevin A. McGrail <KM...@pccc.com> wrote:
> I think the core issue is that age of domains is a good indicator of spam.
> So there is merit in building a distributed look-up system using SA.
>
> I have more ideas than resources, of course...
>
I repeat my question: which domain? HELO, MAIL FROM, From:, ...?
-- Matthias
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 6/9/2014 2:38 PM, David F. Skoll wrote:
> On Mon, 09 Jun 2014 14:24:19 -0400
> Patrick Domack <pa...@patrickdk.com> wrote:
>
>> That could be easily done. Only issue is, if you trust the
>> distributed lookups to have accurate infomation.
>> I suppose we could build in a trust system, where if enough
>> distributed clients upload the same info, it could be trusted.
> There's a company that offers a domain-age-like service:
> https://www.farsightsecurity.com/Services/NOD/
>
> Their approach is interesting (they receive a huge volume of DNS
> traffic and keep track of domain lookups that are "newly seen".)
>
> Their price for practical volumes of lookups, unfortunately, is
> ridiculously expensive, which has prevented us from pursuing this
> any further.
I think the core issue is that age of domains is a good indicator of
spam. So there is merit in building a distributed look-up system using SA.
I have more ideas than resources, of course...
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Patrick Domack <pa...@patrickdk.com>.
Quoting Axb <ax...@gmail.com>:
> On 06/10/2014 06:51 PM, Patrick Domack wrote:
>>
>> Quoting Axb <ax...@gmail.com>:
>>
>>> On 06/10/2014 05:11 PM, Patrick Domack wrote:
>>>> There are all kinds of way to use the infomation. I just don't
>>>> understand why people are so against it, cause it's not 100% foolproof.
>>>
>>> Nobody is against the idea, problem is scalability and trust.
>>> To make domain age usable, the BLs I mentioned make use of it as well
>>> as many other daata points to gain trust that a listing won' tbite the
>>> globe, as well as they can.
>>>
>>> Consider certain factors wich *can* contribute to delay in listings
>>> produce a positive hit,for example, mirror lag due to rsync, negative
>>> TTL, etc. as reasosn why you seem to see these domains being listed
>>> after you got the spams.
>>> (If your size/budget permits, datafeeds would probably help a lot)
>>>
>>> For a small site doing a few whois lookups/hour it may work, but what
>>> if suddenly an ISP/ASP doing many thousands of msgs/sec would
>>> implement this?
>>
>> I did consider those factors, and they where not the problem.
>>
>> I do rsync the data feeds locally, and feeds did not contain the lookups
>> till hours later.
>> It wasn't a negative ttl issue, as the ttl is non-existant for these
>> lookups
>
> When you come up with a couple of such cases, please post them here
> as quickly as you can so BL ops or users lurking here can check
> their logs and maybe compare results.
>
>> I fail to understand why you would be doing thousands of whois lookups
>> per second. You see that many new domain names per second?
>> Mostly it's the same domain names over and over again, and a few new
>> ones per day.
>
> You do lookups on URIS in your mailflow right? so you do it for HAM/SPAM
>
>> Domains don't expire, moved around, and updated a lot, and even if it
>> did, that isn't really much a concern. To cache this infomation for
>> atleast 3 years, would be fine, likely even longer.
>
> Check & keep track of daily changes and you'll be surprised how
> often stuff gets moved around.
>
>> Also, the point of having a central body do this, would cause the cached
>> results to be even better, and less lookups needed.
> if found...ok. if not found negative TTL applies and short TTL means
> evne more lookups.
>
>> I'm not a huge isp, but I don't seem to be any where near as tiny as you
>> suggest.
>
> I'm not assuming/suggesting anything
I'm not interested in how much stuff gets moved around, if a domain
has been registered, and been moved around, it will have a reputation.
So I don't really care if the data is 100% accurate or up to date.
I'm not sure why negative ttl would cause more whois lookups? yes it
will cause more dns lookups, but those are not an issue, expecially if
you have a local data feed available, if you do, set your negative ttl
to 5seconds.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Axb <ax...@gmail.com>.
On 06/10/2014 06:51 PM, Patrick Domack wrote:
>
> Quoting Axb <ax...@gmail.com>:
>
>> On 06/10/2014 05:11 PM, Patrick Domack wrote:
>>> There are all kinds of way to use the infomation. I just don't
>>> understand why people are so against it, cause it's not 100% foolproof.
>>
>> Nobody is against the idea, problem is scalability and trust.
>> To make domain age usable, the BLs I mentioned make use of it as well
>> as many other daata points to gain trust that a listing won' tbite the
>> globe, as well as they can.
>>
>> Consider certain factors wich *can* contribute to delay in listings
>> produce a positive hit,for example, mirror lag due to rsync, negative
>> TTL, etc. as reasosn why you seem to see these domains being listed
>> after you got the spams.
>> (If your size/budget permits, datafeeds would probably help a lot)
>>
>> For a small site doing a few whois lookups/hour it may work, but what
>> if suddenly an ISP/ASP doing many thousands of msgs/sec would
>> implement this?
>
> I did consider those factors, and they where not the problem.
>
> I do rsync the data feeds locally, and feeds did not contain the lookups
> till hours later.
> It wasn't a negative ttl issue, as the ttl is non-existant for these
> lookups
When you come up with a couple of such cases, please post them here as
quickly as you can so BL ops or users lurking here can check their logs
and maybe compare results.
> I fail to understand why you would be doing thousands of whois lookups
> per second. You see that many new domain names per second?
> Mostly it's the same domain names over and over again, and a few new
> ones per day.
You do lookups on URIS in your mailflow right? so you do it for HAM/SPAM
> Domains don't expire, moved around, and updated a lot, and even if it
> did, that isn't really much a concern. To cache this infomation for
> atleast 3 years, would be fine, likely even longer.
Check & keep track of daily changes and you'll be surprised how often
stuff gets moved around.
> Also, the point of having a central body do this, would cause the cached
> results to be even better, and less lookups needed.
if found...ok. if not found negative TTL applies and short TTL means
evne more lookups.
> I'm not a huge isp, but I don't seem to be any where near as tiny as you
> suggest.
I'm not assuming/suggesting anything
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Patrick Domack <pa...@patrickdk.com>.
Quoting Axb <ax...@gmail.com>:
> On 06/10/2014 05:11 PM, Patrick Domack wrote:
>> There are all kinds of way to use the infomation. I just don't
>> understand why people are so against it, cause it's not 100% foolproof.
>
> Nobody is against the idea, problem is scalability and trust.
> To make domain age usable, the BLs I mentioned make use of it as
> well as many other daata points to gain trust that a listing won'
> tbite the globe, as well as they can.
>
> Consider certain factors wich *can* contribute to delay in listings
> produce a positive hit,for example, mirror lag due to rsync,
> negative TTL, etc. as reasosn why you seem to see these domains
> being listed after you got the spams.
> (If your size/budget permits, datafeeds would probably help a lot)
>
> For a small site doing a few whois lookups/hour it may work, but
> what if suddenly an ISP/ASP doing many thousands of msgs/sec would
> implement this?
I did consider those factors, and they where not the problem.
I do rsync the data feeds locally, and feeds did not contain the
lookups till hours later.
It wasn't a negative ttl issue, as the ttl is non-existant for these lookups
I fail to understand why you would be doing thousands of whois lookups
per second. You see that many new domain names per second?
Mostly it's the same domain names over and over again, and a few new
ones per day.
Domains don't expire, moved around, and updated a lot, and even if it
did, that isn't really much a concern. To cache this infomation for
atleast 3 years, would be fine, likely even longer.
Also, the point of having a central body do this, would cause the
cached results to be even better, and less lookups needed.
I'm not a huge isp, but I don't seem to be any where near as tiny as
you suggest.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Axb <ax...@gmail.com>.
On 06/10/2014 05:11 PM, Patrick Domack wrote:
> There are all kinds of way to use the infomation. I just don't
> understand why people are so against it, cause it's not 100% foolproof.
Nobody is against the idea, problem is scalability and trust.
To make domain age usable, the BLs I mentioned make use of it as well as
many other daata points to gain trust that a listing won' tbite the
globe, as well as they can.
Consider certain factors wich *can* contribute to delay in listings
produce a positive hit,for example, mirror lag due to rsync, negative
TTL, etc. as reasosn why you seem to see these domains being listed
after you got the spams.
(If your size/budget permits, datafeeds would probably help a lot)
For a small site doing a few whois lookups/hour it may work, but what if
suddenly an ISP/ASP doing many thousands of msgs/sec would implement this?
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Patrick Domack <pa...@patrickdk.com>.
Quoting Rob McEwen <ro...@invaluement.com>:
> On 6/10/2014 10:21 AM, Axb wrote:
>> All URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check & track
>> domain reputation otherwise they'd be unusable.
>> Their listings are not blind - they all have their secret sauce to
>> process before listing a domain.
>
> Absolutely. As Axb and KAM and others stated, a very young domain age is
> too dangerous to outright block or score high on... but might be a good
> factor or good for combining with other rules.
>
> Also, if anyone does see spam that contain domains in the clickable
> links where that spam should have been blocked, but was not... then
> check the domain contained within the spam again the lookup found at
> http://multirbl.valli.org and/or http://mxtoolbox.com/blacklists.aspx
> (some months ago, MX Toolbox upgraded their system to check domains
> against URI/domain blacklists. In some cases, this could have been a
> game of inches where your user caught the "tip of the spear" and
> received the very first spams in a spam campaign that otherwise was
> quickly listed by the well known URI BLs. However, you may find that one
> or two good URI BLs are simply not implemented in your system--where
> that would have made all the difference! Those lookup forms will point
> you in the right direction.
>
> The same can also be true for checking sending IPs--then reviewing your
> current mix of sender's IP dnsbls (aka RBLs).
>
> Of course, don't fall into the trap of using a BL that catches much, but
> has too many FPs. But the list of URI BLs that Axb gave above are all
> extremely low-FP URI blacklists.
In my case, Yes, I am using all the above and more.
I had a user that normally never gets spam, started receiving around
20 per day, that where not marked.
I found that around 18per day of these where from a new domain. These
did appear on multirbl.valli.org lists, like invaluement, and uribl
after a day or two. I hadn't seen them hit dbl or surbl though.
This is what caused me to seriously look into if this method was
useful, just greylisting the email for a day, would cause a huge
benifit, for new domains, without causing an extreem backlash.
There are all kinds of way to use the infomation. I just don't
understand why people are so against it, cause it's not 100% foolproof.
Nothing about marking spam is 100% foolproof.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Rob McEwen <ro...@invaluement.com>.
On 6/10/2014 10:21 AM, Axb wrote:
> All URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check & track
> domain reputation otherwise they'd be unusable.
> Their listings are not blind - they all have their secret sauce to
> process before listing a domain.
Absolutely. As Axb and KAM and others stated, a very young domain age is
too dangerous to outright block or score high on... but might be a good
factor or good for combining with other rules.
Also, if anyone does see spam that contain domains in the clickable
links where that spam should have been blocked, but was not... then
check the domain contained within the spam again the lookup found at
http://multirbl.valli.org and/or http://mxtoolbox.com/blacklists.aspx
(some months ago, MX Toolbox upgraded their system to check domains
against URI/domain blacklists. In some cases, this could have been a
game of inches where your user caught the "tip of the spear" and
received the very first spams in a spam campaign that otherwise was
quickly listed by the well known URI BLs. However, you may find that one
or two good URI BLs are simply not implemented in your system--where
that would have made all the difference! Those lookup forms will point
you in the right direction.
The same can also be true for checking sending IPs--then reviewing your
current mix of sender's IP dnsbls (aka RBLs).
Of course, don't fall into the trap of using a BL that catches much, but
has too many FPs. But the list of URI BLs that Axb gave above are all
extremely low-FP URI blacklists.
--
Rob McEwen
+1 (478) 475-9032
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Axb <ax...@gmail.com>.
On 06/10/2014 04:34 PM, Patrick Domack wrote:
> Quoting Axb <ax...@gmail.com>:
>
>> On 06/10/2014 04:14 PM, Patrick Domack wrote:
>>>
>>> Quoting Axb <ax...@gmail.com>:
>>>
>>>> On 06/10/2014 12:28 PM, Patrick Domack wrote:
>>>>>
>>>>> Not saying this doesn't happen. But also, how often does someone
>>>>> register a domain, move all their users to the new domain, have the
>>>>> server all reconfigured to use this new domain, all within the first
>>>>> day?
>>>>>
>>>>> I know personally, I have always taken at least a week to do it,
>>>>> mainly
>>>>> just to make sure I didn't miss anything, and to double check
>>>>> everything
>>>>> as I go. The Last thing I do is force users to change their email
>>>>> addresses.
>>>>
>>>> domains don't have to have users on them.
>>>>
>>>> "coming up" film sites, parents setting up sweet16 sites, wedding
>>>> sites, cosmetic vendors, art festivals, etc, etc, TONS of etc, use
>>>> new domains for marketing, often seen in mail even before DNS is has
>>>> fully replicated.
>>>
>>> Yes, anything is possible.
>>>
>>> I have yet, to see any ligit email though, I'm sure I will a few times a
>>> year. I have seen email before dns/whois even is updated.
>>>
>>> But personally, one should work to establish their reputation before
>>> blasting out emails. You have to do this when moving ip addresses, and
>>> also for domains, though not as many servers track domain reputation as
>>> much as ip reputation.
>>
>> you honestly expect marketing drones to understand/care?
>>
>> All URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check & track
>> domain reputation otherwise they'd be unusable.
>>
>> Their listings are not blind - they all have their secret sauce to
>> process before listing a domain.
>
> So, we are unwilling to look into any new ideas cause there might be an
> issue? that we haven't scoped or checked into?
>
> How is progress made, when your unwilling to check and collect stats and
> figures.
>
> This was meant to be another metric that could, or might not be used. I
> personally got tired of everyone talking about it, many shooting it
> down, and NO ONE actually looking into it, and reporting real stats
> about it.
>
> Personally, I thought it was a pointless test, but it is proving useful.
> Does it single handily solve spam and has no side effects? No, but if
> you find that solution, you will be rich.
Your ideas/approach has been dealt with a decade ago - they're not new.
While it may work for you, they may not scale on a global user base.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Rob McEwen <ro...@invaluement.com>.
On 6/10/2014 10:34 AM, Patrick Domack wrote:
> So, we are unwilling to look into any new ideas cause there might be
> an issue? that we haven't scoped or checked into?
Patrick,
I don't think Axe was arguing against this idea.. I think he was arguing
against irrational exuberance by some who may be taking this idea to the
point of outright blocking (or high scoring) on it, which would generate
significant FPs. His examples were solid real world examples that DO
happen and WOULD FP if this idea were taken too far. But using
extremely-young-domain-age for low scoring or in combination with other
rules could be very helpful.
--
Rob McEwen
+1 (478) 475-9032
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Patrick Domack <pa...@patrickdk.com>.
Quoting Axb <ax...@gmail.com>:
> On 06/10/2014 04:14 PM, Patrick Domack wrote:
>>
>> Quoting Axb <ax...@gmail.com>:
>>
>>> On 06/10/2014 12:28 PM, Patrick Domack wrote:
>>>>
>>>> Not saying this doesn't happen. But also, how often does someone
>>>> register a domain, move all their users to the new domain, have the
>>>> server all reconfigured to use this new domain, all within the first
>>>> day?
>>>>
>>>> I know personally, I have always taken at least a week to do it, mainly
>>>> just to make sure I didn't miss anything, and to double check everything
>>>> as I go. The Last thing I do is force users to change their email
>>>> addresses.
>>>
>>> domains don't have to have users on them.
>>>
>>> "coming up" film sites, parents setting up sweet16 sites, wedding
>>> sites, cosmetic vendors, art festivals, etc, etc, TONS of etc, use
>>> new domains for marketing, often seen in mail even before DNS is has
>>> fully replicated.
>>
>> Yes, anything is possible.
>>
>> I have yet, to see any ligit email though, I'm sure I will a few times a
>> year. I have seen email before dns/whois even is updated.
>>
>> But personally, one should work to establish their reputation before
>> blasting out emails. You have to do this when moving ip addresses, and
>> also for domains, though not as many servers track domain reputation as
>> much as ip reputation.
>
> you honestly expect marketing drones to understand/care?
>
> All URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check &
> track domain reputation otherwise they'd be unusable.
>
> Their listings are not blind - they all have their secret sauce to
> process before listing a domain.
So, we are unwilling to look into any new ideas cause there might be
an issue? that we haven't scoped or checked into?
How is progress made, when your unwilling to check and collect stats
and figures.
This was meant to be another metric that could, or might not be used.
I personally got tired of everyone talking about it, many shooting it
down, and NO ONE actually looking into it, and reporting real stats
about it.
Personally, I thought it was a pointless test, but it is proving
useful. Does it single handily solve spam and has no side effects? No,
but if you find that solution, you will be rich.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Axb <ax...@gmail.com>.
On 06/10/2014 04:14 PM, Patrick Domack wrote:
>
> Quoting Axb <ax...@gmail.com>:
>
>> On 06/10/2014 12:28 PM, Patrick Domack wrote:
>>>
>>> Not saying this doesn't happen. But also, how often does someone
>>> register a domain, move all their users to the new domain, have the
>>> server all reconfigured to use this new domain, all within the first
>>> day?
>>>
>>> I know personally, I have always taken at least a week to do it, mainly
>>> just to make sure I didn't miss anything, and to double check everything
>>> as I go. The Last thing I do is force users to change their email
>>> addresses.
>>
>> domains don't have to have users on them.
>>
>> "coming up" film sites, parents setting up sweet16 sites, wedding
>> sites, cosmetic vendors, art festivals, etc, etc, TONS of etc, use
>> new domains for marketing, often seen in mail even before DNS is has
>> fully replicated.
>
> Yes, anything is possible.
>
> I have yet, to see any ligit email though, I'm sure I will a few times a
> year. I have seen email before dns/whois even is updated.
>
> But personally, one should work to establish their reputation before
> blasting out emails. You have to do this when moving ip addresses, and
> also for domains, though not as many servers track domain reputation as
> much as ip reputation.
you honestly expect marketing drones to understand/care?
All URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check & track
domain reputation otherwise they'd be unusable.
Their listings are not blind - they all have their secret sauce to
process before listing a domain.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Patrick Domack <pa...@patrickdk.com>.
Quoting Axb <ax...@gmail.com>:
> On 06/10/2014 12:28 PM, Patrick Domack wrote:
>>
>> Not saying this doesn't happen. But also, how often does someone
>> register a domain, move all their users to the new domain, have the
>> server all reconfigured to use this new domain, all within the first day?
>>
>> I know personally, I have always taken at least a week to do it, mainly
>> just to make sure I didn't miss anything, and to double check everything
>> as I go. The Last thing I do is force users to change their email
>> addresses.
>
> domains don't have to have users on them.
>
> "coming up" film sites, parents setting up sweet16 sites, wedding
> sites, cosmetic vendors, art festivals, etc, etc, TONS of etc, use
> new domains for marketing, often seen in mail even before DNS is has
> fully replicated.
Yes, anything is possible.
I have yet, to see any ligit email though, I'm sure I will a few times
a year. I have seen email before dns/whois even is updated.
But personally, one should work to establish their reputation before
blasting out emails. You have to do this when moving ip addresses, and
also for domains, though not as many servers track domain reputation
as much as ip reputation.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Axb <ax...@gmail.com>.
On 06/10/2014 12:28 PM, Patrick Domack wrote:
>
> Not saying this doesn't happen. But also, how often does someone
> register a domain, move all their users to the new domain, have the
> server all reconfigured to use this new domain, all within the first day?
>
> I know personally, I have always taken at least a week to do it, mainly
> just to make sure I didn't miss anything, and to double check everything
> as I go. The Last thing I do is force users to change their email
> addresses.
domains don't have to have users on them.
"coming up" film sites, parents setting up sweet16 sites, wedding sites,
cosmetic vendors, art festivals, etc, etc, TONS of etc, use new domains
for marketing, often seen in mail even before DNS is has fully replicated.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Patrick Domack <pa...@patrickdk.com>.
Quoting Lucio Chiappetti <lu...@lambrate.inaf.it>:
> On Mon, 9 Jun 2014, Rob McEwen wrote:
>
>> Domain age is a good metric to factor in. But I'm always fascinated with
>> some people's desire to block all messages with extremely new domains.
>
>> Keep in mind that many large and famous businesses... who have fairly
>> good mail sending practices... sometimes launch a new products complete
>> with links to very newly registered domains. Same is often true for ...
>
> Or for public research organizations which are often "reformed" by
> the Government, with change of name and consequential change of
> domain (even if the IP of the DNS and MX is unchanged :-))
>
> Take my case, I've been working at the same physical place since
> 1982 and the name of my institute or of the organization it belongs
> to has changed about 7 times. And it does not only occur in this
> country (Italy), I've seen (mainly dealing with mailing list
> re-subscriptions) similar changes at least in France and UK ..
Not saying this doesn't happen. But also, how often does someone
register a domain, move all their users to the new domain, have the
server all reconfigured to use this new domain, all within the first
day?
I know personally, I have always taken at least a week to do it,
mainly just to make sure I didn't miss anything, and to double check
everything as I go. The Last thing I do is force users to change their
email addresses.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Lucio Chiappetti <lu...@lambrate.inaf.it>.
On Mon, 9 Jun 2014, Rob McEwen wrote:
> Domain age is a good metric to factor in. But I'm always fascinated with
> some people's desire to block all messages with extremely new domains.
> Keep in mind that many large and famous businesses... who have fairly
> good mail sending practices... sometimes launch a new products complete
> with links to very newly registered domains. Same is often true for ...
Or for public research organizations which are often "reformed" by the
Government, with change of name and consequential change of domain (even
if the IP of the DNS and MX is unchanged :-))
Take my case, I've been working at the same physical place since 1982 and
the name of my institute or of the organization it belongs to has changed
about 7 times. And it does not only occur in this country (Italy), I've
seen (mainly dealing with mailing list re-subscriptions) similar changes
at least in France and UK ..
--
------------------------------------------------------------------------
Lucio Chiappetti - INAF/IASF - via Bassini 15 - I-20133 Milano (Italy)
For more info : http://www.iasf-milano.inaf.it/~lucio/personal.html
Re: Domain ages (was Re: SPAM from a registrar)
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 6/9/2014 3:02 PM, Rob McEwen wrote:
> Domain age is a good metric to factor in. But I'm always fascinated with
> some people's desire to block all messages with extremely new domains.
> (NOT saying that this applies to everyone who posted on this thread!)
>
> Keep in mind that many large and famous businesses... who have fairly
> good mail sending practices... sometimes launch a new products complete
> with links to very newly registered domains. Same is often true for
> advertisments for things like rock concerts, etc. Or web sites that deal
> with specific events or hot-topic political issues that appeared out of
> nowhere. Yes, some of these are UBE. But many are NOT!
>
> These example provide one of the largest source of FPs for all the major
> domain/URI blacklists. But the better domain/URI blacklists have good
> mechanisms in place to (a) PREVENT... MANY of these from ever becoming
> FPs in the first place, and (b) and where those mechanism failed, they
> have good triggers/feedback to remove & whitelist such FPs VERY QUICKLY
> if/when they do occur.
>
> In contrast, many who might go overboard by outright blocking on
> newness... and/or scoring too agressively on newness... may find
> too-high FP problems kicking their butts in the long run. And when such
> a FP starts happening, they may not have the proper telemetry to
> catch/fix it until AFTER much FP damage has happened.
>
> Personally, I think that the real problem here is that some of the most
> famous URI/domain blacklists are NOT catching everything and/or NOT
> catching everything fast enough... combined with many sys admins failing
> to make use of ALL the good and low-FP URI/domain blacklists... where
> they 'd see MUCH better results if they were using ALL of the good URI
> blacklists! ...but I'm a little biased on this point! :)
A great point. My goal is simply to build a system to identify the age
of domains and use it as YAIOS or yet another indicator of spamminess
not as a poison pill.
Re: Domain ages (was Re: SPAM from a registrar)
Posted by Rob McEwen <ro...@invaluement.com>.
Domain age is a good metric to factor in. But I'm always fascinated with
some people's desire to block all messages with extremely new domains.
(NOT saying that this applies to everyone who posted on this thread!)
Keep in mind that many large and famous businesses... who have fairly
good mail sending practices... sometimes launch a new products complete
with links to very newly registered domains. Same is often true for
advertisments for things like rock concerts, etc. Or web sites that deal
with specific events or hot-topic political issues that appeared out of
nowhere. Yes, some of these are UBE. But many are NOT!
These example provide one of the largest source of FPs for all the major
domain/URI blacklists. But the better domain/URI blacklists have good
mechanisms in place to (a) PREVENT... MANY of these from ever becoming
FPs in the first place, and (b) and where those mechanism failed, they
have good triggers/feedback to remove & whitelist such FPs VERY QUICKLY
if/when they do occur.
In contrast, many who might go overboard by outright blocking on
newness... and/or scoring too agressively on newness... may find
too-high FP problems kicking their butts in the long run. And when such
a FP starts happening, they may not have the proper telemetry to
catch/fix it until AFTER much FP damage has happened.
Personally, I think that the real problem here is that some of the most
famous URI/domain blacklists are NOT catching everything and/or NOT
catching everything fast enough... combined with many sys admins failing
to make use of ALL the good and low-FP URI/domain blacklists... where
they 'd see MUCH better results if they were using ALL of the good URI
blacklists! ...but I'm a little biased on this point! :)
--
Rob McEwen
+1 (478) 475-9032