Faster rDNS lookups

[Rob McEwen <ro...@powerviewsystems.com>]

I've got an idea for faster rDNS lookups.

Before I present the solution. Here is the problem...

...basically, rDNS checks are "expensive". They sometimes take a few seconds
when done in real time and depend on the timeliness and of other people's
DNS servers. This 1-5 (or more) seconds may not seem like a big deal... but
if this can be more instantaneous, it increases the volume of messages a
server-side spam filter can handle. (every little bit counts)

Anyone doubt me? Well... my point is valid for the same reason that it is
typically MUCH faster to check again a DNSBL (like SURBL) than it is to
convert domains found within the e-mail to IP addresses and then check these
again SpamHaus's RBL. Both of these are good techniques... but DNSBL
checking where each domain doesn't have to be resolved to an IP address is
much faster. For the very same reasons... again... rDNS checks are a bit
"expensive".

But consider that a given IP's rDNS doesn't change that much (on average).
Therefore, it would be a huge time and resource saver to specially program a
DNS server to receive rDNS queries in RBL-like format, to do the rDNS lookup
on the IP address passed in the RBL-like query, and then to return results
much like an RBL would... "Null" for having an rDNS, and "127.0.0.x" if the
IP does NOT have rDNS. This way, the mail server is "tricked" into thinking
that this is just another RBL.

You might ask, "where is the time savings if the DNS server has to fetch the
rDNS all the same?"

Here is the clincher... because, as I mentioned, rDNS settings don't change
often, therefore, PERHAPS.... the DNS server could be configured or
programmed to cache BOTH positive and negative results for 24 hours (or
whatever the config file requests... maybe even DAYS). This way, although
the 1st rDNS query would still be expensive, subsequent queries would be
lightening fast. Also, since rDNS should (ideally) never be enough of a
factor by itself to mark a message as spam, then, therefore, a rare mishap
on someone's legit mail server wouldn't block mail if all else is well. (And
those few who have no problem blocking soly on rDNS... they'd not mind such
extremely rare FPs anyways) Therefore, the negative caching would have
virtually no effect on temporary mistakes made by other mail admins... and
this is in contrast to the damage that might be done if a regular DNSBL's
records were cached for too long and a FP comes along that was quickly and
already "fixed" but then still "stuck" as blacklisted on a server due to
such caching.

Also, before anyone mentions TTL values providing caching anyways...
consider that you typically get not long or zero caching of negative
results.

This feature I describe would be VERY beneficial because MANY spam filters
which have to option to do RBL lookups could benefit. They could just remove
their regular rDNS feature and then do the lookup I described and use the
RBL-based return in the same way that they previously used the rDNS for.

Does anyone know of a publicly available RBL that already does this? Anyone
interested in creating such a service? (I don't think I have the expertise
to do this.)

Thanks for your help!!

Rob McEwen
PowerView Systems

Re: Faster rDNS lookups

[Cami <ca...@mweb.co.za>]

Rob McEwen wrote:
> Please understand, I'm only proposing this as an alternative idea for
> checking to see if a sending server's IP address has proper rDNS... NOT any
> other type of DNS lookups.
> 
> Also, I don't have stats on this, but I know that most mail is spam and I
> know that MUCH of this spam has no rDNS properly configured.

If 9% of incoming mail is spam (at least it is on my mail systems), how
does that help? (It used to be >80% but Postfix+policyd has reduced it
to barely anything). Lets use an extreme case and say that 100% of that
9% does not have reverse dns records and are spam mails. What kind of
saving is there?

> Furthermore, when rDNS is not properly configured and a subsequent lookup is
> done on the same IP, I'm pretty sure that the negative (no found) lookup
> isn't cached at all or at least not for long.

Thats a good thing.

> In my testing, that last lookup DOES find something... indicating that DNS
> servers don't generally cache previously "not found" lookups... at least not
> for long.

Again, thats a good thing.

> rDNS on a spammer's IP which doesn't have rDNS configured involves a similar
> situation. Therefore, I don't think DNS servers cache the negative results
> of rDNS checks either at all or for very long. But, ironically, these types
> of checks are more expensive than most lookups. I've found that DNS server
> often take longer to return a "not found" result when the IP address is from
> some 3rd party network in Korea... in contrast to an "is-found" lookup from
> a DNS server on your own network.

I do not see that in my tests.

Random hosts with reverse DNS:

[/]# time host 70.106.32.110
110.32.106.70.in-addr.arpa domain name pointer 
pool-70-106-32-110.hag.east.verizon.net.
real    0m0.005s
user    0m0.001s
sys     0m0.003s
[/]# time host 213.93.30.26
26.30.93.213.in-addr.arpa domain name pointer e30026.upc-e.chello.nl.
real    0m0.005s
user    0m0.001s
sys     0m0.006s


Random hosts without reverse DNS:


[/]# time host 211.195.99.146
Host 146.99.195.211.in-addr.arpa not found: 3(NXDOMAIN)
real    0m0.005s
user    0m0.000s
sys     0m0.002s
[/]# time host 61.149.3.145
Host 145.3.149.61.in-addr.arpa not found: 3(NXDOMAIN)
real    0m0.004s
user    0m0.000s
sys     0m0.004s

Typically hosts without any rdns are faster to lookup than hosts with
rdns records. From some poking around, it appears that negative rdns
lookups are cached for 10 minutes, up to a maximum of up to 3 hours:

Negative Caching of DNS Queries -> http://www.faqs.org/rfcs/rfc2308.html

Cami



Cami

RE: Faster rDNS lookups

[Rob McEwen <ro...@powerviewsystems.com>]

Please understand, I'm only proposing this as an alternative idea for
checking to see if a sending server's IP address has proper rDNS... NOT any
other type of DNS lookups.

Also, I don't have stats on this, but I know that most mail is spam and I
know that MUCH of this spam has no rDNS properly configured.

Furthermore, when rDNS is not properly configured and a subsequent lookup is
done on the same IP, I'm pretty sure that the negative (no found) lookup
isn't cached at all or at least not for long.

For example, if you do a lookup on blahblah.yourdomain.com and that domain
doesn't exist. Next, seconds later, create an A-record for
blahblah.yourdomain.com ...finally, seconds later, do another lookup for
blahblah.yourdomain.com

In my testing, that last lookup DOES find something... indicating that DNS
servers don't generally cache previously "not found" lookups... at least not
for long.

rDNS on a spammer's IP which doesn't have rDNS configured involves a similar
situation. Therefore, I don't think DNS servers cache the negative results
of rDNS checks either at all or for very long. But, ironically, these types
of checks are more expensive than most lookups. I've found that DNS server
often take longer to return a "not found" result when the IP address is from
some 3rd party network in Korea... in contrast to an "is-found" lookup from
a DNS server on your own network.

Also, if absolutely no caching of negative lookups takes place, this would
be even more beneficial for spam attacks from a single IP just prior to
tarpitting kicking in... but that particular need is not a prerequisite to
my idea being a good one overall... just "iceing on the cake".

--Rob McEwen

Re: Faster rDNS lookups

[Cami <ca...@mweb.co.za>]

Rob McEwen wrote:
>> Cami wrote
>>
>> Exactly how is this faster than using a dns caching nameserver?
> 
> As I mentioned, (1) artificially long caching times (well beyond TTL) can be
> set for both negative and positive return and (2) once cached, the lookup is
> not dependent on another 3rd party server which might have been overloaded
> or mis-configured.

Artificially long caching the result of overloaded or mis-configured
has the same drawbacks as regular dns caching servers, possibly even
worse.

You're welcome to provide some type of tests/data to backup your
ideas/claim but i'm somewhat specticle if anything positive/usefull
will come out of it. I see no problem whatsoever with regular dns
caching nameservers, even dealing with 10's of millions of mails /
lookups per day on my systems.

Doing non-cached lookups are expensive, thats a given. Use a caching
nameserver and its no longer expensive.

Cami

RE: Faster rDNS lookups

[Rob McEwen <ro...@powerviewsystems.com>]

>Cami asked
Exactly how is this faster than using a dns caching nameserver?

As I mentioned, (1) artificially long caching times (well beyond TTL) can be
set for both negative and positive return and (2) once cached, the lookup is
not dependent on another 3rd party server which might have been overloaded
or mis-configured.

--Rob McEwen

Re: Faster rDNS lookups

[Cami <ca...@mweb.co.za>]

Rob McEwen wrote:
> I've got an idea for faster rDNS lookups.
> 
> Before I present the solution. Here is the problem...
> 
> ...basically, rDNS checks are "expensive". They sometimes take a few seconds
> when done in real time and depend on the timeliness and of other people's
> DNS servers. This 1-5 (or more) seconds may not seem like a big deal... but
> if this can be more instantaneous, it increases the volume of messages a
> server-side spam filter can handle. (every little bit counts)
> 
> Anyone doubt me? Well... my point is valid for the same reason that it is
> typically MUCH faster to check again a DNSBL (like SURBL) than it is to
> convert domains found within the e-mail to IP addresses and then check these
> again SpamHaus's RBL. Both of these are good techniques... but DNSBL
> checking where each domain doesn't have to be resolved to an IP address is
> much faster. For the very same reasons... again... rDNS checks are a bit
> "expensive".

Exactly how is this faster than using a dns caching nameserver?

Cami

Re: Faster rDNS lookups

[mouss <us...@free.fr>]

Rob McEwen wrote:
> I've got an idea for faster rDNS lookups.
> 
> Before I present the solution. Here is the problem...
> 
> ...basically, rDNS checks are "expensive". They sometimes take a few seconds
> when done in real time and depend on the timeliness and of other people's
> DNS servers. This 1-5 (or more) seconds may not seem like a big deal... but
> if this can be more instantaneous, it increases the volume of messages a
> server-side spam filter can handle. (every little bit counts)
> 
> Anyone doubt me? Well... my point is valid for the same reason that it is
> typically MUCH faster to check again a DNSBL (like SURBL) than it is to
> convert domains found within the e-mail to IP addresses and then check these
> again SpamHaus's RBL. Both of these are good techniques... but DNSBL
> checking where each domain doesn't have to be resolved to an IP address is
> much faster. For the very same reasons... again... rDNS checks are a bit
> "expensive".
> 
> But consider that a given IP's rDNS doesn't change that much (on average).
> Therefore, it would be a huge time and resource saver to specially program a
> DNS server to receive rDNS queries in RBL-like format, to do the rDNS lookup
> on the IP address passed in the RBL-like query, and then to return results
> much like an RBL would... "Null" for having an rDNS, and "127.0.0.x" if the
> IP does NOT have rDNS. This way, the mail server is "tricked" into thinking
> that this is just another RBL.
> 
> You might ask, "where is the time savings if the DNS server has to fetch the
> rDNS all the same?"
> 
> Here is the clincher... because, as I mentioned, rDNS settings don't change
> often, therefore, PERHAPS.... the DNS server could be configured or
> programmed to cache BOTH positive and negative results for 24 hours (or
> whatever the config file requests... maybe even DAYS). This way, although
> the 1st rDNS query would still be expensive, subsequent queries would be
> lightening fast. Also, since rDNS should (ideally) never be enough of a
> factor by itself to mark a message as spam, then, therefore, a rare mishap
> on someone's legit mail server wouldn't block mail if all else is well. (And
> those few who have no problem blocking soly on rDNS... they'd not mind such
> extremely rare FPs anyways) Therefore, the negative caching would have
> virtually no effect on temporary mistakes made by other mail admins... and
> this is in contrast to the damage that might be done if a regular DNSBL's
> records were cached for too long and a FP comes along that was quickly and
> already "fixed" but then still "stuck" as blacklisted on a server due to
> such caching.
> 
> Also, before anyone mentions TTL values providing caching anyways...
> consider that you typically get not long or zero caching of negative
> results.
> 
> This feature I describe would be VERY beneficial because MANY spam filters
> which have to option to do RBL lookups could benefit. They could just remove
> their regular rDNS feature and then do the lookup I described and use the
> RBL-based return in the same way that they previously used the rDNS for.
> 
> Does anyone know of a publicly available RBL that already does this? Anyone
> interested in creating such a service? (I don't think I have the expertise
> to do this.)

If you cache negative results, you won't catch new dnsbl entries.