You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by "Warren Togami Jr." <wt...@gmail.com> on 2011/07/04 12:46:15 UTC
SpamTips.org: Why run your own DNS server?
Hey folks,
http://www.spamtips.org/2011/07/spamassassin-why-run-your-own-dns.html
I wrote this article about why it can be important to run your own DNS
server if you have a busy Spamassassin deployment.
Anyone have any better tips of an alternate DNS resolver, or
configuration options to improve this suggested configuration?
http://www.spamtips.org/p/ultimate-setup-guide.html
Please see my Ultimate Setup Guide for all the latest tweaks to maximize
your Spamassassin effectiveness and safety. Do you have any tips or
tricks that are not mentioned here?
https://admin.fedoraproject.org/mailman/listinfo/spamassassin-news
Subscribe here for my Spamassassin for Sysadmins Newsletter
Thanks,
Warren Togami
warren@togami.com
Re: SpamTips.org: Why run your own DNS server?
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
On 04.07.11 00:46, Warren Togami Jr. wrote:
>http://www.spamtips.org/2011/07/spamassassin-why-run-your-own-dns.html
>I wrote this article about why it can be important to run your own
>DNS server if you have a busy Spamassassin deployment.
it CAN but it doesn't always have to be.
We provide 6 DNS servers (behind 2 L3 switches), with forwarders set to
our local mirrors for some RBL's.
The point is the reachability and RTT.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I drive way too fast to worry about cholesterol.
Re: SpamTips.org: Why run your own DNS server?
Posted by "Warren Togami Jr." <wt...@gmail.com>.
On 7/4/2011 1:52 AM, Axb wrote:
> On 2011-07-04 12:46, Warren Togami Jr. wrote:
>> Hey folks,
>>
>> http://www.spamtips.org/2011/07/spamassassin-why-run-your-own-dns.html
>> I wrote this article about why it can be important to run your own DNS
>> server if you have a busy Spamassassin deployment.
>>
>> Anyone have any better tips of an alternate DNS resolver, or
>> configuration options to improve this suggested configuration?
>
> Warren
>
> Sadly, your post has unleashed a sequel of pretty useless hints & rants.
>
> "There is a drawback to running pdns-recursor. The above pdns-recursor
> instance is using ~400MB of memory. If you cannot afford this kind of
> memory use, you can reduce the limits in options max-cache-entries and
> max-packetcache-entries in /etc/pdns-recursor/recursor.conf as
> documented upstream. You will need to find a balance between memory use
> and effective cache hit performance."
>
> A small site will never use 400MB of DNS cacheing... don't scare ppl
> unnecessarily :)
> Larger sites already do local recursion and have the iron to to it.
> (other recursors will also use a lot of memory under high-ish load)
I am not 100% certain about this, but it appears that pdns-recursor is
tuned to "normal" patterns of DNS lookups (like web browsing or maybe a
squid proxy server). It is caching a large amount of useless data,
evidenced by the piss terrible cache hit ratio. My in-brain logic
without testing suggested that timing out most of that nearly-useless
cache may shrink memory usage considerably without making that poor
cache hit ratio much worse, since more recent data is often more
relevant. That is my theory anyway. I'm testing that now.
>
> Be careful when endorsing:
>
> "For example, DNS results of DNSBL and URIBL's are very transient in
> nature with tiny TTL's, so perhaps we could substantially reduce memory
> usage by forcing max-cache-ttl and max-negative-ttl to a much smaller
> duration. It also appears that the packetcache is far more effective
> than the cache at achieving hits, so we may be better off favoring the
> packetcache rather than the memory hogging and less effective cache."
>
> Reducing negative TTL time should ONLY be done the user runs *local*
> copies of most of the queried BLs, otherwsise he may hit BL abuse
> threshold way earlier.
>
> BLs generally adjust their negative TTL to get a practical balance
> between query load and positive hits.
> Gaming these settings can become a costly process.
>
> Axb
Good point, I'll remove that paragraph for now and actually test that
theory myself to see how it effects the actual hit/miss ratio.
Warren
Re: SpamTips.org: Why run your own DNS server?
Posted by "Warren Togami Jr." <wt...@gmail.com>.
On 7/4/2011 1:52 AM, Axb wrote:
> A small site will never use 400MB of DNS cacheing... don't scare ppl
> unnecessarily :)
> Larger sites already do local recursion and have the iron to to it.
> (other recursors will also use a lot of memory under high-ish load)
It is also possible that pdns-recursor just sucks and I should be trying
other daemons. I will try unbound next.
Warren
Re: TTL and DNSBLs [per-IP negative-cache TTL]
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>"David F. Skoll" <df...@roaringpenguin.com> wrote:
>> On Thu, 7 Jul 2011 11:50:44 +0200
>> Matus UHLAR - fantomas <uh...@fantomas.sk> wrote:
>>
>>> Negative caching can be effective or in this case even
>>> ineffective too, can't it?
>>
>> The point is that by definition, you can't have a per-IP
>> negative-cache TTL.
On 07.07.11 15:31, Andrzej Adam Filip wrote:
>But it is possible to use a wildcard DNS record for "not listed",
>is not it? :-)
it can, but it would have no positive effect due to way how wildcards
in DNS work.
>The question is: Would it be cost effective?
there would be a cost of returning positive answer instead of negative.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
One OS to rule them all, One OS to find them,
One OS to bring them all and into darkness bind them
Re: TTL and DNSBLs [how to improve DNSBL/DNSWL "cache-ability"]
Posted by Andrzej Adam Filip <an...@gmail.com>.
"David F. Skoll" <df...@roaringpenguin.com> wrote:
> On Thu, 07 Jul 2011 15:31:47 +0200
> Andrzej Adam Filip <an...@gmail.com> wrote:
>
>> > The point is that by definition, you can't have a per-IP
>> > negative-cache TTL.
>
>> But it is possible to use a wildcard DNS record for "not listed",
>> is not it? :-)
>
> That would not work well at all... think about the ramifications. :)
> Either the cached wildcard record would prevent you from querying IPs
> you've never seen before (in which case the DNSBL would be useless) or
> it would forward the specific query anyway (in which case it does nothing
> to solve the problem.)
In the email you replied I have merely suggested using wildcards for */24
and */16 nets with "not listed" record to allow setting different TTL.
[ As I understand it will do little beyond allowing custom TTL]
> Anyway, IMO, DNS should not be used for blacklist lookups over the
> Internet. DNS was never designed for that; it just happens to
> sort-of-work. DNSRBLs are frequently-changing and are supposed to
> provide up-to-the-minute information. DNS is a good protocol for
> querying a local authoritative name server, but it's not good for
> distributing large volumes of quickly-changing, required-to-be-fresh
> data across the Internet.
*But*
a1) DNS servers (forwarders) can be modified e.g. to treat cached record
for 3.2.1.dnsbl.example.net as matching query for 4.3.2.1.dnsbl.example.net
[ on per domain basis or per *new* DNS record ]
a2) DNS servers (authoritative) may be modified to *append* such "more general"
reply when possible
b) I bet [not too much :-) ] that some tricks with delegations
(NS records) in 4.3.2.1.in-addr.arpa style structure would allow
to achieve better "negative caching" for */24 and */16 nets.
> (What's saving us is computing power and bandwidth. Modern name servers
> can handle a huge number of queries without too much trouble, so people
> never even noticed that caching wasn't buying them anything.)
To put in a way we may both accept:
Improving DNSBL/DNWL "cache-abbility" is not an urgent need :-)
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu
Eeny, Meeny, Jelly Beanie, the spirits are about to speak!
-- Bullwinkle Moose
Re: TTL and DNSBLs [per-IP negative-cache TTL]
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Thu, 07 Jul 2011 15:31:47 +0200
Andrzej Adam Filip <an...@gmail.com> wrote:
> > The point is that by definition, you can't have a per-IP
> > negative-cache TTL.
> But it is possible to use a wildcard DNS record for "not listed",
> is not it? :-)
That would not work well at all... think about the ramifications. :)
Either the cached wildcard record would prevent you from querying IPs
you've never seen before (in which case the DNSBL would be useless) or
it would forward the specific query anyway (in which case it does nothing
to solve the problem.)
Anyway, IMO, DNS should not be used for blacklist lookups over the
Internet. DNS was never designed for that; it just happens to
sort-of-work. DNSRBLs are frequently-changing and are supposed to
provide up-to-the-minute information. DNS is a good protocol for
querying a local authoritative name server, but it's not good for
distributing large volumes of quickly-changing, required-to-be-fresh
data across the Internet.
(What's saving us is computing power and bandwidth. Modern name servers
can handle a huge number of queries without too much trouble, so people
never even noticed that caching wasn't buying them anything.)
Regards,
David.
Re: TTL and DNSBLs [per-IP negative-cache TTL]
Posted by Andrzej Adam Filip <an...@gmail.com>.
"David F. Skoll" <df...@roaringpenguin.com> wrote:
> On Thu, 7 Jul 2011 11:50:44 +0200
> Matus UHLAR - fantomas <uh...@fantomas.sk> wrote:
>
>> Negative caching can be effective or in this case even
>> ineffective too, can't it?
>
> The point is that by definition, you can't have a per-IP
> negative-cache TTL.
But it is possible to use a wildcard DNS record for "not listed",
is not it? :-)
The question is: Would it be cost effective?
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu
I never met a man I didn't want to fight.
-- Lyle Alzado, professional football lineman
Re: TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS
server?)
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Thu, 7 Jul 2011 16:10:36 +0200
Matus UHLAR - fantomas <uh...@fantomas.sk> wrote:
> Do you have memory for your nameserver limited or not?
No. I simulated a name server with an infinite cache size.
> Does it only expire RR's when they time out?
Yes.
> what logs did you procvess?
The mail log from sendmail.
> Do you log responses with informations if they came out of cache?
I ran a simulation. I had a Perl script that simulated a name server
with infinite cache, and I based the lookup timeouts on the time
stamps in the logs. Basically, I kept a big hash of when each IP
address would be looked up and the timestamp of last lookup. If the
IP address was encountered again within the TTL, then it was counted
as a cache hit. If it was encountered after the TTL would have expired,
it was counted as a cache miss and the timestamp was updated (to simulate
a cache refresh.)
> well, connections to our mailservers _do_ repeat, expecially for
> servers like google, facebook or similar services.
Yes, *good* ones repeat. But *bad* ones don't, and unfortunately the bad
ones outnumber the good ones by a wide margin.
Regards,
David.
Re: TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS
server?)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>On Thu, 7 Jul 2011 14:39:48 +0200
>Matus UHLAR - fantomas <uh...@fantomas.sk> wrote:
>> And in case of repeating the same IP's (which happens especially with
>> remote mailservers) the negative cache helps much.
On 07.07.11 09:09, David F. Skoll wrote:
>No, it does not. I have run experiments on real mail servers. I'm not
>just making this up. If you like, I can send you my log analysis code
>so you can run the same experiments on your own mail servers.
Do you have memory for your nameserver limited or not? Does it only
expire RR's when they time out?
what logs did you procvess? Do you log responses with informations if
they came out of cache?
>> In fact, I think that DNSBL's operators should provide the same TTL
>> for both positive and negative answers, that can be even as big as
>> 12-24h for dynamic/policy block lists.
>As I mentioned, it's not in their interests to do that (if they sell access
>to heavy users.) Additionally, though I haven't experimented very extensively,
>my tests show that cache effectiveness is not very sensitive to TTL. Real
>mail servers tend to be hit by a *lot* of different IP addresses, many of which
>don't repeat for hours (if ever).
well, connections to our mailservers _do_ repeat, expecially for
servers like google, facebook or similar services. Of course, there ARE
bots, zombies and drones from the internet that do not reconect to our
servers but there are many that do.
However I did not measure statistic effectiveness. OTOH, it's good when
we have cached result for google servers and not for drones... :)
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I don't have lysdexia. The Dog wouldn't allow that.
Re: TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS
server?)
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Thu, 7 Jul 2011 14:39:48 +0200
Matus UHLAR - fantomas <uh...@fantomas.sk> wrote:
> On 07.07.11 08:26, David F. Skoll wrote:
> >The point is that by definition, you can't have a per-IP
> >negative-cache TTL.
> We can have per-IP positive cache and per-zone negative cache.
That does not help.
> And in case of repeating the same IP's (which happens especially with
> remote mailservers) the negative cache helps much.
No, it does not. I have run experiments on real mail servers. I'm not
just making this up. If you like, I can send you my log analysis code
so you can run the same experiments on your own mail servers.
> In fact, I think that DNSBL's operators should provide the same TTL
> for both positive and negative answers, that can be even as big as
> 12-24h for dynamic/policy block lists.
As I mentioned, it's not in their interests to do that (if they sell access
to heavy users.) Additionally, though I haven't experimented very extensively,
my tests show that cache effectiveness is not very sensitive to TTL. Real
mail servers tend to be hit by a *lot* of different IP addresses, many of which
don't repeat for hours (if ever).
Regards,
David.
Re: TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS
server?)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>On Thu, 7 Jul 2011 11:50:44 +0200
>Matus UHLAR - fantomas <uh...@fantomas.sk> wrote:
>> Negative caching can be effective or in this case even
>> ineffective too, can't it?
On 07.07.11 08:26, David F. Skoll wrote:
>The point is that by definition, you can't have a per-IP negative-cache TTL.
We can have per-IP positive cache and per-zone negative cache. As you
mentioned earlier, spamhaus uses 15-30 min TTL. I found it use 900s
(=15m) positive TTL and 150s (=2.5min) negative TTL.
And in case of repeating the same IP's (which happens especially with
remote mailservers) the negative cache helps much.
In fact, I think that DNSBL's operators should provide the same TTL for
both positive and negative answers, that can be even as big as 12-24h
for dynamic/policy block lists. It's alwayt onto clients' nameserver to
decide which it will cache and which not, mostly based on their usage.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
They that can give up essential liberty to obtain a little temporary
safety deserve neither liberty nor safety. -- Benjamin Franklin, 1759
Re: TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS
server?)
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Thu, 7 Jul 2011 11:50:44 +0200
Matus UHLAR - fantomas <uh...@fantomas.sk> wrote:
> Negative caching can be effective or in this case even
> ineffective too, can't it?
The point is that by definition, you can't have a per-IP negative-cache TTL.
Regards,
David.
Re: TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS
server?)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>> > My experiments on real mail servers show that DNS caching is quite
>> > ineffective for DNSBLs (at least for typical ones like Spamhaus that
>> > use a short TTL on the order of 15-30 minutes.)
>On Tue, 5 Jul 2011 12:20:18 +0200
>Michelle Konzack <li...@tamay-dogan.net> wrote:
>> Is the TTL set global or are the TTLs set by IP?
On 05.07.11 16:37, David F. Skoll wrote:
>My simulation assumed the same TTL for all addresses. But that's not
>relevant to cache effectiveness since most IPs are not listed in a
>DNSBL (and you end up using the negative-cache-TTL.)
Negative caching can be effective or in this case even
ineffective too, can't it?
>> I have a local bind9 which implement my private list and it set the
>> TTL per target which is quiet effectiv
>
>If most targets are not listed, per-IP TTLs don't matter.
They do, if the targets repeat.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
How does cat play with mouse? cat /dev/mouse
Re: TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS
server?)
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Tue, 5 Jul 2011 12:20:18 +0200
Michelle Konzack <li...@tamay-dogan.net> wrote:
> > My experiments on real mail servers show that DNS caching is quite
> > ineffective for DNSBLs (at least for typical ones like Spamhaus that
> > use a short TTL on the order of 15-30 minutes.)
> Is the TTL set global or are the TTLs set by IP?
My simulation assumed the same TTL for all addresses. But that's not
relevant to cache effectiveness since most IPs are not listed in a
DNSBL (and you end up using the negative-cache-TTL.)
> I have a local bind9 which implement my private list and it set the
> TTL per target which is quiet effectiv
If most targets are not listed, per-IP TTLs don't matter.
Regards,
David.
Re: TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS
server?)
Posted by Martin Gregorie <ma...@gregorie.org>.
On Tue, 2011-07-05 at 12:20 +0200, Michelle Konzack wrote:
> Am 2011-07-04 09:24:19, hacktest Du folgendes herunter:
> > My experiments on real mail servers show that DNS caching is quite
> > ineffective for DNSBLs (at least for typical ones like Spamhaus that
> > use a short TTL on the order of 15-30 minutes.)
>
> Is the TTL set global or are the TTLs set by IP?
>
TTL is set for each zone file.
Martin
Re: TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS
server?)
Posted by Michelle Konzack <li...@tamay-dogan.net>.
Hello David F. Skoll,
Am 2011-07-04 09:24:19, hacktest Du folgendes herunter:
> My experiments on real mail servers show that DNS caching is quite
> ineffective for DNSBLs (at least for typical ones like Spamhaus that
> use a short TTL on the order of 15-30 minutes.)
Is the TTL set global or are the TTLs set by IP?
> Executive summary: On a very quiet mail server, assuming a 15-minute
> TTL, there was only a 50% cache hit rate on DNSBL lookups. On a
> fairly busy mail server, the cache hit rate fell to 22%.
I have a local bind9 which implement my private list and it set the TTL
per target which is quiet effectiv
> The problem, of course, is that most mail servers are hit by
> connections from all over the place... spammers have a lot of IP
> addresses to choose from, so you don't get much repetition within the
> TTL of a typical DNSBL. If you really need high-performance DNSBL
> lookups, you need to arrange for a zone transfer and run a local
> authoritative name server for the DNSBL.
1+
This can be costly, since the zone transfers are only for bigger usage
and not more free...
> Regards,
> David.
Thanks, Greetings and nice Day/Evening
Michelle Konzack
--
##################### Debian GNU/Linux Consultant ######################
Development of Intranet and Embedded Systems with Debian GNU/Linux
itsystems@tdnet France itsystems@tdnet
Owner Michelle Konzack Owner Michelle Konzack
Apt. 917 (homeoffice) Gewerbe Straße 3
50, rue de Soultz 77694 Kehl/Germany
67100 Strasbourg/France Tel: +49-177-9351947 mobil
Tel: +33-6-61925193 mobil Tel: +49-176-86004575 office
<http://www.itsystems.tamay-dogan.net/> <http://www.flexray4linux.org/>
<http://www.debian.tamay-dogan.net/> <http://www.can4linux.org/>
Jabber linux4michelle@jabber.ccc.de
ICQ #328449886
Linux-User #280138 with the Linux Counter, http://counter.li.org/
DUL/DUL+ redesign to improve DNS cache hit ratio [Was: TTL and DNSBLs]
Posted by Andrzej Adam Filip <an...@gmail.com>.
"David F. Skoll" <df...@roaringpenguin.com> wrote:
> On Mon, 04 Jul 2011 13:52:00 +0200
> Axb <ax...@gmail.com> wrote:
>
>> BLs generally adjust their negative TTL to get a practical balance
>> between query load and positive hits.
>> Gaming these settings can become a costly process.
>
> My experiments on real mail servers show that DNS caching is quite
> ineffective for DNSBLs (at least for typical ones like Spamhaus that
> use a short TTL on the order of 15-30 minutes.)
>
> Results of my experiments are in these slides (PDF):
> http://ipv6summit.ca/index.php/v6/2011/paper/view/8/4
>
> Executive summary: On a very quiet mail server, assuming a 15-minute
> TTL, there was only a 50% cache hit rate on DNSBL lookups. On a
> fairly busy mail server, the cache hit rate fell to 22%.
>
> The problem, of course, is that most mail servers are hit by
> connections from all over the place... spammers have a lot of IP
> addresses to choose from, so you don't get much repetition within the
> TTL of a typical DNSBL. If you really need high-performance DNSBL
> lookups, you need to arrange for a zone transfer and run a local
> authoritative name server for the DNSBL.
Would you recommend redesigning (mainly) DUL/DUL+ DNSBL lists to improve
DNS cache hit ratio?
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu
The Second Law of Thermodynamics:
If you think things are in a mess now, just wait!
-- Jim Warner
Re: DUL/DUL+ redesign to improve DNS cache hit ratio [SA v. MTA]
Posted by Andrzej Adam Filip <an...@gmail.com>.
"David F. Skoll" <df...@roaringpenguin.com> wrote:
> On Wed, 06 Jul 2011 08:15:47 +0200
> Andrzej Adam Filip <an...@gmail.com> wrote:
>
>> It may improve performance e.g. in case of hundredths mail servers in
>> a data/co-location center using shared forwarder and rejecting on
>> first DNSBL hit. Somehow I doubt buying data for such "reseller"
>> configuration is legally encouraged by paid DNSBL operators.
>
> This is true. But it's also not in paid DNSBL operators interest to
> improve the hit ratio. If the cache hit ratio is improved too much,
> the DNSBL operators would be unable to detect heavy users and ask
> (threaten) them for money. :) In the limiting case, if the cache
> becomes *too* effective, the organization hosting the cache *is*
> effectively providing the whole data set to its users.
To put it short:
a) Only DNSBL listing "net ranges" (e.g. DUL/DUL+, network "reputation")
can be quite easily redesigned to improve DNS hit ratio (IMHO)
b) Free of charge DNSBL would benefit the most
c) In case of DUL list quality is not (IMHO) defined by big */16 entries
(e.g. home ADSL ranges) that will generate most DNS cache hits
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu
I do not believe that this generation of Americans is willing to resign
itself to going to bed each night by the light of a Communist moon...
-- Lyndon B. Johnson
Re: DUL/DUL+ redesign to improve DNS cache hit ratio [SA v. MTA]
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Wed, 06 Jul 2011 08:15:47 +0200
Andrzej Adam Filip <an...@gmail.com> wrote:
> It may improve performance e.g. in case of hundredths mail servers in
> a data/co-location center using shared forwarder and rejecting on
> first DNSBL hit. Somehow I doubt buying data for such "reseller"
> configuration is legally encouraged by paid DNSBL operators.
This is true. But it's also not in paid DNSBL operators interest to
improve the hit ratio. If the cache hit ratio is improved too much,
the DNSBL operators would be unable to detect heavy users and ask
(threaten) them for money. :) In the limiting case, if the cache
becomes *too* effective, the organization hosting the cache *is*
effectively providing the whole data set to its users.
Regards,
David.
Re: DUL/DUL+ redesign to improve DNS cache hit ratio [SA v. MTA]
Posted by Andrzej Adam Filip <an...@gmail.com>.
"David F. Skoll" <df...@roaringpenguin.com> wrote:
> On Tue, 05 Jul 2011 23:26:16 +0200
> Andrzej Adam Filip <an...@gmail.com> wrote:
>
>> Would you recommend redesigning (mainly) DUL/DUL+ DNSBL lists to
>> improve DNS cache hit ratio?
>
> No, not really. The poor cache hit ratio doesn't seem to be a problem
> in practice (most people were surprised by the results). If you have
> a high-enough lookup volume that it does become a problem, you just arrange
> to obtain (or buy) the data and run a local authoritative name server.
You are most likely right in case of SA asking all configured DNSBL to
generate spam score - improvement of some/minority DNS cache hit ration
would not be impressive in improving overall preference.
*BUT*
It may improve performance e.g. in case of hundredths mail servers in a
data/co-location center using shared forwarder and rejecting on first
DNSBL hit. Somehow I doubt buying data for such "reseller" configuration
is legally encouraged by paid DNSBL operators.
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu
Welcome to Lake Wobegon, where all the men are strong, the women are pretty,
and the children are above-average.
-- Garrison Keillor
Re: DUL/DUL+ redesign to improve DNS cache hit ratio [Was: TTL and
DNSBLs]
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Tue, 05 Jul 2011 23:26:16 +0200
Andrzej Adam Filip <an...@gmail.com> wrote:
> Would you recommend redesigning (mainly) DUL/DUL+ DNSBL lists to
> improve DNS cache hit ratio?
No, not really. The poor cache hit ratio doesn't seem to be a problem
in practice (most people were surprised by the results). If you have
a high-enough lookup volume that it does become a problem, you just arrange
to obtain (or buy) the data and run a local authoritative name server.
Regards,
David.
DUL/DUL+ redesign to improve DNS cache hit ratio [Was: TTL and DNSBLs]
Posted by Andrzej Adam Filip <an...@gmail.com>.
"David F. Skoll" <df...@roaringpenguin.com> wrote:
> On Mon, 04 Jul 2011 13:52:00 +0200
> Axb <ax...@gmail.com> wrote:
>
>> BLs generally adjust their negative TTL to get a practical balance
>> between query load and positive hits.
>> Gaming these settings can become a costly process.
>
> My experiments on real mail servers show that DNS caching is quite
> ineffective for DNSBLs (at least for typical ones like Spamhaus that
> use a short TTL on the order of 15-30 minutes.)
>
> Results of my experiments are in these slides (PDF):
> http://ipv6summit.ca/index.php/v6/2011/paper/view/8/4
>
> Executive summary: On a very quiet mail server, assuming a 15-minute
> TTL, there was only a 50% cache hit rate on DNSBL lookups. On a
> fairly busy mail server, the cache hit rate fell to 22%.
>
> The problem, of course, is that most mail servers are hit by
> connections from all over the place... spammers have a lot of IP
> addresses to choose from, so you don't get much repetition within the
> TTL of a typical DNSBL. If you really need high-performance DNSBL
> lookups, you need to arrange for a zone transfer and run a local
> authoritative name server for the DNSBL.
Would you recommend redesigning (mainly) DUL/DUL+ DNSBL lists to improve
DNS cache hit ratio?
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@onet.eu
The Second Law of Thermodynamics:
If you think things are in a mess now, just wait!
-- Jim Warner
Re: caches, was TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS server?)
Posted by John Levine <jo...@taugh.com>.
>> But if you're looking for a DNS cache, I highly recommend unbound.
>> I used to use dnscache but got tired of its limitations (due entirely
>> to it being unchanged since 1998.) My copy of unbound runs about
>> 27M real RAM, 44M virtual, which is pretty modest on my 12G server.
>
>how many q/s is that instance handling?
It's quite variable. I get hourly reports with the queries per hour
ranging from 10K to 90K, so I'd guess that typically it's 10-15 qps.
R's,
John
Re: TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS server?)
Posted by Axb <ax...@gmail.com>.
On 2011-07-04 21:26, John Levine wrote:
>> My experiments on real mail servers show that DNS caching is quite
>> ineffective for DNSBLs (at least for typical ones like Spamhaus that
>> use a short TTL on the order of 15-30 minutes.)
>
> That's consistent with what I've seen, although you probably won't be
> surprised to hear that I have higher hopes for my range published
> DNSxLs than David does, partly because I expect them to be used for
> whitelist which tend to cache better for technical reasons.
>
> But if you're looking for a DNS cache, I highly recommend unbound.
> I used to use dnscache but got tired of its limitations (due entirely
> to it being unchanged since 1998.) My copy of unbound runs about
> 27M real RAM, 44M virtual, which is pretty modest on my 12G server.
how many q/s is that instance handling?
Re: TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS server?)
Posted by John Levine <jo...@taugh.com>.
>My experiments on real mail servers show that DNS caching is quite
>ineffective for DNSBLs (at least for typical ones like Spamhaus that
>use a short TTL on the order of 15-30 minutes.)
That's consistent with what I've seen, although you probably won't be
surprised to hear that I have higher hopes for my range published
DNSxLs than David does, partly because I expect them to be used for
whitelist which tend to cache better for technical reasons.
But if you're looking for a DNS cache, I highly recommend unbound.
I used to use dnscache but got tired of its limitations (due entirely
to it being unchanged since 1998.) My copy of unbound runs about
27M real RAM, 44M virtual, which is pretty modest on my 12G server.
R's,
John
TTL and DNSBLs (was Re: SpamTips.org: Why run your own DNS server?)
Posted by "David F. Skoll" <df...@roaringpenguin.com>.
On Mon, 04 Jul 2011 13:52:00 +0200
Axb <ax...@gmail.com> wrote:
> BLs generally adjust their negative TTL to get a practical balance
> between query load and positive hits.
> Gaming these settings can become a costly process.
My experiments on real mail servers show that DNS caching is quite
ineffective for DNSBLs (at least for typical ones like Spamhaus that
use a short TTL on the order of 15-30 minutes.)
Results of my experiments are in these slides (PDF):
http://ipv6summit.ca/index.php/v6/2011/paper/view/8/4
Executive summary: On a very quiet mail server, assuming a 15-minute
TTL, there was only a 50% cache hit rate on DNSBL lookups. On a
fairly busy mail server, the cache hit rate fell to 22%.
The problem, of course, is that most mail servers are hit by
connections from all over the place... spammers have a lot of IP
addresses to choose from, so you don't get much repetition within the
TTL of a typical DNSBL. If you really need high-performance DNSBL
lookups, you need to arrange for a zone transfer and run a local
authoritative name server for the DNSBL.
Regards,
David.
Re: SpamTips.org: Why run your own DNS server?
Posted by Axb <ax...@gmail.com>.
On 2011-07-04 12:46, Warren Togami Jr. wrote:
> Hey folks,
>
> http://www.spamtips.org/2011/07/spamassassin-why-run-your-own-dns.html
> I wrote this article about why it can be important to run your own DNS
> server if you have a busy Spamassassin deployment.
>
> Anyone have any better tips of an alternate DNS resolver, or
> configuration options to improve this suggested configuration?
Warren
Sadly, your post has unleashed a sequel of pretty useless hints & rants.
"There is a drawback to running pdns-recursor. The above pdns-recursor
instance is using ~400MB of memory. If you cannot afford this kind of
memory use, you can reduce the limits in options max-cache-entries and
max-packetcache-entries in /etc/pdns-recursor/recursor.conf as
documented upstream. You will need to find a balance between memory use
and effective cache hit performance."
A small site will never use 400MB of DNS cacheing... don't scare ppl
unnecessarily :)
Larger sites already do local recursion and have the iron to to it.
(other recursors will also use a lot of memory under high-ish load)
Be careful when endorsing:
"For example, DNS results of DNSBL and URIBL's are very transient in
nature with tiny TTL's, so perhaps we could substantially reduce memory
usage by forcing max-cache-ttl and max-negative-ttl to a much smaller
duration. It also appears that the packetcache is far more effective
than the cache at achieving hits, so we may be better off favoring the
packetcache rather than the memory hogging and less effective cache."
Reducing negative TTL time should ONLY be done the user runs *local*
copies of most of the queried BLs, otherwsise he may hit BL abuse
threshold way earlier.
BLs generally adjust their negative TTL to get a practical balance
between query load and positive hits.
Gaming these settings can become a costly process.
Axb
Re: SpamTips.org: Why run your own DNS server?
Posted by Jorn Argelo <jo...@wcborstel.com>.
On 4-7-2011 20:35, Toni Mueller wrote:
> Hi,
>
> On Mon, 04.07.2011 at 13:23:42 +0200, Axb<ax...@gmail.com> wrote:
>>> Well, together with the auth server it creates an "ecosystem" with
>>> some (limited) vendor lock-in capability.
>> "vendor lock-in" ? be explicit, please.
> last I looked PowerDNS highlighted some custom DNS RR types (on the
> wire) that were not usable with any other software. Migrating from one
> DNS server package to another is imho difficult enough without that
> already.
>
>
> Kind regards,
> --Toni++
Well, if you're concerned about that then don't use the custom RR types
that PowerDNS provides. I don't really see the fuss about it and saying
it's vendor lock-in is simply not true. It's just some non-standard RR
types and that's that. You're not being forced to use them or anything.
Regards,
Jorn
Re: SpamTips.org: Why run your own DNS server?
Posted by Toni Mueller <su...@oeko.net>.
Hi,
On Mon, 04.07.2011 at 13:23:42 +0200, Axb <ax...@gmail.com> wrote:
> >Well, together with the auth server it creates an "ecosystem" with
> >some (limited) vendor lock-in capability.
>
> "vendor lock-in" ? be explicit, please.
last I looked PowerDNS highlighted some custom DNS RR types (on the
wire) that were not usable with any other software. Migrating from one
DNS server package to another is imho difficult enough without that
already.
Kind regards,
--Toni++
Re: SpamTips.org: Why run your own DNS server?
Posted by Axb <ax...@gmail.com>.
On 2011-07-04 13:15, Toni Mueller wrote:
>> I don't believe pdns-recursor is guilty of this particular complaint
>> as it is ONLY a recursor?
>
> Well, together with the auth server it creates an "ecosystem" with
> some (limited) vendor lock-in capability.
"vendor lock-in" ? be explicit, please.
Re: SpamTips.org: Why run your own DNS server?
Posted by Toni Mueller <su...@oeko.net>.
Hi Warren,
On Mon, 04.07.2011 at 01:03:46 -1000, Warren Togami Jr. <wt...@gmail.com> wrote:
> I heard others recommend unbound, but I haven't tried it yet. Is it
> more RAM efficient than other alternatives, and fast?
I haven't specifically conducted tests about its memory efficiency, but
I do use it on several servers of mine, and of customers, too. It is an
ISP style DNS server, and subjectively runs quite fast. But I don't use
DNSSEC...
For us, unbound is the natural successor of dnscache, which was
excellent in its day, but (imho) doesn't seem to keep up with today's
growing wishlist of features (eg. IPv6 support), unless you're ready
for some serious hackery.
> I don't believe pdns-recursor is guilty of this particular complaint
> as it is ONLY a recursor?
Well, together with the auth server it creates an "ecosystem" with
some (limited) vendor lock-in capability.
Kind regards,
--Toni++
Re: SpamTips.org: Why run your own DNS server?
Posted by "Warren Togami Jr." <wt...@gmail.com>.
On 7/4/2011 12:58 AM, Toni Mueller wrote:
>
> Hi Warren,
>
> On Mon, 04.07.2011 at 00:46:15 -1000, Warren Togami Jr.<wt...@gmail.com> wrote:
>> http://www.spamtips.org/2011/07/spamassassin-why-run-your-own-dns.html
>>
>> Anyone have any better tips of an alternate DNS resolver, or
>> configuration options to improve this suggested configuration?
>
> while I do agree that it is generally a very good idea to run your own
> DNS resolver, even if you have less than one mail per day, I am
> thorougly unconvinced about the qualities of PowerDNS. I do have a
> suggested alternative, though.
>
> http://unbound.net/
>
> This server doesn't go to proprietary changes to the DNS protocol (like
> inventing new record types that noone else understands), but
> concentrates on delivering DNS according to the latest specs instead.
I heard others recommend unbound, but I haven't tried it yet. Is it
more RAM efficient than other alternatives, and fast?
I don't believe pdns-recursor is guilty of this particular complaint as
it is ONLY a recursor?
Warren
Re: SpamTips.org: Why run your own DNS server?
Posted by Toni Mueller <su...@oeko.net>.
Hi Warren,
On Mon, 04.07.2011 at 00:46:15 -1000, Warren Togami Jr. <wt...@gmail.com> wrote:
> http://www.spamtips.org/2011/07/spamassassin-why-run-your-own-dns.html
>
> Anyone have any better tips of an alternate DNS resolver, or
> configuration options to improve this suggested configuration?
while I do agree that it is generally a very good idea to run your own
DNS resolver, even if you have less than one mail per day, I am
thorougly unconvinced about the qualities of PowerDNS. I do have a
suggested alternative, though.
http://unbound.net/
This server doesn't go to proprietary changes to the DNS protocol (like
inventing new record types that noone else understands), but
concentrates on delivering DNS according to the latest specs instead.
You can most likely install this software via your favourite package
management system.
Kind regards,
--Toni++
Re: SpamTips.org: Why run your own DNS server?
Posted by Toni Mueller <su...@oeko.net>.
Hi,
On Tue, 05.07.2011 at 07:18:30 +0000, Jason Ede <J....@birchenallhowden.co.uk> wrote:
> Andreas Schulze [mailto:sca@andreasschulze.de] wrote:
> > - bind (off course)
although I'm sure that it was meant in a different way, "off course"
hits the nail right onto the head, imnsho.
> Are there any figures on the relative merits/speeds of these servers?
> Bind is the default on at least redhat based installations.
Bind has the following widely acknowledged properties:
* a plethora of security problems across all versions
* generates high system loads
* notoriously hard to configure
* (also partially) their own brand of DNS standards
* and an a**h*le style support ("pay through your nose, and you'll get
security fixes x months in advance")
Therefore, at least some people are striving to replace Bind with other
software. Although I'm far from intimate to the decision, OpenBSD has
imported NSD into their base quite some time ago, and ships it with
their current release 4.9 in the base system. Afaik, the plan is to
deprecate Bind some time down the road. NSD is by the same guys who
(later) wrote unbound, and it was, afair, partially funded by RIPE.
Subjectively, and without any kind of benchmarking, I can only say that
both dnscache and unbound perform at least five, but probably more like
ten times as fast as Bind does.
Kind regards,
--Toni++
Re: SpamTips.org: Why run your own DNS server?
Posted by Jose Borges Ferreira <un...@gmail.com>.
Got this table from this very nice book (
http://www.amazon.com/Alternative-DNS-Servers-Deployment-Back-Ends/dp/0954452992/ref=sr_1_1?ie=UTF8&qid=1309891205&sr=8-1).
It's a bit outdated ( well, not for dnscache :p ) but you can have an
idea.
José Borges Ferreira
On Tue, Jul 5, 2011 at 8:18 AM, Jason Ede <J....@birchenallhowden.co.uk>wrote:
> > -----Original Message-----
> > From: Andreas Schulze [mailto:sca@andreasschulze.de]
> > Sent: 04 July 2011 12:11
> > To: Warren Togami Jr.
> > Cc: users@spamassassin.apache.org
> > Subject: Re: SpamTips.org: Why run your own DNS server?
> >
> > Warren,
> >
> > > Anyone have any better tips of an alternate DNS resolver, or
> > > configuration options to improve this suggested configuration?
> >
> > please distinguish between DNS server and recursive+caching resolver.
> > The HowTo meen the second one...
> >
> > http://en.wikipedia.org/wiki/Domain_Name_System#Name_servers
> >
> >
> > other resolvers installable by users are
> > - unbound ( http://unbound.net )
> > - dnscache ( http://cr.yp.to/dnscache.html )
> > - bind (off course)
> > - http://en.wikipedia.org/wiki/Comparison_of_DNS_server_software
> >
> > --
> > Andreas Schulze
> []
> Oops... meant to reply to list.
>
> Are there any figures on the relative merits/speeds of these servers? Bind
> is the default on at least redhat based installations.
>
> Jason
>
RE: SpamTips.org: Why run your own DNS server?
Posted by Jason Ede <J....@birchenallhowden.co.uk>.
> -----Original Message-----
> From: Andreas Schulze [mailto:sca@andreasschulze.de]
> Sent: 04 July 2011 12:11
> To: Warren Togami Jr.
> Cc: users@spamassassin.apache.org
> Subject: Re: SpamTips.org: Why run your own DNS server?
>
> Warren,
>
> > Anyone have any better tips of an alternate DNS resolver, or
> > configuration options to improve this suggested configuration?
>
> please distinguish between DNS server and recursive+caching resolver.
> The HowTo meen the second one...
>
> http://en.wikipedia.org/wiki/Domain_Name_System#Name_servers
>
>
> other resolvers installable by users are
> - unbound ( http://unbound.net )
> - dnscache ( http://cr.yp.to/dnscache.html )
> - bind (off course)
> - http://en.wikipedia.org/wiki/Comparison_of_DNS_server_software
>
> --
> Andreas Schulze
[]
Oops... meant to reply to list.
Are there any figures on the relative merits/speeds of these servers? Bind is the default on at least redhat based installations.
Jason
Re: SpamTips.org: Why run your own DNS server?
Posted by Andreas Schulze <sc...@andreasschulze.de>.
Warren,
> Anyone have any better tips of an alternate DNS resolver, or
> configuration options to improve this suggested configuration?
please distinguish between DNS server and recursive+caching resolver.
The HowTo meen the second one...
http://en.wikipedia.org/wiki/Domain_Name_System#Name_servers
other resolvers installable by users are
- unbound ( http://unbound.net )
- dnscache ( http://cr.yp.to/dnscache.html )
- bind (off course)
- http://en.wikipedia.org/wiki/Comparison_of_DNS_server_software
--
Andreas Schulze
Re: SpamTips.org: Why run your own DNS server?
Posted by Benny Pedersen <me...@junc.org>.
On Mon, 04 Jul 2011 00:46:15 -1000, Warren Togami Jr. wrote:
>
> http://www.spamtips.org/2011/07/spamassassin-why-run-your-own-dns.html
> I wrote this article about why it can be important to run your own
> DNS server if you have a busy Spamassassin deployment.
okay one asked :)
1: do not add forwarders in global named.conf
if one do this the risk is that eg ones isp does not pay spamhaus or
simply blocking domains via hijacking, or even dnssec is gone
2: make sure your "root" hint xone file is less then 30 days old
this file can be fetched via ftp
3: check that port 53 is open both on tcp an udp
4: use nameserver 127.0.0.1 in resolv.conf
5: make progress to get ipv6 that will help to get more dns responses
from ipv6 only dns servers and also help spread loads on dns
last one can be ignored if you want to
Re: SpamTips.org: Why run your own DNS server?
Posted by Per Jessen <pe...@computer.org>.
Jeff Chan wrote:
> On Monday, July 4, 2011, 3:46:15 AM, Warren Jr. wrote:
>> Hey folks,
>
>>
http://www.spamtips.org/2011/07/spamassassin-why-run-your-own-dns.html
>> I wrote this article about why it can be important to run your own
>> DNS server if you have a busy Spamassassin deployment.
>
>> Anyone have any better tips of an alternate DNS resolver, or
>> configuration options to improve this suggested configuration?
>
>> http://www.spamtips.org/p/ultimate-setup-guide.html
>> Please see my Ultimate Setup Guide for all the latest tweaks to
>> maximize
>> your Spamassassin effectiveness and safety. Do you have any tips or
>> tricks that are not mentioned here?
>
> Hi Warren,
> I'd suggest adding something about using rbldnsd to serve lists
> locally. That's usually even faster than having your own
> resolver, and for many different reasons it's how medium to large
> systems should do things.
+1
/Per Jessen, Zürich
Re: SpamTips.org: Why run your own DNS server?
Posted by Jeff Chan <je...@surbl.org>.
On Monday, July 4, 2011, 3:46:15 AM, Warren Jr. wrote:
> Hey folks,
> http://www.spamtips.org/2011/07/spamassassin-why-run-your-own-dns.html
> I wrote this article about why it can be important to run your own DNS
> server if you have a busy Spamassassin deployment.
> Anyone have any better tips of an alternate DNS resolver, or
> configuration options to improve this suggested configuration?
> http://www.spamtips.org/p/ultimate-setup-guide.html
> Please see my Ultimate Setup Guide for all the latest tweaks to maximize
> your Spamassassin effectiveness and safety. Do you have any tips or
> tricks that are not mentioned here?
Hi Warren,
I'd suggest adding something about using rbldnsd to serve lists
locally. That's usually even faster than having your own
resolver, and for many different reasons it's how medium to large
systems should do things.
See:
http://www.surbl.org/surbl-nameserver-setup
and:
http://www.surbl.org/links#mirrors
Cheers,
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/