You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Justin Mason <jm...@jmason.org> on 2004/08/19 03:41:28 UTC
Re: General Comment - SA is processing messages slowly
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Jason Haar writes:
> On Tue, Aug 17, 2004 at 08:31:41AM -0400, Jeff Koch wrote:
> > I question your statement that these DNSRBL can handle the load. Our
> > mailservers are handling over 10K messages per hour - but to be
> > conservative assume there are a million SA boxes checking 1.0K messages per
> > hour. Is it reasonable to assume that each DNSRBL can handle a billion
> > queries an hour?
>
> We really need negative caching for DNS lookups. DNS TTLs are great for
> caching *successful* lookups - but failed lookups aren't cached.
>
> This is the problem with the RBL style. It has retro-fitted DNS to do a job
> it wasn't designed to do. Another example of a product with the same issues
> is the Squid proxy server. They designed negative DNS caching into Squid to
> reduce the amount of network DNS calls Squid makes.
>
> Has anyone looked into adding a DNS cache component into SA? You could cache
> both positive and negative lookups for (say) 5-10 minutes without really
> causing any bad side effects...
We were considering it, since it'd be doable now that we prefork and keep
a spamd process running for a few hundred messages. However, the other
devs were pretty sure that a local caching "named" process would probably
do the trick nicely enough. (me, I'm not quite convinced ;)
So a local caching named won't cache negative lookups? That *could*
be quite an improvement if that's the case...
- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Exmh CVS
iD8DBQFBJAVHQTcbUG5Y7woRAvkDAKDvwmXuClTigClrGYCP04DP7cFNYwCePXnP
Ne3UUyuQxArmYJ4D+AMa+is=
=k9To
-----END PGP SIGNATURE-----
Re: General Comment - SA is processing messages slowly
Posted by Bob Apthorpe <ap...@cynistar.net>.
On 18 Aug 2004 20:03:56 -0700 Daniel Quinlan <qu...@pathname.com> wrote:
> jm@jmason.org (Justin Mason) writes:
>
> > We were considering it, since it'd be doable now that we prefork and keep
> > a spamd process running for a few hundred messages. However, the other
> > devs were pretty sure that a local caching "named" process would probably
> > do the trick nicely enough. (me, I'm not quite convinced ;)
>
> 1. As others have stated and I'm sure you know, negative caching should
> work on a local DNS caching server.
> 2. Are you willing to write the code in the hopes it helps? Show
> me the benchmark and I'll follow along. :-)
Does perl's Memoize module help any? Dunno when this module made it into
perl core but I believe it's available in the 5.8.0 distribution and
onward. If it's that important, you can keep a persistent cache with
Memoize::AnyDBM_File, Memoize::Storable, etc. and expire with something
like Memoize::ExpireLRU, etc.
No benchmarks, just comments from the peanut gallery...
-- Bob
Re: General Comment - SA is processing messages slowly
Posted by Daniel Quinlan <qu...@pathname.com>.
jm@jmason.org (Justin Mason) writes:
> We were considering it, since it'd be doable now that we prefork and keep
> a spamd process running for a few hundred messages. However, the other
> devs were pretty sure that a local caching "named" process would probably
> do the trick nicely enough. (me, I'm not quite convinced ;)
1. As others have stated and I'm sure you know, negative caching should
work on a local DNS caching server.
2. Are you willing to write the code in the hopes it helps? Show
me the benchmark and I'll follow along. :-)
Daniel
--
Daniel Quinlan
http://www.pathname.com/~quinlan/
Re: General Comment - SA is processing messages slowly
Posted by Jeff Chan <je...@surbl.org>.
On Wednesday, August 18, 2004, 6:41:28 PM, Justin Mason wrote:
> Jason Haar writes:
>> We really need negative caching for DNS lookups. DNS TTLs are great for
>> caching *successful* lookups - but failed lookups aren't cached.
> We were considering it, since it'd be doable now that we prefork and keep
> a spamd process running for a few hundred messages. However, the other
> devs were pretty sure that a local caching "named" process would probably
> do the trick nicely enough. (me, I'm not quite convinced ;)
> So a local caching named won't cache negative lookups? That *could*
> be quite an improvement if that's the case...
Regular local name servers do cache negative answers AFIAK.
A repeated RBL NXDOMAIN response (occurring within the name
server's negative caching TTL) should already be served from
the cache of local name servers.
Jeff C.
--
Jeff Chan
mailto:jeffc@surbl.org
http://www.surbl.org/
Re: General Comment - SA is processing messages slowly
Posted by Jason Haar <Ja...@trimble.co.nz>.
On Thu, Aug 19, 2004 at 01:48:21PM +1200, Simon Byrnand wrote:
> Actually named *does* cache negative lookups, so the original poster that
> said failed lookups aren't cached was wrong...
Yeah - he's correct - or at least correct for other people's environments :-(
I've just checked. I have dnscache installed on our SA servers, and they
*ARE NOT* caching negative values. But they forward to our official
site-wide dnscache servers, and they *ARE* caching the negative values....
I've got to figure that one out :-(
--
Cheers
Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: General Comment - SA is processing messages slowly
Posted by Simon Byrnand <si...@igrin.co.nz>.
At 13:41 19/08/2004, Justin Mason wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>
>Jason Haar writes:
> > On Tue, Aug 17, 2004 at 08:31:41AM -0400, Jeff Koch wrote:
> > > I question your statement that these DNSRBL can handle the load. Our
> > > mailservers are handling over 10K messages per hour - but to be
> > > conservative assume there are a million SA boxes checking 1.0K
> messages per
> > > hour. Is it reasonable to assume that each DNSRBL can handle a billion
> > > queries an hour?
> >
> > We really need negative caching for DNS lookups. DNS TTLs are great for
> > caching *successful* lookups - but failed lookups aren't cached.
> >
> > This is the problem with the RBL style. It has retro-fitted DNS to do a job
> > it wasn't designed to do. Another example of a product with the same issues
> > is the Squid proxy server. They designed negative DNS caching into Squid to
> > reduce the amount of network DNS calls Squid makes.
> >
> > Has anyone looked into adding a DNS cache component into SA? You could
> cache
> > both positive and negative lookups for (say) 5-10 minutes without really
> > causing any bad side effects...
>
>We were considering it, since it'd be doable now that we prefork and keep
>a spamd process running for a few hundred messages. However, the other
>devs were pretty sure that a local caching "named" process would probably
>do the trick nicely enough. (me, I'm not quite convinced ;)
>
>So a local caching named won't cache negative lookups? That *could*
>be quite an improvement if that's the case...
Actually named *does* cache negative lookups, so the original poster that
said failed lookups aren't cached was wrong...
There was a little bit of discussion on the SURBL list just the other day
suggesting the possibility of making the negative TTL much shorter than the
normal TTL - so entires that are listed in surbl will cache for a longer
time (say a few hours) while entries that aren't currently listed wont be
negatively cached for too long, to make newly added entries usable sooner...
Regards,
Simon