Re: How to optimize SA?

[Justin Mason <jm...@jmason.org>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Jason Haar writes:
>On Tue, Mar 09, 2004 at 03:41:48PM -0600, Steve Yuroff wrote:
>> I've been running SA on a Cobalt Qube since 2.42 was current, and it's 
>> been a wonderful solution.  Recently I moved to running spamd -d to 
>> because of the increased load the increase in spam was putting on the 
>> server.  Now that's not cutting it- they're hitting me so hard I'm 
>> seeing processor loads in the 30's, and my poor little Qube is buckling 
>> under the load- I can't ssh in, email exchanges time out, ect.
>
>First off, disable the network checks and move any RBL checks you relied on
>SA for out onto your Internet-frontend SMTP server. That way you lose no
>functionality, but remove a bunch of "hangs" from SA - which  lead to
>loading issues.
>
>The DNS checks cannot be underestimated. We run Qmail-Scanner on all our
>e-mails - and it's a perl script. It typically processes an e-mail in <0.3
>secs, but calling spamc adds 2-8sec to it due to all the RBL lookups going
>on. 
>
>[I prefer running them via SA and will do so until the load gets too much -
>then I will take my own advice and move RBL back out to where it was
>designed for ;-)]

FWIW, though, SpamAssassin 2.6x has a very sophisticated DNSBL lookup
algorithm which (a) runs all queries in parallel and (b) aborts ones
that are taking significantly longer than the others, reducing problems
if one or two hosts go down.  So nowadays this should be *more* efficient
run from SpamAssassin than from the MTA.

- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFATkqKQTcbUG5Y7woRAjW6AKDxOA86Q4FhT+tWccGi0Dnhue2WTwCfdhSh
cK6u9F9PTiemsC5iUkTeCrk=
=t35S
-----END PGP SIGNATURE-----

Re: How to optimize SA?

[Daniel Quinlan <qu...@pathname.com>]

Justin Mason wrote:

>> FWIW, though, SpamAssassin 2.6x has a very sophisticated DNSBL lookup
>> algorithm which (a) runs all queries in parallel and (b) aborts ones
>> that are taking significantly longer than the others, reducing problems
>> if one or two hosts go down.  So nowadays this should be *more* efficient
>> run from SpamAssassin than from the MTA.

Jason Haar <Ja...@trimble.co.nz> writes:

> I don't doubt it. But that means spamc hangs around for up to 10 seconds
> awaiting a response on a system that runs "spamd -m XX" - i.e. XX max
> processes.

It's more like 3 seconds when some DNSBL are timing out, not 10.

Sometimes, the MX checking tests can be slow, but you can turn those off
independently of the DNSBL (RBL)tests.

> If you ran RBL checks out on the SMTP server, then you may be
> inefficient, but at least that 25K SMTP process is dealing with it
> rather than a 28M spamd process...

You realize that fork() in Linux (and BSD too, I'm sure) doesn't copy
every page?  28M virtual size != the actual additional memory used by a
spamd process.

Nevermind the fact that most DNSBLs aren't accurate enough as outright
reject/blocks.  SpamAssassin is much more accurate than MTA blacklists.

Daniel

-- 
Daniel Quinlan                     anti-spam (SpamAssassin), Linux,
http://www.pathname.com/~quinlan/    and open source consulting

Re: How to optimize SA?

[Jason Haar <Ja...@trimble.co.nz>]

On Tue, Mar 09, 2004 at 02:51:54PM -0800, Justin Mason wrote:
> FWIW, though, SpamAssassin 2.6x has a very sophisticated DNSBL lookup
> algorithm which (a) runs all queries in parallel and (b) aborts ones
> that are taking significantly longer than the others, reducing problems
> if one or two hosts go down.  So nowadays this should be *more* efficient
> run from SpamAssassin than from the MTA.

I don't doubt it. But that means spamc hangs around for up to 10 seconds
awaiting a response on a system that runs "spamd -m XX" - i.e. XX max
processes. If you ran RBL checks out on the SMTP server, then you may be
inefficient, but at least that 25K SMTP process is dealing with it rather
than a 28M spamd process...

-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +64 3 9635 377 Fax: +64 3 9635 417
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1