You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Mark Rigby-Jones <ma...@rigby-jones.net> on 2007/12/09 18:20:50 UTC
spamd throughput issues
Hello everyone,
We've been using SpamAssassin on our front-end to provide anti-spam
services to our customers for some years now, but are trying to set
up some dedicated scanning boxes to take the load off the main SMTP
servers. Unfortunately, I'm struggling to get spamd to handle
anything like the throughput of which the servers should be cable.
Hardware configuration: 2x quad-core Xeon, 10 GB RAM
Software: CentOS 5, SpamAssassin 3.2.3
We're using the updates from updates.spamassassin.org and
saupdates.openprotect.com. SMTP server is exim with some customer
embedded perl which passes the messages through to spamd.
This combination manages a throughput of, at best, around 1 email
scanned per second. Initial debug runs flagged lockfiles for the
Bayes and AWL db files as a limiting factor - the former is now on
MySQL and the latter disabled and neither seem to be problematic.
Increasing the number of spamd child processes simply increases the
time taken by each to complete a scan. At no time does the server
show any signs of being CPU, IO or memory limited.
The sticking point that makes me suspect an issue with spamd itself
rather than the scanning process is this: With spamd set to 128 child
processes, all kept continuously busy, scan times (as reported by
spamd's logging) appear to average around 200 seconds. At the same
time, I can run emails manually through command-line spamassassin,
and they complete in around 15 seconds of wall-clock time.
Testing on a less heavily-loaded server suggests that scan times get
significantly longer when all the spamd child processes are busy
compared to when there are some idle. Again, this is from spamd's
logging, the times seen from exim which include the connection delay
while waiting fro a free child process are longer still.
I've checked Google and the mailing lists without much avail - if
anyone has seen similar issues in the past and can point me in the
direction of a solution, or flag anything obvious that might of
overlooked, it would be much appreciated!
mrj
--
Mark Rigby-Jones, System Operations Manager
CI-Net, Network House, Langford Locks, Kidlington, OX5 1GA
CI-Net is the trading name for Community Internet plc
A company registered in England and Wales number 3155758
t: 01865 856009 m: 07747 862201 e: mark.rigby-jones@ci-net.com
w: www.ci-net.com
Re: spamd throughput issues
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
> On 9 Dec 2007, at 21:40, Steven Stern wrote:
> >Have you tried running a local caching name server? That can cut
> >down on times to do repetitive name lookups.
On 09.12.07 21:58, Mark Rigby-Jones wrote:
> Yes indeed, it's something we've always had on mail servers even
> before we had SpamAssassin, for exactly that reason.
using local copies of blacklists SA uses would help too.
use rbldnsd for that, and you may get access to sorbs, njabl, rfc-ignorant
and surbl (usually) w/o any fee.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"Where do you want to go to die?" [Microsoft]
Re: spamd throughput issues
Posted by Philipp Snizek <ma...@belfin.ch>.
> Philipp Snizek wrote:
>> You use Bayes?
>> Have you tried turning off auto_expire? From my expierence this can
>> cause
>> significant performance issues.
>>
> It shouldn't cause performance issues. It should only cause, at worst,
> one message every 12 hours or so to take a long time (ie: 10 minutes).
>
> Unless you've got something that times out and winds up killing it
> mid-expire.. in which case you'll end up retrying that expire
> indefinitely.
Quite a number of emails experienced a 10-minute timeout at spamc because
of per-user bayes and timed out. They were forwarded unchecked to the
user's inbox. Back then I had the impression that the more often the spamc
children timed out the worse everything got.
By manually expiring single user accounts I could reduce the timeouts even
to zero withing 48 hours but it never could be considered as a solved
issue.
Running bayes expire manually on a per user basis was not a good idea.
> Also, if you do turn it off, you *MUST* run sa-learn --force-expire on a
> regular basis, preferably via a cronjob. If you've got a per-user bayes
> config, you *MUST* run this as every user. Otherwise your bayes DB's
> will grow without bound.
That's what I do every 24 hours (environment: 1700 users).
Also, I changed from per-user bayes to global bayes. Scanning performance
increased, bayes db is cleaned up and never larger than 300k tokens
(before on a bayes-per-user basis: 70 million tokens).
Spamd has been running very nicely since the changes.
- Philipp
>
>
>
Re: spamd throughput issues
Posted by Matt Kettler <mk...@verizon.net>.
Philipp Snizek wrote:
> You use Bayes?
> Have you tried turning off auto_expire? From my expierence this can cause
> significant performance issues.
>
It shouldn't cause performance issues. It should only cause, at worst,
one message every 12 hours or so to take a long time (ie: 10 minutes).
Unless you've got something that times out and winds up killing it
mid-expire.. in which case you'll end up retrying that expire indefinitely.
Also, if you do turn it off, you *MUST* run sa-learn --force-expire on a
regular basis, preferably via a cronjob. If you've got a per-user bayes
config, you *MUST* run this as every user. Otherwise your bayes DB's
will grow without bound.
Re: spamd throughput issues
Posted by UxBoD <ux...@splatnix.net>.
would perhaps be useful to see a spamassassin -D --lint ?
Regards,
--[ UxBoD ]--
// PGP Key: "curl -s https://www.splatnix.net/uxbod.asc | gpg --import"
// Fingerprint: C759 8F52 1D17 B3C5 5854 36BD 1FB1 B02F 5DB5 687B
// Keyserver: www.keyserver.net Key-ID: 0x5DB5687B
// Phone: +44 845 869 2749 SIP Phone: uxbod@sip.splatnix.net
----- Original Message -----
From: "Philipp Snizek" <ma...@belfin.ch>
To: "Mark Rigby-Jones" <ma...@ci-net.com>
Cc: users@spamassassin.apache.org
Sent: Monday, December 10, 2007 12:49:26 PM (GMT) Europe/London
Subject: Re: spamd throughput issues
You use Bayes?
Have you tried turning off auto_expire? From my expierence this can cause
significant performance issues.
Moreover, have you tried turning off bayes? without bayes scanning too a
quarter of a second per email on a 2cpu, 8GB standard i686 arch, sa
compiled as 32-bit app.
Philipp
> On 9 Dec 2007, at 21:40, Steven Stern wrote:
>> Have you tried running a local caching name server? That can cut
>> down on times to do repetitive name lookups.
>
>
> Yes indeed, it's something we've always had on mail servers even
> before we had SpamAssassin, for exactly that reason.
>
> Thanks,
> mrj
> --
> Mark Rigby-Jones, System Operations Manager
> CI-Net, Network House, Langford Locks, Kidlington, OX5 1GA
> CI-Net is the trading name for Community Internet plc
> A company registered in England and Wales number 3155758
> t: 01865 856009 m: 07747 862201 e: mark.rigby-jones@ci-net.com
> w: www.ci-net.com
>
>
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.
Re: spamd throughput issues
Posted by Philipp Snizek <ma...@belfin.ch>.
You use Bayes?
Have you tried turning off auto_expire? From my expierence this can cause
significant performance issues.
Moreover, have you tried turning off bayes? without bayes scanning too a
quarter of a second per email on a 2cpu, 8GB standard i686 arch, sa
compiled as 32-bit app.
Philipp
> On 9 Dec 2007, at 21:40, Steven Stern wrote:
>> Have you tried running a local caching name server? That can cut
>> down on times to do repetitive name lookups.
>
>
> Yes indeed, it's something we've always had on mail servers even
> before we had SpamAssassin, for exactly that reason.
>
> Thanks,
> mrj
> --
> Mark Rigby-Jones, System Operations Manager
> CI-Net, Network House, Langford Locks, Kidlington, OX5 1GA
> CI-Net is the trading name for Community Internet plc
> A company registered in England and Wales number 3155758
> t: 01865 856009 m: 07747 862201 e: mark.rigby-jones@ci-net.com
> w: www.ci-net.com
>
>
Re: spamd throughput issues
Posted by Rick Macdougall <ri...@ummm-beer.com>.
Mark Rigby-Jones wrote:
> On 9 Dec 2007, at 21:40, Steven Stern wrote:
>> Have you tried running a local caching name server? That can cut down
>> on times to do repetitive name lookups.
>
>
> Yes indeed, it's something we've always had on mail servers even before
> we had SpamAssassin, for exactly that reason.
>
> Thanks,
> mrj
Where is your mysql database for Bayes located ?
If the spamd is doing TCP connections over the network rather than a
local UNIX socket, we may have found your problem.
Regards,
Rick
Re: spamd throughput issues
Posted by Mark Rigby-Jones <ma...@ci-net.com>.
On 9 Dec 2007, at 21:40, Steven Stern wrote:
> Have you tried running a local caching name server? That can cut
> down on times to do repetitive name lookups.
Yes indeed, it's something we've always had on mail servers even
before we had SpamAssassin, for exactly that reason.
Thanks,
mrj
--
Mark Rigby-Jones, System Operations Manager
CI-Net, Network House, Langford Locks, Kidlington, OX5 1GA
CI-Net is the trading name for Community Internet plc
A company registered in England and Wales number 3155758
t: 01865 856009 m: 07747 862201 e: mark.rigby-jones@ci-net.com
w: www.ci-net.com
Re: spamd throughput issues
Posted by Steven Stern <su...@sterndata.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 12/09/2007 03:27 PM, Mark Rigby-Jones wrote:
> On 9 Dec 2007, at 21:03, Paweł Sasin wrote:
>> are you using network tests?
>> Try to evaluate spamd performance when run with the -L flag.
>
> We are running network tests. Disabling them helps somewhat, in that the
> emails which were already scanning relatively quickly do so even faster.
> However, once the number of child processes is increased, there are
> still a significant proportion which are taking several minutes to scan...
>
Have you tried running a local caching name server? That can cut down on
times to do repetitive name lookups.
- --
Steve
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org
iD8DBQFHXGDieERILVgMyvARAnlLAJkB8NDU7ZsYy6PhyXFAg8emyP5CDQCfc2Y7
cEwCMBwVGz4D+LnqqQlM2oA=
=9QIN
-----END PGP SIGNATURE-----
Re: spamd throughput issues
Posted by Mark Rigby-Jones <ma...@rigby-jones.net>.
On 9 Dec 2007, at 21:03, Paweł Sasin wrote:
> are you using network tests?
> Try to evaluate spamd performance when run with the -L flag.
We are running network tests. Disabling them helps somewhat, in that
the emails which were already scanning relatively quickly do so even
faster. However, once the number of child processes is increased,
there are still a significant proportion which are taking several
minutes to scan...
Thanks,
mrj
--
Mark Rigby-Jones, System Operations Manager
CI-Net, Network House, Langford Locks, Kidlington, OX5 1GA
CI-Net is the trading name for Community Internet plc
A company registered in England and Wales number 3155758
t: 01865 856009 m: 07747 862201 e: mark.rigby-jones@ci-net.com
w: www.ci-net.com
Re: spamd throughput issues
Posted by Paweł Sasin <ha...@wp-sa.pl>.
Hi,
are you using network tests?
Try to evaluate spamd performance when run with the -L flag.
--
Pawel Sasin
WIRTUALNA POLSKA SA, ul. Traugutta 115c, 80-226 Gdansk; NIP: 957-07-51-216;
Sad Rejonowy Gdansk-Polnoc KRS 0000068548, kapital zakladowy 62.880.024 zlotych (w calosci wplacony)