You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Dermot Paikkos <de...@sciencephoto.com> on 2006/05/16 14:04:49 UTC
Performance
Hi
Spamassassin 3.02 running from SA-Exim (exim 4.5).
OPTIONS="--nouser-config --max-children 6 --helper-home-
dir=/var/spool/spamassassin/ -s /var/log/spamd.log --username=nobody"
I recently went live with the above system and am noticing some very
heavy memory usage. Each spamd is using near or over 200MB. The
machine (Debian Alpha 1.2GB RAM) soon eat up all the swap space
(4GB). To stablise the system for now I have added more swap and
reduced the number of SA children ( was 10) and exim processes.
Are there any performance tweaks I can make?
I can't see what all my memory is going and I suspect that it's not
being recycled properly.
Any thoughts?
TIA.
Dp.
Re: Performance
Posted by Marc Perkel <ma...@perkel.com>.
Dermot Paikkos wrote:
> Hi
>
> Spamassassin 3.02 running from SA-Exim (exim 4.5).
>
> OPTIONS="--nouser-config --max-children 6 --helper-home-
> dir=/var/spool/spamassassin/ -s /var/log/spamd.log --username=nobody"
>
> I recently went live with the above system and am noticing some very
> heavy memory usage. Each spamd is using near or over 200MB. The
> machine (Debian Alpha 1.2GB RAM) soon eat up all the swap space
> (4GB). To stablise the system for now I have added more swap and
> reduced the number of SA children ( was 10) and exim processes.
>
> Are there any performance tweaks I can make?
>
> I can't see what all my memory is going and I suspect that it's not
> being recycled properly.
>
> Any thoughts?
> TIA.
> Dp.
>
>
Faster processors and lots of ram always help. Once you start hitting
swap you slow down. The more you slow down the more it backs up, the
more it backs up the more it slows down. So - first and foremost - buy
more ram.
Also - you can offload things to different computers. Adding a separare
computer running MySQL for the bays database helps a lot. Use a
dedicated SA computer and have it hand off the good email to a separate
computer running your POP and IMAP server. More computers, more memory,
faster processors.
Also - if you are running Exim or some other advanced MTA there are
tricks you can do at connect time to reject email BEFORE SA sees it and
therefore reduce the number of messages going into SA.
Re: Performance
Posted by Dermot Paikkos <de...@sciencephoto.com>.
On 16 May 2006 at 16:25, Justin Mason wrote:
>
> check to ensure your SA-Exim checks are conditional on a message size
> check; at least in 2005, it didn't use the recommended size limits by
> default for some reason, which meant it allowed spamd to balloon out
> of control. Maybe that is still the case. see this thread:
>
> http://www.exim.org/mail-archives/exim-users/Week-of-Mon-20050221/msg0
> 0218.html
It doesn't scan messages over 250KB.
# How much of the body we feed to spamassassin (in bytes)
# Default is 250KB
SAmaxbody: 256000
# Default is 0: do not scan messages that are too big
# (note that this is parsed as a condition)
SATruncBodyCond: 0
======= from the mail log ====
SA: Action: check skipped due to message size (5476972 bytes) and
SATruncBodyCond expanded to false
I must admit that things have cooled down a bit since I 1) turned
SAEximDebug off and 2) reduced the number of spamd children from 10
to 6.
I also took the ridiculous step of dedicating an entire 36GB disk to
swap as the system looked like it was about to fall over. The amount
of swapping has fallen although not to a level I would like.
Mem: 1289944k total, 914336k used, 375608k free, 31680k buffers
Swap: 35993400k total, 253848k used, 35739552k free, 35624k cached
There is this in the ANNOUNCE notes for SpamAssassin 3.1:
"Apache preforking algorithm adopted; number of spamd child processes
is now scaled, according to demand. This provides better VM
behaviour when not under peak load."
Which indicates a performance enhancement in 3.1 over 3.0. Perhaps an
upgrade would help.
Thanx.
Dp.
> Dermot Paikkos writes:
> > On 16 May 2006 at 10:07, Matt Kettler wrote:
> > > Dermot Paikkos wrote:
> > > > Hi
> > > >
> > > > Spamassassin 3.02 running from SA-Exim (exim 4.5).
> > > >
> > > > OPTIONS="--nouser-config --max-children 6 --helper-home-
> > > > dir=/var/spool/spamassassin/ -s /var/log/spamd.log
> > > > --username=nobody"
> > > >
> > > > I recently went live with the above system and am noticing some
> > > > very heavy memory usage. Each spamd is using near or over 200MB.
> > > That's highly unusual.. Did you download any add-on rulesets? In
> > > particular, did you add sa-blacklist? If so, remove it.
> > >
> > I agree - I am no expert - but this seems high.
> >
> > ========= top output ========
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> > 16099 nobody 18 0 202m 171m 59m R 85.8 13.6 7:30.15 spamd
> > 16098 nobody 12 0 245m 198m 39m S 11.2 15.8 8:00.58 spamd
> > 32020 root 18 0 1664 1664 1296 R 2.2 0.1 0:02.05 top
> > 50 root 12 0 0 0 0 S 0.0 0.0 0:48.54 kjournald
> > 15818 root 12 0 2696 2216 2176 S 0.0 0.2 0:03.55 sshd
> > ==========================
> >
> > On the matter of more computers; between 06:30 and 15:47 today we
> > have received 2040 mail and delivered 2407. Today has been
> > exceptional as there is a backlog of mail from the server upgrade. I
> > am not sure what sort of bracket that puts us in but I wouldn't say
> > we're a big site in terms of number emails. What's more we don't
> > have the budget or resources to build a server farm for mail and I
> > don't think it's really necessary. I think the problem is either in
> > my configuration or with my version of SA.
> >
> > Autolearn isn't on. I would have thought that would have given me a
> > performance boost.
> >
> > The config does have several blacklists by default. I haven't added
> > any. For example:
> >
> > # the black list. See /usr/share/doc/exim4-config/default_acl for
> > details.
> > deny
> > message = sender envelope address $sender_address is locally
> > blacklisted here. If you think this is wrong, get in touch with
> > postmaster
> > !acl = acl_whitelist_local_deny
> > senders = ${if exists{CONFDIR/local_sender_blacklist}\
> > {CONFDIR/local_sender_blacklist}\
> > {}}
> >
> > ditto for hosts. I do have a local_hosts_blacklist which has 1118
> > lines in. Is this what you mean by remove the sa-blacklist?
> >
> > I had SA 3.0 running on a test system and I noticed this behaviour
> > before. It seemed to go away with V3.1. Problem is there isn't a
> > version of 3.1 in the stable distro of Debian just yet and I don't
> > want to use un-tested stuff on a production system.
> >
> > Thanx.
> > Dp.
> >
> >
> > These are some custom rulesets I have installed.
> > 70_sare_bayes_poison_nxm.cf
> > 70_sare_evilnum0.cf
> > 70_sare_evilnum1.cf
> > 70_sare_evilnum2.cf
> > 70_sare_header0.cf
> > 70_sare_header1.cf
> > 70_sare_header2.cf
> > 70_sare_header3.cf
> > 70_sare_html.cf
> > 70_sare_obfu0.cf
> > 70_sare_obfu1.cf
> > 70_sare_oem.cf
> > 70_sare_random.cf
> > 70_sare_specific.cf
> > 70_sare_unsub.cf
> > 70_sare_uri0.cf
> > 72_sare_redirect_post3.0.0.cf
> > 99_FVGT_Tripwire.cf
> > 99_sare_fraud_post25x.cf
> > anti_bayes.cf
> > bogus-virus-warnings.cf
Re: Performance
Posted by Dermot Paikkos <de...@sciencephoto.com>.
On 16 May 2006 at 10:07, Matt Kettler wrote:
> Dermot Paikkos wrote:
> > Hi
> >
> > Spamassassin 3.02 running from SA-Exim (exim 4.5).
> >
> > OPTIONS="--nouser-config --max-children 6 --helper-home-
> > dir=/var/spool/spamassassin/ -s /var/log/spamd.log
> > --username=nobody"
> >
> > I recently went live with the above system and am noticing some very
> > heavy memory usage. Each spamd is using near or over 200MB.
> That's highly unusual.. Did you download any add-on rulesets? In
> particular, did you add sa-blacklist? If so, remove it.
>
I agree - I am no expert - but this seems high.
========= top output ========
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16099 nobody 18 0 202m 171m 59m R 85.8 13.6 7:30.15 spamd
16098 nobody 12 0 245m 198m 39m S 11.2 15.8 8:00.58 spamd
32020 root 18 0 1664 1664 1296 R 2.2 0.1 0:02.05 top
50 root 12 0 0 0 0 S 0.0 0.0 0:48.54 kjournald
15818 root 12 0 2696 2216 2176 S 0.0 0.2 0:03.55 sshd
==========================
On the matter of more computers; between 06:30 and 15:47 today we
have received 2040 mail and delivered 2407. Today has been
exceptional as there is a backlog of mail from the server upgrade. I
am not sure what sort of bracket that puts us in but I wouldn't say
we're a big site in terms of number emails. What's more we don't have
the budget or resources to build a server farm for mail and I don't
think it's really necessary. I think the problem is either in my
configuration or with my version of SA.
Autolearn isn't on. I would have thought that would have given me a
performance boost.
The config does have several blacklists by default. I haven't added
any. For example:
# the black list. See /usr/share/doc/exim4-config/default_acl for
details.
deny
message = sender envelope address $sender_address is locally
blacklisted here. If you think this is wrong, get in touch with
postmaster
!acl = acl_whitelist_local_deny
senders = ${if exists{CONFDIR/local_sender_blacklist}\
{CONFDIR/local_sender_blacklist}\
{}}
ditto for hosts. I do have a local_hosts_blacklist which has 1118
lines in. Is this what you mean by remove the sa-blacklist?
I had SA 3.0 running on a test system and I noticed this behaviour
before. It seemed to go away with V3.1. Problem is there isn't a
version of 3.1 in the stable distro of Debian just yet and I don't
want to use un-tested stuff on a production system.
Thanx.
Dp.
These are some custom rulesets I have installed.
70_sare_bayes_poison_nxm.cf
70_sare_evilnum0.cf
70_sare_evilnum1.cf
70_sare_evilnum2.cf
70_sare_header0.cf
70_sare_header1.cf
70_sare_header2.cf
70_sare_header3.cf
70_sare_html.cf
70_sare_obfu0.cf
70_sare_obfu1.cf
70_sare_oem.cf
70_sare_random.cf
70_sare_specific.cf
70_sare_unsub.cf
70_sare_uri0.cf
72_sare_redirect_post3.0.0.cf
99_FVGT_Tripwire.cf
99_sare_fraud_post25x.cf
anti_bayes.cf
bogus-virus-warnings.cf
Re: Performance
Posted by Matt Kettler <mk...@comcast.net>.
Dermot Paikkos wrote:
> Hi
>
> Spamassassin 3.02 running from SA-Exim (exim 4.5).
>
> OPTIONS="--nouser-config --max-children 6 --helper-home-
> dir=/var/spool/spamassassin/ -s /var/log/spamd.log --username=nobody"
>
> I recently went live with the above system and am noticing some very
> heavy memory usage. Each spamd is using near or over 200MB.
That's highly unusual.. Did you download any add-on rulesets? In
particular, did you add sa-blacklist? If so, remove it.