You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Spam Admin <sp...@sil.org> on 2008/03/13 15:29:21 UTC
Slow processing with 3.2.4
I have two mail servers running Spamassassin. One is running 3.1.9 and
the other 3.2.4, both with the same set of local rules, plus the
standard rules that come with each version.
The 'load' on the processors for 3.2.4 is about *4 times more *than the
'load' on 3.1.9.
Do others have the same problem? Is this a typical change between the
pre 3.2.x versions and the current version?
Dan Zachary
Re: Slow processing with 3.2.4
Posted by Spam Admin <sp...@sil.org>.
Details on configuration. Both machines, ma1 and ma2 are identical.
We are running:
sendmail
spamhaus.org blacklist which rejects the majority of mail so
SpamAssassin does not have to process that chunk of mail.
spamd (spamassassin) - 2 instances of spamd on each machine.
mimedefang
clamav
a few other milters
Dell 1750's 2.5G ram.
Dual Processor 2.4 GHz
Our graphs of memory usage do not show that we are using all of the RAM.
volume of mail is about 100K messages per day, per server, nearly
equally balanced.
About 50-60% are discarded by one or more of the various filters.
About 40-50% is blocked by spamhaus.org -- thus not processed by
SpamAssassin
About 10% is quarantined by Spam Assassin
The ma1 machine processors report a pretty steady CPU usage of about
80-100% running SA 3.1.9.
The ma2 machine processors report a pretty steady CPU usage of about
80-100% running SA 3.1.9, but when we upgrade ma2 to 3.2.4...
The ma2 machine processors report a pretty steady CPU usage of about
300-400% running SA 3.2.4. With compiled rules (sa-compile). Also the
similar performance with non-compiled rules.
Dan Zachary
>
>
>> - is spamd taking up the CPU time or is it your MTA queuing taking up
>> CPU time? 3.2.3+ will wait longer for DNS responses, decreasing
>> throughput per child but not increasing CPU time per message
>>
>> If you describe *how* you're using SA someone might have some more
>> suggestions.
>>
>> Daryl
>>
>
>
Re: Slow processing with 3.2.4
Posted by jp <jp...@saucer.midcoast.com>.
We have local DNS servers and cache/have feeds to some of the blacklists
to help with the network testing processing.
This is what we have observed too. We have watched top for observing
memory use, CPU use (user versus idle versus wait), and slow network
tests will cause the spamd childs to keep busy but not be using as much
user CPU time. Other causes of wait cpu time are I/O related as well,
such as having a slow hard disk (perhaps being used for swap).
For example a recent Cpanel problem with spamd was it was continuosly
trying to do something with a users' file while it processed, this extra
I/O was slowing things down. (My advice in this message is not regarding
cpanel servers though)
Basically get enough ram to run as many processes as it takes to max out
the CPUs with almost all "user" processing.
An older server running with 2GB ram hadn't enough simultaneous spamd
processes to process the volume of messages that we need. If you can
have a motherboard that uses DDR2 memory, it is dirt cheap these days
to fill it with lots of ram.
We upped ram to 8GB so we can run more simultaneous spamd processes than
previous servers, and that's just about the right amount of ram
currently for the volume a quad-core AMD machine can process and fully
utilize the CPUs.
>
> - is spamd taking up the CPU time or is it your MTA queuing taking up
> CPU time? 3.2.3+ will wait longer for DNS responses, decreasing
> throughput per child but not increasing CPU time per message
>
> If you describe *how* you're using SA someone might have some more
> suggestions.
>
> Daryl
--
/*
Jason Philbrook | Midcoast Internet Solutions - Wireless and DSL
KB1IOJ | Broadband Internet Access, Dialup, and Hosting
http://f64.nu/ | for Midcoast Maine http://www.midcoast.com/
*/
Re: Slow processing with 3.2.4
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
On 18/03/2008 1:39 PM, Spam Admin wrote:
> Spam Admin wrote:
>> Yes, the hardware is identical. The MX records are both '10', and the
>> volume of mail is slightly LESS on the 3.2.4 machine over the 3.1.9
>> it's taking more time to process less mail on the newer machine.
- are you running the same config on both (bayes, awl, sql, etc)
- what's your maximum message size?
3.2 will process large messages slower
- is spamd taking up the CPU time or is it your MTA queuing taking up
CPU time? 3.2.3+ will wait longer for DNS responses, decreasing
throughput per child but not increasing CPU time per message
If you describe *how* you're using SA someone might have some more
suggestions.
Daryl
Re: Slow processing with 3.2.4
Posted by Spam Admin <sp...@sil.org>.
I've had the 3.2.4 with sa-compiled rules running since Friday afternoon
on one server. The other server continues with 3.1.9.
The 3.2.4 seemed OK over the weekend with just slightly more cpu load,
but Monday the 3.2.4 is peaking at 400% (four processors all running
100%), and Tuesday it is running close to 400% most of the time.
SA 3.1.9:
SA 3.2.4 - with compiled rules
You can see the difference. If anything the ma2 server is running
slightly less volume of mail than ma2, yet the CPU usage is much higher.
If I run 3.1.9 on both machines, the graphs are nearly identical.
Dan Zachary
Spam Admin wrote:
> Yes, the hardware is identical. The MX records are both '10', and the
> volume of mail is slightly LESS on the 3.2.4 machine over the 3.1.9
> it's taking more time to process less mail on the newer machine.
>
> We have 2.5 Gig memory on each machine. Our graphs show that we are
> using about 1.5 Gig of this. Swap usage is the same on both machines
> at 500M each.
>
> Thanks for the suggestions. They are good factors to consider.
>
> Dan Zachary
>
>
> Kris Deugau wrote:
>> Spam Admin wrote:
>>> I have two mail servers running Spamassassin. One is running 3.1.9
>>> and the other 3.2.4, both with the same set of local rules, plus the
>>> standard rules that come with each version.
>>>
>>> The 'load' on the processors for 3.2.4 is about *4 times more *than
>>> the 'load' on 3.1.9.
>>
>> I'm assuming the hardware is something resembling identical?
>> Otherwise it's really hard to compare.
>>
>> -> CPU speed?
>> -> Memory?
>> -> SA child limit parameters?
>>
>> 3.2 sucks down more memory than 3.1; if the hardware is identical
>> but the machine running 3.1 shows a small swap usage, the 3.2 machine
>> is likely hitting swap a lot harder, causing your high load.
>>
>>> Do others have the same problem? Is this a typical change between
>>> the pre 3.2.x versions and the current version?
>>
>> I don't know about 4x from 3.1.x to 3.2.x, but I certainly avoided
>> upgrading from (patched-for-URI-RBLs) 2.64 for a LONG time because of
>> the memory load of 3.x.
>>
>> Of course, more rules means more CPU load, but the memory load was a
>> far larger problem for me; any more memory use on the old machine
>> would have pushed the system into swap, which would have *really*
>> killed performance...
>>
>> -kgd
Re: Slow processing with 3.2.4
Posted by Spam Admin <sp...@sil.org>.
Yes, the hardware is identical. The MX records are both '10', and the
volume of mail is slightly LESS on the 3.2.4 machine over the 3.1.9 it's
taking more time to process less mail on the newer machine.
We have 2.5 Gig memory on each machine. Our graphs show that we are
using about 1.5 Gig of this. Swap usage is the same on both machines at
500M each.
Thanks for the suggestions. They are good factors to consider.
Dan Zachary
Kris Deugau wrote:
> Spam Admin wrote:
>> I have two mail servers running Spamassassin. One is running 3.1.9
>> and the other 3.2.4, both with the same set of local rules, plus the
>> standard rules that come with each version.
>>
>> The 'load' on the processors for 3.2.4 is about *4 times more *than
>> the 'load' on 3.1.9.
>
> I'm assuming the hardware is something resembling identical?
> Otherwise it's really hard to compare.
>
> -> CPU speed?
> -> Memory?
> -> SA child limit parameters?
>
> 3.2 sucks down more memory than 3.1; if the hardware is identical but
> the machine running 3.1 shows a small swap usage, the 3.2 machine is
> likely hitting swap a lot harder, causing your high load.
>
>> Do others have the same problem? Is this a typical change between the
>> pre 3.2.x versions and the current version?
>
> I don't know about 4x from 3.1.x to 3.2.x, but I certainly avoided
> upgrading from (patched-for-URI-RBLs) 2.64 for a LONG time because of
> the memory load of 3.x.
>
> Of course, more rules means more CPU load, but the memory load was a
> far larger problem for me; any more memory use on the old machine
> would have pushed the system into swap, which would have *really*
> killed performance...
>
> -kgd
Re: Slow processing with 3.2.4
Posted by Kris Deugau <kd...@vianet.ca>.
Spam Admin wrote:
> I have two mail servers running Spamassassin. One is running 3.1.9 and
> the other 3.2.4, both with the same set of local rules, plus the
> standard rules that come with each version.
>
> The 'load' on the processors for 3.2.4 is about *4 times more *than the
> 'load' on 3.1.9.
I'm assuming the hardware is something resembling identical? Otherwise
it's really hard to compare.
-> CPU speed?
-> Memory?
-> SA child limit parameters?
3.2 sucks down more memory than 3.1; if the hardware is identical but
the machine running 3.1 shows a small swap usage, the 3.2 machine is
likely hitting swap a lot harder, causing your high load.
> Do others have the same problem? Is this a typical change between the
> pre 3.2.x versions and the current version?
I don't know about 4x from 3.1.x to 3.2.x, but I certainly avoided
upgrading from (patched-for-URI-RBLs) 2.64 for a LONG time because of
the memory load of 3.x.
Of course, more rules means more CPU load, but the memory load was a far
larger problem for me; any more memory use on the old machine would
have pushed the system into swap, which would have *really* killed
performance...
-kgd
Re: Slow processing with 3.2.4
Posted by Ralf Hildebrandt <Ra...@charite.de>.
* Spam Admin <sp...@sil.org>:
> I have two mail servers running Spamassassin. One is running 3.1.9 and
> the other 3.2.4, both with the same set of local rules, plus the standard
> rules that come with each version.
>
> The 'load' on the processors for 3.2.4 is about *4 times more *than the
> 'load' on 3.1.9.
Did you already compile the rules?
--
Ralf Hildebrandt (i.A. des IT-Zentrums) Ralf.Hildebrandt@charite.de
Charite - Universitätsmedizin Berlin Tel. +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-Berlin Fax. +49 (0)30-450 570-962
IT-Zentrum Standort CBF send no mail to snickebo@charite.de