You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Steve Dimoff <sd...@usnetworksinc.com> on 2004/07/29 16:06:58 UTC
HELP! Performance Issues with Spamassassin
Hello,
I'm using SA 2.63 with qmail, Qmail-Scanner 1.22 and clamAV 0.72
System Info: RH 7.3
CPU:
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Pentium(R) 4 CPU 2.40GHz
stepping : 9
cpu MHz : 2400.186
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips : 4784.12
Memory:
Mem: 514324K av, 426548K used, 87776K free,
Swap: 1044216K av, 70680K used, 973536K free
We process the following messages usually a day:
Statistics by Hour
----------------------------------------------------
Hour Spam Ham
------------- ----------------- --------------
2004-07-28 00 1595 ( 87%) 234 ( 12%)
2004-07-28 01 1818 ( 83%) 356 ( 16%)
2004-07-28 02 1279 ( 79%) 326 ( 20%)
2004-07-28 03 1693 ( 88%) 230 ( 11%)
2004-07-28 04 1840 ( 80%) 455 ( 19%)
2004-07-28 05 1394 ( 84%) 264 ( 15%)
2004-07-28 06 1591 ( 87%) 236 ( 12%)
2004-07-28 07 2136 ( 88%) 284 ( 11%)
2004-07-28 08 1541 ( 71%) 612 ( 28%)
2004-07-28 09 1507 ( 57%) 1109 ( 42%)
2004-07-28 10 1611 ( 70%) 673 ( 29%)
2004-07-28 11 581 ( 68%) 272 ( 31%)
2004-07-28 12 1343 ( 66%) 685 ( 33%)
2004-07-28 13 2106 ( 68%) 962 ( 31%)
2004-07-28 14 1644 ( 73%) 596 ( 26%)
2004-07-28 15 1411 ( 61%) 898 ( 38%)
2004-07-28 16 1820 ( 72%) 693 ( 27%)
2004-07-28 17 1411 ( 75%) 464 ( 24%)
2004-07-28 18 1491 ( 81%) 342 ( 18%)
2004-07-28 19 1296 ( 73%) 457 ( 26%)
2004-07-28 20 1116 ( 75%) 372 ( 25%)
2004-07-28 21 1461 ( 77%) 426 ( 22%)
2004-07-28 22 1286 ( 78%) 345 ( 21%)
2004-07-28 23 1325 ( 79%) 344 ( 20%)
Around the times where we get flooded and receive around 2000+ messages an
hour the system goes into a meltdown.
SPAMD is taking up most of the processes. The system stops accepting new
mail connections. Once it's processed what's in Qmail's connection then it
starts to accept until it's full again.
Here is my RUN scipt for qmail
#!/bin/sh
QMAILDUID=`id -u qmaild`
NOFILESGID=`id -g qmaild`
# when QMAILQUEUE is set, all mail will be sent to the nominated script
QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl" export QMAILQUEUE
exec /usr/local/bin/softlimit -m 30000000 \
/usr/local/bin/tcpserver \
-H -l systemname_removed \
-v -x /etc/tcp.smtp.cdb \
-c 35 -R -u "$QMAILDUID" -g "$NOFILESGID" 0 smtp \
/usr/local/bin/fixcrio \
/usr/local/bin/rblsmtpd -b -C \
-r 'relays.ordb.org:Your message was rejected because the mail server you
use is configured to allow OPEN RELAY - More detailed in
formation regarding this problem is available from
http://www.ordb.org/lookup/?%IP% - Please forward this error through to your
emai
l server support staff for easy resolution.' \
-t 5 \
/var/qmail/bin/qmail-smtpd 2>&1
When this flood comes it starts taking longer to process the mail.
630.7 seconds is the highest I've seen, I'm sure there is a higher number in
there though.
We do have 2 systems setup. The 2nd is a Sun E250 2x400 with 2 GIG mem. The
same thing happens to this system when we receive a "SPAM Flood".
Is there anything I can check or change to make it process faster when a lot
of mail is hitting the server?
Thanks!
Steve
Re: HELP! Performance Issues with Spamassassin
Posted by "Scot L. Harris" <we...@cfl.rr.com>.
On Thu, 2004-07-29 at 10:06, Steve Dimoff wrote:
> We process the following messages usually a day:
>
> Statistics by Hour
> ----------------------------------------------------
> Hour Spam Ham
> ------------- ----------------- --------------
> 2004-07-28 00 1595 ( 87%) 234 ( 12%)
> 2004-07-28 01 1818 ( 83%) 356 ( 16%)
> 2004-07-28 02 1279 ( 79%) 326 ( 20%)
> 2004-07-28 03 1693 ( 88%) 230 ( 11%)
> 2004-07-28 04 1840 ( 80%) 455 ( 19%)
> 2004-07-28 05 1394 ( 84%) 264 ( 15%)
> 2004-07-28 06 1591 ( 87%) 236 ( 12%)
> 2004-07-28 07 2136 ( 88%) 284 ( 11%)
> 2004-07-28 08 1541 ( 71%) 612 ( 28%)
> 2004-07-28 09 1507 ( 57%) 1109 ( 42%)
> 2004-07-28 10 1611 ( 70%) 673 ( 29%)
> 2004-07-28 11 581 ( 68%) 272 ( 31%)
> 2004-07-28 12 1343 ( 66%) 685 ( 33%)
> 2004-07-28 13 2106 ( 68%) 962 ( 31%)
> 2004-07-28 14 1644 ( 73%) 596 ( 26%)
> 2004-07-28 15 1411 ( 61%) 898 ( 38%)
> 2004-07-28 16 1820 ( 72%) 693 ( 27%)
> 2004-07-28 17 1411 ( 75%) 464 ( 24%)
> 2004-07-28 18 1491 ( 81%) 342 ( 18%)
> 2004-07-28 19 1296 ( 73%) 457 ( 26%)
> 2004-07-28 20 1116 ( 75%) 372 ( 25%)
> 2004-07-28 21 1461 ( 77%) 426 ( 22%)
> 2004-07-28 22 1286 ( 78%) 345 ( 21%)
> 2004-07-28 23 1325 ( 79%) 344 ( 20%)
>
> Around the times where we get flooded and receive around 2000+ messages an
> hour the system goes into a meltdown.
>
> SPAMD is taking up most of the processes. The system stops accepting new
> mail connections. Once it's processed what's in Qmail's connection then it
> starts to accept until it's full again.
One option you should consider is implementing greylisting. Not sure
what is available to use with Qmail. I used milter-greylist with my
sendmail configuration and have managed to knock spam down to literally
a handful a day. With your volume you can probably knock the spam you
process with spamassassin down to a 100 or so per day.
And I have found that setting a relatively small timeout (a couple of
minutes) works just fine. I have seen virtually no difference to that
or using 30 minutes.
The combination of greylisting and spamassassin has worked wonders.
--
Scot L. Harris
webid@cfl.rr.com
I'll meet you... on the dark side of the moon...
-- Pink Floyd
Re: HELP! Performance Issues with Spamassassin
Posted by Martin Hepworth <ma...@solid-state-logic.com>.
Steve
what rules are u running over the 'standard' 2.63 ??
--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
Steve Dimoff wrote:
> Hello,
>
> I'm using SA 2.63 with qmail, Qmail-Scanner 1.22 and clamAV 0.72
>
> System Info: RH 7.3
>
> CPU:
>
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 15
> model : 2
> model name : Intel(R) Pentium(R) 4 CPU 2.40GHz
> stepping : 9
> cpu MHz : 2400.186
> cache size : 512 KB
> fdiv_bug : no
> hlt_bug : no
> f00f_bug : no
> coma_bug : no
> fpu : yes
> fpu_exception : yes
> cpuid level : 2
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
> bogomips : 4784.12
>
>
> Memory:
> Mem: 514324K av, 426548K used, 87776K free,
> Swap: 1044216K av, 70680K used, 973536K free
>
> We process the following messages usually a day:
>
> Statistics by Hour
> ----------------------------------------------------
> Hour Spam Ham
> ------------- ----------------- --------------
> 2004-07-28 00 1595 ( 87%) 234 ( 12%)
> 2004-07-28 01 1818 ( 83%) 356 ( 16%)
> 2004-07-28 02 1279 ( 79%) 326 ( 20%)
> 2004-07-28 03 1693 ( 88%) 230 ( 11%)
> 2004-07-28 04 1840 ( 80%) 455 ( 19%)
> 2004-07-28 05 1394 ( 84%) 264 ( 15%)
> 2004-07-28 06 1591 ( 87%) 236 ( 12%)
> 2004-07-28 07 2136 ( 88%) 284 ( 11%)
> 2004-07-28 08 1541 ( 71%) 612 ( 28%)
> 2004-07-28 09 1507 ( 57%) 1109 ( 42%)
> 2004-07-28 10 1611 ( 70%) 673 ( 29%)
> 2004-07-28 11 581 ( 68%) 272 ( 31%)
> 2004-07-28 12 1343 ( 66%) 685 ( 33%)
> 2004-07-28 13 2106 ( 68%) 962 ( 31%)
> 2004-07-28 14 1644 ( 73%) 596 ( 26%)
> 2004-07-28 15 1411 ( 61%) 898 ( 38%)
> 2004-07-28 16 1820 ( 72%) 693 ( 27%)
> 2004-07-28 17 1411 ( 75%) 464 ( 24%)
> 2004-07-28 18 1491 ( 81%) 342 ( 18%)
> 2004-07-28 19 1296 ( 73%) 457 ( 26%)
> 2004-07-28 20 1116 ( 75%) 372 ( 25%)
> 2004-07-28 21 1461 ( 77%) 426 ( 22%)
> 2004-07-28 22 1286 ( 78%) 345 ( 21%)
> 2004-07-28 23 1325 ( 79%) 344 ( 20%)
>
> Around the times where we get flooded and receive around 2000+ messages an
> hour the system goes into a meltdown.
>
> SPAMD is taking up most of the processes. The system stops accepting new
> mail connections. Once it's processed what's in Qmail's connection then it
> starts to accept until it's full again.
>
> Here is my RUN scipt for qmail
>
> #!/bin/sh
> QMAILDUID=`id -u qmaild`
> NOFILESGID=`id -g qmaild`
> # when QMAILQUEUE is set, all mail will be sent to the nominated script
> QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl" export QMAILQUEUE
>
> exec /usr/local/bin/softlimit -m 30000000 \
> /usr/local/bin/tcpserver \
> -H -l systemname_removed \
> -v -x /etc/tcp.smtp.cdb \
> -c 35 -R -u "$QMAILDUID" -g "$NOFILESGID" 0 smtp \
> /usr/local/bin/fixcrio \
> /usr/local/bin/rblsmtpd -b -C \
> -r 'relays.ordb.org:Your message was rejected because the mail server you
> use is configured to allow OPEN RELAY - More detailed in
> formation regarding this problem is available from
> http://www.ordb.org/lookup/?%IP% - Please forward this error through to your
> emai
> l server support staff for easy resolution.' \
> -t 5 \
> /var/qmail/bin/qmail-smtpd 2>&1
>
>
> When this flood comes it starts taking longer to process the mail.
>
> 630.7 seconds is the highest I've seen, I'm sure there is a higher number in
> there though.
>
> We do have 2 systems setup. The 2nd is a Sun E250 2x400 with 2 GIG mem. The
> same thing happens to this system when we receive a "SPAM Flood".
>
> Is there anything I can check or change to make it process faster when a lot
> of mail is hitting the server?
>
> Thanks!
> Steve
>
>
**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.
**********************************************************************
Re: HELP! Performance Issues with Spamassassin
Posted by Matt Kettler <mk...@evi-inc.com>.
At 10:06 AM 7/29/2004, Steve Dimoff wrote:
>Hello,
>
> I'm using SA 2.63 with qmail, Qmail-Scanner 1.22 and clamAV 0.72
<snip>
>Around the times where we get flooded and receive around 2000+ messages an
>hour the system goes into a meltdown.
>
>SPAMD is taking up most of the processes. The system stops accepting new
>mail connections. Once it's processed what's in Qmail's connection then it
>starts to accept until it's full again.
>
<snip>
>same thing happens to this system when we receive a "SPAM Flood".
>
>Is there anything I can check or change to make it process faster when a lot
>of mail is hitting the server?
Be sure to start spamd with the -m parameter. This will at least try to
limit the number of child processes spamd can fork. If you can keep spamd
from forking uncontrollably, you can prevent the system from running into
the swap partition and grinding to a halt. Ideally you want to use large
enough number of processes to keep the server fully loaded, but small
enough to prevent driving it into the swapfile where it becomes inefficient.
You've got 512mb of ram, and a spamd process on my setup is about
30mb. Giving yourself some headroom for other processes I'd suggest trying
a limit between 10 and 15 for starters.
Also note I've heard several people mention that -m did not work for them,
however one other person reported it worked, but only if he daemonized
spamd with the -d parameter. Thus, I'd also suggest daemonizing spamd.