You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Piers Kittel <pi...@bda.org.uk> on 2005/05/06 14:51:13 UTC
Abnormally high CPU usage when spamd is running
Hello all,
Have set up 5 email servers in various locations across the UK and they
all are connected to each other. They all are using Debian Sarge. As I
wanted spam detection, I installed spamassassin using the official
debian packages and following
http://www.clues.ltd.uk/howto/debian-sa-fprot-HOWTO.txt 's section on
spamassassin, I set up spamassassin (but not fprot) and continiously
found that after a few hours, I suddenly experience very high load
averages, and extremely slow server performance for everything else, but
if I restart spamassassin, the server works fine again, but it would
start getting high load averages again later on and so on. So I
disabled spamassassin on all servers, and tried a different method on
only one server, and used sa-exim
http://marc.merlins.org/linux/exim/sa.html - I just did an apt-get
install sa-exim, made the necessary changes to make spamassassin work,
and I got the same problem back again. While I realise spamassassin is
very load intensive, but we are talking about a Compaq Proliant P3 1GHz
server, around 350MB RAM, and maybe only 200-300 emails a hour.
I have an email server using exim and spamassassin myself at home, and I
decided to do an "apt-get update && apt-get upgrade" which upgraded
spamassassin, and now I got the same problem msyelf at home, where it
was working just fine with spamassassin for the past 1 1/2 years?
Here's a snippet from top when the server is crunching through spamassassin:
top - 13:07:59 up 51 days, 36 min, 2 users, load average: 7.16, 7.38, 6.28
Tasks: 116 total, 1 running, 113 sleeping, 0 stopped, 2 zombie
Cpu(s): 1.9% us, 1.9% sy, 0.0% ni, 0.0% id, 96.2% wa, 0.0% hi, 0.0% si
And after disabling spamassassin, I get:
top - 13:37:07 up 51 days, 1:06, 2 users, load average: 0.07, 0.04, 0.94
Tasks: 96 total, 1 running, 95 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.5% us, 0.0% sy, 0.0% ni, 98.0% id, 1.5% wa, 0.0% hi, 0.0% si
Can I ask if this is normal, or am I doing something wrong somewhere?
Spamassassin is version 3.0.2-1, and the server I'm working on has a
kernel version of 2.6.8, debian prepackaged version 2.6.8-2-686 to be exact.
Thanks very much for your help in advance
Regards - Piers
Re: blocking Asian IPs?
Posted by Martyn Drake <ma...@drake.org.uk>.
You might want to take a look at this:
http://www.blackholes.us/
Very useful for inclusion into RBLs if you so desire. I myself am not
very keen at all on blocking entire countries, but the option is there
if you need/want it.
Regards,
Martyn
Andy Spiegl wrote:
> Hi Carlo,
>
> back in May you wrote:
>
>> Moreover, you might want to firewall (or reject their mail
>> otherwise before it reaches spamassassin) all of South Korea and
>> all of China -- that will reduce the ammount of spam you
>> receive with about 99% ... So, it is more than worth it.
>
>
> When I read this I thought it's overkill but in the meantime and after
> looking at my logs (not only from SA but also ssh-attacks) I DO think that
> it's a good idea to block these IPs.
>
> Is the list of IPs you had included deducted from you logfiles or did you
> find a somewhat official list of networks in Asia? Do you maybe already
> have an updated list? There is one Korean IP that's bothering me
> incredibly these days with hundreds of thousands of ssh-connections:
> 220.65.232.100
> But I didn't find it on your list although it's Korean. So maybe I just
> misunderstood you.
>
> Thanks in advance,
> Andy.
>
--
Martyn Drake
http://www.drake.org.uk
http://www.drake.org.uk/hosting
http://www.ourlittleduckling.com
Re: blocking Asian IPs?
Posted by Jim Knuth <jk...@jkart.de>.
Hallo und guten Tag Andy,
Heute (am 24.06.2005 - 15:38 Uhr)
schriebst Du:
> When I read this I thought it's overkill but in the meantime and after
> looking at my logs (not only from SA but also ssh-attacks) I DO think that
> it's a good idea to block these IPs.
take a look at http://www.pettingers.org/code/SSHBlack.html
I use this and I`m very happy. ;)
--
Viele Grüße, Kind regards,
Jim Knuth
jk@jkart.de
ICQ #277289867
PGP Fingerprint:
54C9 1A46 D3B2 95B6 454D
74FA AC73 773E 1F78 066F
----------
Zufalls-Zitat
----------
Das beliebteste Haustier der Deutschen ist und bleibt das
halbe Hähnchen. (Lutz Hager Amherst)
----------
Der Text hat nichts mit dem Empfänger der Mail zu tun
----------
Virus free. Checked by NOD32 Version 1.1152 Update 23.06.2005
blocking Asian IPs?
Posted by Andy Spiegl <sp...@spiegl.de>.
Hi Carlo,
back in May you wrote:
> Moreover, you might want to firewall (or reject their mail
> otherwise before it reaches spamassassin) all of South Korea and
> all of China -- that will reduce the ammount of spam you
> receive with about 99% ... So, it is more than worth it.
When I read this I thought it's overkill but in the meantime and after
looking at my logs (not only from SA but also ssh-attacks) I DO think that
it's a good idea to block these IPs.
Is the list of IPs you had included deducted from you logfiles or did you
find a somewhat official list of networks in Asia? Do you maybe already
have an updated list? There is one Korean IP that's bothering me
incredibly these days with hundreds of thousands of ssh-connections:
220.65.232.100
But I didn't find it on your list although it's Korean. So maybe I just
misunderstood you.
Thanks in advance,
Andy.
--
o _ _ _
------- __o __o /\_ _ \\o (_)\__/o (_) -o)
----- _`\<,_ _`\<,_ _>(_) (_)/<_ \_| \ _|/' \/ /\\
---- (_)/ (_) (_)/ (_) (_) (_) (_) (_)' _\o_ _\_v
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Great spirits have always encountered violent opposition from
mediocre minds. - Albert Einstein
Re: Abnormally high CPU usage when spamd is running
Posted by Carlo Wood <ca...@alinoe.com>.
On Fri, May 06, 2005 at 01:51:13PM +0100, Piers Kittel wrote:
> found that after a few hours, I suddenly experience very high load
> averages, and extremely slow server performance for everything else, but
> if I restart spamassassin, the server works fine again, but it would
> start getting high load averages again later on and so on.
[...]
> Here's a snippet from top when the server is crunching through spamassassin:
>
> top - 13:07:59 up 51 days, 36 min, 2 users, load average: 7.16, 7.38, 6.28
> Tasks: 116 total, 1 running, 113 sleeping, 0 stopped, 2 zombie
> Cpu(s): 1.9% us, 1.9% sy, 0.0% ni, 0.0% id, 96.2% wa, 0.0% hi, 0.0% si
>
> And after disabling spamassassin, I get:
>
> top - 13:37:07 up 51 days, 1:06, 2 users, load average: 0.07, 0.04, 0.94
> Tasks: 96 total, 1 running, 95 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.5% us, 0.0% sy, 0.0% ni, 98.0% id, 1.5% wa, 0.0% hi, 0.0% si
>
> Can I ask if this is normal, or am I doing something wrong somewhere?
Hi Piers,
I had the same problem, which is basically that more spam comes in
than the server can handle. The main reason for this is not cpu (on
_average_ 100% cpu all the time should be enough to handle all spam)
but because you run out of memory, the server starts swapping and it
becomes too slow to handle the incoming spam. As a result the number
of child processes run up till very high numbers (30 till 100 say)
and the machine becomes totally unusable.
There are several things you can do about this:
1) Reduce the maximum number of child processes that are allowed to
run simultaneously.
You can do this by passing --max-children 4 to /usr/bin/spamd
when starting it. That is, I use 4, you might need more if
you have more spam to handle (and the capability to run 4 of
them in parallel), which means however:
2) Increase the ammount of memory in the box.
You can balance this out with the number of child processes
you run thus (see above). My guess is that you need about 32 Mb
per child process, but that is a very wild guess, it might be
more.
The above didn't help me, therefore:
3) Reduce the ammount of spam that spamassassin has to handle!
This might seem stupid, but the reason that my server started
to lockup every few days turned out to be caused by ONE spam
source! Some total idiot had started to send me bursts of
spam, all from a single IP-number. By just firewalling that
single IP-number I reduced the spam with 90%. Got rid of the
burst, and everything worked again.
Moreover, you might want to firewall (or reject their mail
otherwise before it reaches spamassassin) all of South Korea and
all of China -- that will reduce the ammount of spam you
receive with about 99% ... So, it is more than worth it.
If that is too drastic for you, then try to get the statistics
of who is sending you nothing but spam. Likely there are a
few B or C classes that ONLY send spam to you and are responsible
for over 90% of the spam you get.
Finally, you can reduce the spam again DRASTICALLY if you suffer
from 'dictionary' attacks: try to find out what the bulk of the
spam that you receive is addressed to (that spamassassin is seeing).
If the majority of the spam is addressed to non-existant addresses
(as was the case in my case) then adding a filter that rejects
mail on recipient before it reaches spamassassin again greatly
reduces the ammount of spam it has to process.
I am doing this by having the following in my /etc/tcpcontrol/smtp.rules:
[...]
# Use Qmail-Scanner with SpamAssassin on any mail from the rest of the world
:allow,RCPTCHECK="/usr/local/bin/rcptcheck",QMAILQUEUE="/var/qmail/bin/qmail-scanner-queue.pl"
Where /usr/local/bin/rcptcheck is a little program that I wrote
myself which simply rejects mails based on the account name it is
sent too (Google for RCPTCHECK, assuming you are using qmail here).
Well, you get the idea. When my mail server started locking up
and I had been resetting it for weeks (like you are doing now) I got
real mad, stopped with what I was doing, started to investigate it,
and ended up with reducing the amount of spam that spamassassin had
to deal with a factor of one thousand, if not more.
--
Carlo Wood <ca...@alinoe.com>
PS Here is a list of IP-numbers that I firewall to reduce spam
with a factor of (more than) 100.
CHINANET Shanghai province network:
218.1.0.0/16
222.64.0.0/13
222.72.0.0/15
And South Korea:
59.0.0.0/11
59.150.0.0/16
59.186.0.0/15
60.196.0.0/15
61.4.192.0/19
61.32.0.0/13
61.40.0.0/14
61.72.0.0/13
61.80.0.0/14
61.84.0.0/15
61.96.0.0/12
61.247.128.0/19
61.248.0.0/13
128.134.0.0/16
129.254.0.0/16
134.75.0.0/16
137.68.0.0/16
141.223.0.0/16
143.248.0.0/16
147.6.0.0/16
147.43.0.0/16
150.150.0.0/16
150.183.0.0/16
152.99.0.0/16
152.149.0.0/16
154.10.0.0/16
155.230.0.0/16
156.147.0.0/16
158.44.0.0/16
161.122.0.0/16
163.152.0.0/16
163.180.0.0/16
163.239.0.0/16
164.124.0.0/15
165.132.0.0/15
165.141.0.0/16
165.194.0.0/16
165.213.0.0/16
165.243.0.0/16
165.244.0.0/16
165.246.0.0/16
166.79.0.0/16
166.103.0.0/16
166.104.0.0/16
166.125.0.0/16
168.78.0.0/16
168.115.0.0/16
168.126.0.0/16
168.131.0.0/16
168.154.0.0/16
168.188.0.0/16
168.219.0.0/16
168.248.0.0/15
169.140.0.0/16
192.5.90.0/24
192.100.2.0/24
192.104.15.0/24
192.132.15.0/24
192.132.247.0/24
192.132.248.0/22
192.195.39.0/24
192.195.40.0/24
192.203.138.0/23
192.203.140.0/22
192.203.144.0/23
192.203.146.0/24
192.245.249.0/24
192.245.250.0/23
192.249.16.0/20
202.6.95.0/24
202.14.103.0/24
202.14.165.0/24
202.20.82.0/23
202.20.84.0/23
202.20.86.0/24
202.20.99.0/24
202.20.119.0/24
202.20.128.0/17
202.21.0.0/21
202.30.0.0/15
202.86.8.0/21
202.189.128.0/20
203.81.128.0/19
203.83.128.0/19
203.90.32.0/19
203.100.160.0/19
203.224.0.0/11
210.16.192.0/18
210.80.96.0/19
210.90.0.0/15
210.92.0.0/14
210.96.0.0/11
210.178.0.0/15
210.180.0.0/14
210.204.0.0/14
210.216.0.0/13
211.32.0.0/11
211.104.0.0/13
211.112.0.0/13
211.168.0.0/13
211.176.0.0/12
211.192.0.0/10
218.36.0.0/14
218.48.0.0/13
218.101.128.0/17
218.144.0.0/12
218.232.0.0/13
219.240.0.0/15
219.248.0.0/13
220.64.0.0/11
220.103.0.0/16
220.116.0.0/14
220.120.0.0/13
220.149.0.0/16
221.132.64.0/19
221.133.128.0/18
221.138.0.0/15
221.140.0.0/14
221.144.0.0/12
221.160.0.0/13
221.168.0.0/16
222.96.0.0/12
222.112.0.0/13
222.120.0.0/15
222.122.0.0/16
222.231.0.0/18
222.232.0.0/13
Re: Abnormally high CPU usage when spamd is running
Posted by Jim Maul <jm...@elih.org>.
Piers Kittel wrote:
> Hmm...
>
> So if I get a 512MB strip and add it to the 384MB making it 896MB should
> be enough to stop spamassassin bogging down the server somewhat chronic?
>
I've been running qmail/qmail-scanner/clamav/SA 2.64 on a p4 2.8 with
256mb ram for over a year now with no problems. I recently upgraded the
machine to 512mb but only because it was laying around..there were no
performance issues in sight. Then again, its only processing about 2k
messages/day for about 100 users.
> Your second idea makes lots of sense, but for a small office (~8 people)
> with an extremely limited budget, I don't think it's really worth it,
> especially that I'm getting difficulity in getting £90 for 512MB let
> alone a new computer ;)
>
I know the feeling...we're a very small non profit hospital ;)
-Jim
Re: Abnormally high CPU usage when spamd is running
Posted by Piers Kittel <pi...@bda.org.uk>.
Aha - true, but the servers are spead very long way apart (other sides
of the country) and bandwidth is very narrow too.
Yeah £90 expensive, but it's server RAM, also the server only takes ECC
memory which pushes up the cost. It's from Crucial so can't get better
for cheaper.
Thanks very much for your help again
Cheers - Piers
Martin Hepworth wrote:
> Piers
>
> yes it will help.... but £90 for 512MB is expensive!
>
> and I meant 2 machines to gateway all five.....but I do concur it's a
> bit overkill when budgets are tight.
Re: Abnormally high CPU usage when spamd is running
Posted by Martin Hepworth <ma...@solid-state-logic.com>.
Piers
yes it will help.... but £90 for 512MB is expensive!
and I meant 2 machines to gateway all five.....but I do concur it's a
bit overkill when budgets are tight.
--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
Piers Kittel wrote:
> Hmm...
>
> So if I get a 512MB strip and add it to the 384MB making it 896MB should
> be enough to stop spamassassin bogging down the server somewhat chronic?
>
> Your second idea makes lots of sense, but for a small office (~8 people)
> with an extremely limited budget, I don't think it's really worth it,
> especially that I'm getting difficulity in getting £90 for 512MB let
> alone a new computer ;)
>
> Thanks very much for your help though, much appreciated!
>
> Cheers - Piers
>
> Martin Hepworth wrote:
>
>> Piers
>>
>> that amount of memory is not alot for use with SA. I find you need at
>> least 512mb, esp when I've got lots of rule additions, a local caching
>> namesever to the uri-rbls etc etc
>>
>> also you might to throttle back the amount of children spamd is
>> spawning as there is a known issue with spamd generating too many
>> forks. Theres a patch or too floating the bugzilla and this email list
>> that will be incorporated into 3.1 when it appears, but they seem to
>> work fine with 3.0.2
>>
>> Also have you considered a couple of machines acting as a front end to
>> your actual email boxes, therefore taking the load off your servers?
**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.
**********************************************************************
Re: Abnormally high CPU usage when spamd is running
Posted by Piers Kittel <pi...@bda.org.uk>.
Hmm...
So if I get a 512MB strip and add it to the 384MB making it 896MB should
be enough to stop spamassassin bogging down the server somewhat chronic?
Your second idea makes lots of sense, but for a small office (~8 people)
with an extremely limited budget, I don't think it's really worth it,
especially that I'm getting difficulity in getting £90 for 512MB let
alone a new computer ;)
Thanks very much for your help though, much appreciated!
Cheers - Piers
Martin Hepworth wrote:
> Piers
>
> that amount of memory is not alot for use with SA. I find you need at
> least 512mb, esp when I've got lots of rule additions, a local caching
> namesever to the uri-rbls etc etc
>
> also you might to throttle back the amount of children spamd is spawning
> as there is a known issue with spamd generating too many forks. Theres a
> patch or too floating the bugzilla and this email list that will be
> incorporated into 3.1 when it appears, but they seem to work fine with
> 3.0.2
>
> Also have you considered a couple of machines acting as a front end to
> your actual email boxes, therefore taking the load off your servers?
Re: Abnormally high CPU usage when spamd is running
Posted by Martin Hepworth <ma...@solid-state-logic.com>.
Piers
that amount of memory is not alot for use with SA. I find you need at
least 512mb, esp when I've got lots of rule additions, a local caching
namesever to the uri-rbls etc etc
also you might to throttle back the amount of children spamd is spawning
as there is a known issue with spamd generating too many forks. Theres a
patch or too floating the bugzilla and this email list that will be
incorporated into 3.1 when it appears, but they seem to work fine with 3.0.2
Also have you considered a couple of machines acting as a front end to
your actual email boxes, therefore taking the load off your servers?
--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
Piers Kittel wrote:
> Hello all,
>
> Have set up 5 email servers in various locations across the UK and they
> all are connected to each other. They all are using Debian Sarge. As I
> wanted spam detection, I installed spamassassin using the official
> debian packages and following
> http://www.clues.ltd.uk/howto/debian-sa-fprot-HOWTO.txt 's section on
> spamassassin, I set up spamassassin (but not fprot) and continiously
> found that after a few hours, I suddenly experience very high load
> averages, and extremely slow server performance for everything else, but
> if I restart spamassassin, the server works fine again, but it would
> start getting high load averages again later on and so on. So I
> disabled spamassassin on all servers, and tried a different method on
> only one server, and used sa-exim
> http://marc.merlins.org/linux/exim/sa.html - I just did an apt-get
> install sa-exim, made the necessary changes to make spamassassin work,
> and I got the same problem back again. While I realise spamassassin is
> very load intensive, but we are talking about a Compaq Proliant P3 1GHz
> server, around 350MB RAM, and maybe only 200-300 emails a hour.
>
> I have an email server using exim and spamassassin myself at home, and I
> decided to do an "apt-get update && apt-get upgrade" which upgraded
> spamassassin, and now I got the same problem msyelf at home, where it
> was working just fine with spamassassin for the past 1 1/2 years?
>
> Here's a snippet from top when the server is crunching through
> spamassassin:
>
> top - 13:07:59 up 51 days, 36 min, 2 users, load average: 7.16, 7.38,
> 6.28
> Tasks: 116 total, 1 running, 113 sleeping, 0 stopped, 2 zombie
> Cpu(s): 1.9% us, 1.9% sy, 0.0% ni, 0.0% id, 96.2% wa, 0.0% hi,
> 0.0% si
>
> And after disabling spamassassin, I get:
>
> top - 13:37:07 up 51 days, 1:06, 2 users, load average: 0.07, 0.04, 0.94
> Tasks: 96 total, 1 running, 95 sleeping, 0 stopped, 0 zombie
> Cpu(s): 0.5% us, 0.0% sy, 0.0% ni, 98.0% id, 1.5% wa, 0.0% hi,
> 0.0% si
>
> Can I ask if this is normal, or am I doing something wrong somewhere?
>
> Spamassassin is version 3.0.2-1, and the server I'm working on has a
> kernel version of 2.6.8, debian prepackaged version 2.6.8-2-686 to be
> exact.
>
> Thanks very much for your help in advance
>
> Regards - Piers
**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.
This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.
**********************************************************************