spamd causing high load

[K Anand <ka...@sail-steel.com>]

Hi all,

   I am running SA 3.1.1. I have seen that sometimes spamd processes using 
up a lot of CPU. The cpu load goes up very high to ~ 10. I have checked that 
RAM is not the problem since free shows that memory is still free. I have 1 
GB RAM. Another thing is that my AWL file is around 85 MB. I did a du -k and 
it shows 65036. My bayes_seen file is around 25 MB. I have set auto_expire 
to 1. There's also a sa-learn --sync thats running hourly.

 My line is a 64k leased line. I also see that my smtpd connections are also 
maxing out to 100. Generally this happens when a mailing list starts 
bombarding my server with mails. These are legit mails as a lot of my users 
have subscribed to this list.


Any suggestions would be welcome.

Thanx

Anand 

Re: spamassasin --lint

[Nigel Frankcom <ni...@blue-canoe.net>]

On Thu, 9 Nov 2006 18:43:12 +0800, <fl...@eads.com.my> wrote:

>hi
>
>i new here..
>
>can anyone know  after i adding new domain for whitelist,  how can i me it
>active immediately with out stop and start the service.
>
>because when restart , some spam mail already pass throught...
>
>what is the differet betweent   service spamassassin restart  and  service
>spamassassin condrestart ?
>
>or  after add new whitelist  just type   spamassassin --lint will  the
>whitelist domain active immediate ?
>
>thanks
>wong
>

I'm not sure about condrestart but any changes you make to the
local.cf file won't be applied until you restart the SA service.

--lint reads the settings in from the local.cf and associated files as
if SA was running with those settings, not as SA is running at that
point.

I deal with restarts by having multiple SA servers running, if my MTA
can't get a response out of the 1st it goes to the 2nd and so on.

If you are getting that much spam through, you could stop your smtp
service or shut port 25 temporarily while SA restarts.

Hope that helps

Nigel

spamassasin --lint

[fl...@eads.com.my]

hi

i new here..

can anyone know  after i adding new domain for whitelist,  how can i me it
active immediately with out stop and start the service.

because when restart , some spam mail already pass throught...

what is the differet betweent   service spamassassin restart  and  service
spamassassin condrestart ?

or  after add new whitelist  just type   spamassassin --lint will  the
whitelist domain active immediate ?

thanks
wong

Re: spamd causing high load

[K Anand <ka...@sail-steel.com>]

----- Original Message ----- 
From: "Matt Kettler" <mk...@verizon.net>
>>
>>>
>>> Also, with that much mail coming in at the same time, there could be
>>> contention for bayes locks.  You might try adding
>>> "bayes_learn_to_journal 1" to your local.cf, and see if that helps. This
>>> will cause learning to be done into a "journal" file which periodically
>>> gets merged into the main bayes DB. This causes the live bayes to be
>>> delayed in update until the next sync (once a day or every 100k of bayes
>>> data by default), but you can force-sync any manual training runs by
>>> running sa-learn --sync afterwards.
>>
>> I don't have  "bayes_learn_to_journal 1" in my local.cf. But I see
>> bayes_journal file in the bayes directory. So it must be default
>> behaviour. As I had written , I do sa-learn --sync every hour .
>
> By default SA does not learn to the journal. AFAIK, it only updates
> atimes in the journal, by default.
>
> Looking at BayeStore/DBM.pm, tok_touch always uses defer_update, even if
> learn_to_journal is disabled.

I noticed another thing which might give a clue. Although my smptd 
connections were maxed 100/100, the cpu load was still low ~0.4. But when I 
saw the load when the hourly sync was going on, the load was high ~5-7 even 
though the smtpd connections were not maxed. I have set Auto_expire to 1. 
Could that be the problem ? Should I set auto_expire to 0 and 
do --force-expire thro a cronjob and check ?

Anand 

Re: spamd causing high load

[Matt Kettler <mk...@verizon.net>]

K Anand wrote:
>
>>
>> Also, with that much mail coming in at the same time, there could be
>> contention for bayes locks.  You might try adding
>> "bayes_learn_to_journal 1" to your local.cf, and see if that helps. This
>> will cause learning to be done into a "journal" file which periodically
>> gets merged into the main bayes DB. This causes the live bayes to be
>> delayed in update until the next sync (once a day or every 100k of bayes
>> data by default), but you can force-sync any manual training runs by
>> running sa-learn --sync afterwards.
>
> I don't have  "bayes_learn_to_journal 1" in my local.cf. But I see
> bayes_journal file in the bayes directory. So it must be default
> behaviour. As I had written , I do sa-learn --sync every hour .

By default SA does not learn to the journal. AFAIK, it only updates
atimes in the journal, by default.

Looking at BayeStore/DBM.pm, tok_touch always uses defer_update, even if
learn_to_journal is disabled.

Re: spamd causing high load

[K Anand <ka...@sail-steel.com>]

----- Original Message ----- 
From: "Matt Kettler" <mk...@verizon.net>


>K Anand wrote:
>> Hi all,
>>
>>   I am running SA 3.1.1.
> Warning: if you use the -v and -P options to spamd,  your version is
> vulnerable to a remote code exploit. This is not a typical setup, but
> you should be aware of it.
>

No,  I'm not runnning -v or -P options. Thanx for the tip.


>> I have seen that sometimes spamd processes using up a lot of CPU. The
>> cpu load goes up very high to ~ 10. I have checked that RAM is not the
>> problem since free shows that memory is still free. I have 1 GB RAM.
>> Another thing is that my AWL file is around 85 MB. I did a du -k and
>> it shows 65036. My bayes_seen file is around 25 MB. I have set
>> auto_expire to 1. There's also a sa-learn --sync thats running hourly.
>>
>> My line is a 64k leased line. I also see that my smtpd connections are
>> also maxing out to 100. Generally this happens when a mailing list
>> starts bombarding my server with mails. These are legit mails as a lot
>> of my users have subscribed to this list.
>>
>>
>> Any suggestions would be welcome.
>
> The AWL file won't auto-expire, so you'll need to use the
> check_whitelist script from the tools directory of the tarball to clean
> it. It's just a script, and some terse docs are at the top of the file
> if you open it in an editor.

I was reading the forums and I saw that this scrit won't actually lower the 
file size. Another script was suggested to compact the db.

> As for the load.. do you have a local caching DNS server? or is you SA
> box having to always go out over the 64k line to resolve DNS? If it is,
> install a simple cache on your SA box and change the resolv.conf to use
> 127.0.0.1 as a DNS server. This should help considerably with latency,
> which might help a bit with the load.

I'm not running a local caching DNS server ..But I'm using a DNS server 
which is on the same LAN as my mail server. So I don't think thats the 
problem.

>
> Also, with that much mail coming in at the same time, there could be
> contention for bayes locks.  You might try adding
> "bayes_learn_to_journal 1" to your local.cf, and see if that helps. This
> will cause learning to be done into a "journal" file which periodically
> gets merged into the main bayes DB. This causes the live bayes to be
> delayed in update until the next sync (once a day or every 100k of bayes
> data by default), but you can force-sync any manual training runs by
> running sa-learn --sync afterwards.

I don't have  "bayes_learn_to_journal 1" in my local.cf. But I see 
bayes_journal file in the bayes directory. So it must be default behaviour. 
As I had written , I do sa-learn --sync every hour .

need some more ideas.

Thanx .

Re: spamd causing high load

[Matt Kettler <mk...@verizon.net>]

K Anand wrote:
> Hi all,
>
>   I am running SA 3.1.1. 
Warning: if you use the -v and -P options to spamd,  your version is
vulnerable to a remote code exploit. This is not a typical setup, but
you should be aware of it.

http://wiki.apache.org/spamassassin/Security

> I have seen that sometimes spamd processes using up a lot of CPU. The
> cpu load goes up very high to ~ 10. I have checked that RAM is not the
> problem since free shows that memory is still free. I have 1 GB RAM.
> Another thing is that my AWL file is around 85 MB. I did a du -k and
> it shows 65036. My bayes_seen file is around 25 MB. I have set
> auto_expire to 1. There's also a sa-learn --sync thats running hourly.
>
> My line is a 64k leased line. I also see that my smtpd connections are
> also maxing out to 100. Generally this happens when a mailing list
> starts bombarding my server with mails. These are legit mails as a lot
> of my users have subscribed to this list.
>
>
> Any suggestions would be welcome.

The AWL file won't auto-expire, so you'll need to use the
check_whitelist script from the tools directory of the tarball to clean
it. It's just a script, and some terse docs are at the top of the file
if you open it in an editor.

As for the load.. do you have a local caching DNS server? or is you SA
box having to always go out over the 64k line to resolve DNS? If it is,
install a simple cache on your SA box and change the resolv.conf to use
127.0.0.1 as a DNS server. This should help considerably with latency,
which might help a bit with the load.

Also, with that much mail coming in at the same time, there could be
contention for bayes locks.  You might try adding
"bayes_learn_to_journal 1" to your local.cf, and see if that helps. This
will cause learning to be done into a "journal" file which periodically
gets merged into the main bayes DB. This causes the live bayes to be
delayed in update until the next sync (once a day or every 100k of bayes
data by default), but you can force-sync any manual training runs by
running sa-learn --sync afterwards.