You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Richard Hobbs <ri...@crl.toshiba.co.uk> on 2007/08/28 13:09:55 UTC
spamd keeps running at 99% CPU until i kill the process
Hello,
We are running SpamAssassin version 3.1.7-deb running on Perl version
5.8.4 on a Debian Sarge box with exim4.
I am using the following router:
======================================================================
sa_router:
no_verify
check_local_user
# When to scan a message :
# - it isn't already flagged as spam from Spamassassin
# - it isn't already scanned
# - it isn't local
# - it isn't from one internal domain user to another
condition = "${if and { \
{!eq {$received_protocol}{spam-scanned}} \
{!eq {$received_protocol}{local}} \
{!eq {$sender_address_domain}{$domain}} \
} \
{1}{0}}"
driver = accept
transport = sa_spamcheck
local_parts = /etc/spamassassinUsers
======================================================================
and the following transport:
======================================================================
sa_spamcheck:
driver = pipe
command = /usr/sbin/exim4 -oMr spam-scanned -bS
use_bsmtp = true
transport_filter = /usr/bin/spamc
home_directory = "/tmp"
current_directory = "/tmp"
user = spamcheck
group = spamcheck
log_output = true
return_fail_output = true
return_path_add = false
message_prefix =
message_suffix =
======================================================================
This always used to work perfectly until i upgraded spamassassin using
apt-get to the version above (which was done in order to give us access
to the "sa-update" command).
Now, on some emails spamd eats up 99% CPU and doesn't return until i
kill it, at which point the email is delivered as non-spam, whether it
is spam or not.
Does anyone know why this is happening? It's destroying the use of our
mail server, lol!
Also, FYI, here is a copy of /etc/spamassassin/local.cf:
======================================================================
# This is the right place to customize your installation of SpamAssassin.
#
# See 'perldoc Mail::SpamAssassin::Conf' for details of what can be
# tweaked.
#
# Only a small subset of options are listed below
#
###########################################################################
rewrite_subject 1
# Add *****SPAM***** to the Subject header of spam e-mails
#
rewrite_header Subject ****SPAM**** (_HITS_/_REQD_):
# Save spam messages as a message/rfc822 MIME attachment instead of
# modifying the original message (0: off, 2: use text/plain instead)
#
report_safe 1
# Set which networks or hosts are considered 'trusted' by your mail
# server (i.e. not spammers)
#
# trusted_networks 212.17.35.
trusted_networks 192.168.3.
trusted_networks 193.128.142.
# Set file-locking method (flock is not safe over NFS, but is faster)
#
# lock_method flock
# Set the threshold at which a message is considered spam (default: 5.0)
#
required_score 2.5
# Use Bayesian classifier (default: 1)
#
# use_bayes 1
# Bayesian classifier auto-learning (default: 1)
#
# bayes_auto_learn 1
# Set headers which may provide inappropriate cues to the Bayesian
# classifier
#
# bayes_ignore_header X-Bogosity
# bayes_ignore_header X-Spam-Flag
# bayes_ignore_header X-Spam-Status
# WHITELIST
whitelist_from *@equanet.co.uk
whitelist_from *@toshiba.co.jp
whitelist_from *@dsgretail.co.uk
whitelist_from *@dsgibusiness.com
whitelist_from *@dhcltd.com
whitelist_from *@pcwb.com
whitelist_from *@netapp-web.com
======================================================================
Thanks in advance for any assistance! It's greatly appreciated!
Richard.
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: Re: integration with domino....linux
Posted by vi...@itera.ru.
richard venne <rv...@akio.fr> wrote on 29.08.2007 12:54:50:
> Ok, so the next question would be.... how to make sure that thoses MTA
> have the same valid mailboxse lists?
> Domino could export the list under standard formats? openldap? text
> plain files?
> by the way, I've only one server, so that I've to modify domino's port.
Domino has it's own LDAP server. Or you can just fill mailbox' list file
by scheduled script in Domino.
You can easily change SMTP listening port (Server Document in your Domino
Directory -> Ports -> Internet Ports -> Mail).
WBR,
Vitas.
Re: integration with domino....linux
Posted by richard venne <rv...@akio.fr>.
Ok, so the next question would be.... how to make sure that thoses MTA
have the same valid mailboxse lists?
Domino could export the list under standard formats? openldap? text
plain files?
by the way, I've only one server, so that I've to modify domino's port.
Matthias Leisi a écrit :
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
> richard venne schrieb:
>> Hi list
>>
>> I wonder wether it's possible to make works spamassassin with domino
>> server 7.0.x under linux (redhat el4)
>
> Use some combination of MTA (sendmail, Postfix, Exim, ...) integrated
> with SpamAssassin, which then forwards to your Notes box.
>
> [[ And make sure the MTA regularly get's a list of the valid recipient
> addresses. ]]
>
> - -- Matthias
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2 (GNU/Linux)
>
> iD8DBQFG1HuwxbHw2nyi/okRAl7hAJ4xzLb+KLMgMBZvUPE5vm/KDaZVmwCeIoQL
> HP/fswKC3tdLQdQOICloYgU=
> =6+PG
> -----END PGP SIGNATURE-----
>
Re: integration with domino....linux
Posted by Matthias Leisi <ma...@leisi.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
richard venne schrieb:
> Hi list
>
> I wonder wether it's possible to make works spamassassin with domino
> server 7.0.x under linux (redhat el4)
Use some combination of MTA (sendmail, Postfix, Exim, ...) integrated
with SpamAssassin, which then forwards to your Notes box.
[[ And make sure the MTA regularly get's a list of the valid recipient
addresses. ]]
- -- Matthias
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iD8DBQFG1HuwxbHw2nyi/okRAl7hAJ4xzLb+KLMgMBZvUPE5vm/KDaZVmwCeIoQL
HP/fswKC3tdLQdQOICloYgU=
=6+PG
-----END PGP SIGNATURE-----
RE: integration with domino....linux
Posted by "Martin.Hepworth" <ma...@solidstatelogic.com>.
Richard
You could setup a gateway with either MailScanner or Amavis-new
--
Martin Hepworth
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
> -----Original Message-----
> From: richard venne [mailto:rvenne@akio.fr]
> Sent: 28 August 2007 14:32
> Cc: users@spamassassin.apache.org
> Subject: integration with domino....linux
>
> Hi list
>
> I wonder wether it's possible to make works spamassassin with domino
> server 7.0.x under linux (redhat el4)
>
> thanks for your helps
**********************************************************************
Confidentiality : This e-mail and any attachments are intended for the
addressee only and may be confidential. If they come to you in error
you must take no action based on them, nor must you copy or show them
to anyone. Please advise the sender by replying to this e-mail
immediately and then delete the original from your computer.
Opinion : Any opinions expressed in this e-mail are entirely those of
the author and unless specifically stated to the contrary, are not
necessarily those of the author's employer.
Security Warning : Internet e-mail is not necessarily a secure
communications medium and can be subject to data corruption. We advise
that you consider this fact when e-mailing us.
Viruses : We have taken steps to ensure that this e-mail and any
attachments are free from known viruses but in keeping with good
computing practice, you should ensure that they are virus free.
Red Lion 49 Ltd T/A Solid State Logic
Registered as a limited company in England and Wales
(Company No:5362730)
Registered Office: 25 Spring Hill Road, Begbroke, Oxford OX5 1RU,
United Kingdom
**********************************************************************
integration with domino....linux
Posted by richard venne <rv...@akio.fr>.
Hi list
I wonder wether it's possible to make works spamassassin with domino
server 7.0.x under linux (redhat el4)
thanks for your helps
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Zbigniew Szalbot <zb...@szalbot.homedns.org>.
Hello,
One thing - you were given an excellent advice today on exim list by
Graeme. Why don't you follow it?
<quote>
Anyway, this is probably your problem:
http://spamassassin.apache.org/advisories/cve-2007-0451.txt Upgrade to
3.1.8 if you can.
</quote>
On Tue, 28 Aug 2007 15:09:14 +0000, Richard Hobbs
<ri...@crl.toshiba.co.uk> wrote:
>> No, those aren't really that big, but it does look like you have an
>> expiration problem.
>
> I've heard that this file should be around 10MB on a "standard" system,
> so surely 80MB is huge? Also, our mail server deals with around
> 1000-1500 emails per day, and is a single-CPU Pentium(R) 4 CPU 2.26GHz
> with 512MB RAM - should this be able to cope?
More than enough. I handle about 26K emails per day on a Pentium III,
512RAM pc and the system is almost idle most of the time.
--
Zbigniew Szalbot
www.slowo.pl
www.lcwords.com
Re: spamd keeps running at 99% CPU until i kill the process
Posted by "Daryl C. W. O'Shea" <sp...@dostech.ca>.
Richard Hobbs wrote:
> Hello,
>
> Michael Parker wrote:
>> Richard Hobbs wrote:
>>> Hello,
>>>
>>> Could the size of "bayes_seen" and "bayes_toks" be causing this timeout?
>>>
>> No, those aren't really that big, but it does look like you have an
>> expiration problem.
>
> I've heard that this file should be around 10MB on a "standard" system,
> so surely 80MB is huge?
No.
> Also, our mail server deals with around
> 1000-1500 emails per day, and is a single-CPU Pentium(R) 4 CPU 2.26GHz
> with 512MB RAM - should this be able to cope?
Certainly.
> Well, i've just run "sa-learn --force-expire" manually, and despite
> saying it had removed a whole load of entries from the database, the
> bayes_toks and bayes_seen files are still the same size (80MB and 10MB,
> respectively).
They're sparse files, so they're not going to shrink in reported size.
Daryl
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
Michael Parker wrote:
> Richard Hobbs wrote:
>> Hello,
>>
>> Could the size of "bayes_seen" and "bayes_toks" be causing this timeout?
>>
>
> No, those aren't really that big, but it does look like you have an
> expiration problem.
I've heard that this file should be around 10MB on a "standard" system,
so surely 80MB is huge? Also, our mail server deals with around
1000-1500 emails per day, and is a single-CPU Pentium(R) 4 CPU 2.26GHz
with 512MB RAM - should this be able to cope?
> To solve your immediate problem you could just turn off bayes, that will
> get mail flowing again and then you can address your real problem.
How effective would spam checking be with bayes disabled, though? I know
it's better than a broken solution, but just out of interest...
> I'd guess that the spike in CPU you are seeing is due to bayes
> expiration running and then getting killed off. Expiration can be very
> IO intensive and it locks the database while running which can cause a
> backup depending on you setup. To make matters worse it looks like
> you've got something timing out and killing off the expiration so its
> not allowed to complete.
That sounds about right - i have no idea what "spamd" is doing during
that time other than running a lot of "pread" system calls, but that
would make sense. I think exim gives 5 minutes to the spam checker, and
then kills it off and rejects the mail with the error in my previous mail.
> Long term, I'd suggest turning off auto expiration and then running
> sa-learn --force-expire via a cronjob. I believe there is good
> information on setting this up on the wiki.
Well, i've just run "sa-learn --force-expire" manually, and despite
saying it had removed a whole load of entries from the database, the
bayes_toks and bayes_seen files are still the same size (80MB and 10MB,
respectively).
> You can remove any *.expire* files that are older than 5 minutes, they
> are left over from previously timed out expiration attempts.
Done.
> Like I said, check out the wiki, you should find more information there
> on the problem and possible solutions.
Hmm... will do, but you think it's all working perfectly, just slower
than we'd like, right? Please correct me if that statement is incorrect.
Thanks again,
Richard.
> Michael
>> ======================================================================
>> mail:/home/spamcheck/.spamassassin# ls -l
>> total 101060
>> -rw------- 1 spamcheck spamcheck 2637824 2007-08-28 12:42 auto-whitelist
>> -rw------- 1 spamcheck spamcheck 6 2007-02-27 16:35
>> auto-whitelist.mutex
>> -rw------- 1 spamcheck spamcheck 56 2007-08-28 12:52 bayes.lock
>> -rw------- 1 spamcheck spamcheck 6 2007-02-27 16:33 bayes.mutex
>> -rw------- 1 spamcheck spamcheck 10530816 2007-08-28 11:11 bayes_seen
>> -rw------- 1 spamcheck spamcheck 83738624 2007-08-28 12:52 bayes_toks
>> -rw------- 1 spamcheck spamcheck 1298432 2007-02-27 18:14
>> bayes_toks.expire1698
>> -rw------- 1 spamcheck spamcheck 1327104 2007-02-28 02:17
>> bayes_toks.expire17474
>> -rw------- 1 spamcheck spamcheck 2473984 2007-02-28 02:31
>> bayes_toks.expire17865
>> -rw------- 1 spamcheck spamcheck 1302528 2007-02-28 02:48
>> bayes_toks.expire17866
>> -rw------- 1 spamcheck spamcheck 1335296 2007-02-28 03:17
>> bayes_toks.expire18715
>> -rw------- 1 spamcheck spamcheck 2572288 2007-02-28 03:37
>> bayes_toks.expire19618
>> -rw------- 1 spamcheck spamcheck 655360 2007-02-28 09:19
>> bayes_toks.expire28928
>> -rw------- 1 spamcheck spamcheck 1302528 2007-08-28 10:14
>> bayes_toks.expire3058
>> -rw------- 1 spamcheck spamcheck 1302528 2007-08-28 10:34
>> bayes_toks.expire3059
>> -rw------- 1 spamcheck spamcheck 1298432 2007-02-27 17:33
>> bayes_toks.expire31684
>> -rw------- 1 spamcheck spamcheck 1302528 2007-02-27 18:42
>> bayes_toks.expire31685
>> -rw------- 1 spamcheck spamcheck 5017600 2007-08-28 11:08
>> bayes_toks.expire4625
>> -rw------- 1 spamcheck spamcheck 5013504 2007-08-28 12:35
>> bayes_toks.expire7150
>> -rw------- 1 spamcheck spamcheck 5013504 2007-08-28 12:52
>> bayes_toks.expire7925
>> -rw-r--r-- 1 spamcheck spamcheck 1175 2005-07-20 14:23 user_prefs
>> mail:/home/spamcheck/.spamassassin#
>> ======================================================================
>>
>> If so, what can i do about this?
>>
>> Thanks again,
>> Richard.
>>
>>
>> Richard Hobbs wrote:
>>> Hello,
>>>
>>> Mark Martinec wrote:
>>>> Richard,
>>>>
>>>>> To add information to this problem, it appears that spamd does
>>>>> eventually give up after 5 minutes
>>>> Capture a message causing touble from a MTA queue,
>>>> and feed it to a command line spamassassin with -t -D options.
>>> I would love to do this, but it's not the same messages all the time -
>>> in a batch of identical messages, some get through and others cause the
>>> hanging. It seems to be completely random.
>>>
>>> Do you have any other ideas?
>>>
>>> Thanks again,
>>> Richard.
>>>
>
>
> _____________________________________________________________________
> This e-mail has been scanned for viruses by Verizon Business Internet Managed Scanning Services - powered by MessageLabs. For further information visit http://www.verizonbusiness.com/uk
>
>
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Michael Parker <pa...@pobox.com>.
Richard Hobbs wrote:
> Hello,
>
> Could the size of "bayes_seen" and "bayes_toks" be causing this timeout?
>
No, those aren't really that big, but it does look like you have an
expiration problem.
To solve your immediate problem you could just turn off bayes, that will
get mail flowing again and then you can address your real problem.
I'd guess that the spike in CPU you are seeing is due to bayes
expiration running and then getting killed off. Expiration can be very
IO intensive and it locks the database while running which can cause a
backup depending on you setup. To make matters worse it looks like
you've got something timing out and killing off the expiration so its
not allowed to complete.
Long term, I'd suggest turning off auto expiration and then running
sa-learn --force-expire via a cronjob. I believe there is good
information on setting this up on the wiki.
You can remove any *.expire* files that are older than 5 minutes, they
are left over from previously timed out expiration attempts.
Like I said, check out the wiki, you should find more information there
on the problem and possible solutions.
Michael
> ======================================================================
> mail:/home/spamcheck/.spamassassin# ls -l
> total 101060
> -rw------- 1 spamcheck spamcheck 2637824 2007-08-28 12:42 auto-whitelist
> -rw------- 1 spamcheck spamcheck 6 2007-02-27 16:35
> auto-whitelist.mutex
> -rw------- 1 spamcheck spamcheck 56 2007-08-28 12:52 bayes.lock
> -rw------- 1 spamcheck spamcheck 6 2007-02-27 16:33 bayes.mutex
> -rw------- 1 spamcheck spamcheck 10530816 2007-08-28 11:11 bayes_seen
> -rw------- 1 spamcheck spamcheck 83738624 2007-08-28 12:52 bayes_toks
> -rw------- 1 spamcheck spamcheck 1298432 2007-02-27 18:14
> bayes_toks.expire1698
> -rw------- 1 spamcheck spamcheck 1327104 2007-02-28 02:17
> bayes_toks.expire17474
> -rw------- 1 spamcheck spamcheck 2473984 2007-02-28 02:31
> bayes_toks.expire17865
> -rw------- 1 spamcheck spamcheck 1302528 2007-02-28 02:48
> bayes_toks.expire17866
> -rw------- 1 spamcheck spamcheck 1335296 2007-02-28 03:17
> bayes_toks.expire18715
> -rw------- 1 spamcheck spamcheck 2572288 2007-02-28 03:37
> bayes_toks.expire19618
> -rw------- 1 spamcheck spamcheck 655360 2007-02-28 09:19
> bayes_toks.expire28928
> -rw------- 1 spamcheck spamcheck 1302528 2007-08-28 10:14
> bayes_toks.expire3058
> -rw------- 1 spamcheck spamcheck 1302528 2007-08-28 10:34
> bayes_toks.expire3059
> -rw------- 1 spamcheck spamcheck 1298432 2007-02-27 17:33
> bayes_toks.expire31684
> -rw------- 1 spamcheck spamcheck 1302528 2007-02-27 18:42
> bayes_toks.expire31685
> -rw------- 1 spamcheck spamcheck 5017600 2007-08-28 11:08
> bayes_toks.expire4625
> -rw------- 1 spamcheck spamcheck 5013504 2007-08-28 12:35
> bayes_toks.expire7150
> -rw------- 1 spamcheck spamcheck 5013504 2007-08-28 12:52
> bayes_toks.expire7925
> -rw-r--r-- 1 spamcheck spamcheck 1175 2005-07-20 14:23 user_prefs
> mail:/home/spamcheck/.spamassassin#
> ======================================================================
>
> If so, what can i do about this?
>
> Thanks again,
> Richard.
>
>
> Richard Hobbs wrote:
>> Hello,
>>
>> Mark Martinec wrote:
>>> Richard,
>>>
>>>> To add information to this problem, it appears that spamd does
>>>> eventually give up after 5 minutes
>>> Capture a message causing touble from a MTA queue,
>>> and feed it to a command line spamassassin with -t -D options.
>> I would love to do this, but it's not the same messages all the time -
>> in a batch of identical messages, some get through and others cause the
>> hanging. It seems to be completely random.
>>
>> Do you have any other ideas?
>>
>> Thanks again,
>> Richard.
>>
>
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
Why is that? Surely updates are released for current, stable versions of
spamassassin? Most of the world can't run SVN version! lol
Any ideas why this might be?
Thanks again,
Richard.
jidanni@jidanni.org wrote:
>>> However, if there are no updates
>
> Which I bet you will be surprised to find will always be the case
> these days, unless you use yet unreleased SVN versions of spamassassin.
>
> _____________________________________________________________________
> This e-mail has been scanned for viruses by Verizon Business Internet Managed Scanning Services - powered by MessageLabs. For further information visit http://www.verizonbusiness.com/uk
>
>
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: spamd keeps running at 99% CPU until i kill the process
Posted by ji...@jidanni.org.
>> However, if there are no updates
Which I bet you will be surprised to find will always be the case
these days, unless you use yet unreleased SVN versions of spamassassin.
Re: spamd keeps running at 99% CPU until i kill the process
Posted by "John D. Hardin" <jh...@impsec.org>.
On Thu, 30 Aug 2007, Richard Hobbs wrote:
> ======================================================================
> echo "Updating rules..." && /usr/bin/sa-update && echo "Done, now
> checking config syntax..." && spamassassin --lint && echo "Done, now
> restarting spamd..." && /etc/init.d/spamassassin restart && echo "Done"
> ======================================================================
>
> However, if there are no updates (and sa-update exits with an exit code
> of 1) i would like to print "No updates, exiting" and then exit.
>
> So i modified it to become this:
>
> ======================================================================
> echo "Updating rules..." && /usr/bin/sa-update && echo "Done, now
> checking config syntax..." || echo "No updates, exiting"; exit &&
> spamassassin --lint && echo "Done, now restarting spamd..." &&
> /etc/init.d/spamassassin restart && echo "Done"
> ======================================================================
>
> However, whether there are updates or not, the script still exits - do
> you know which syntax i need to use here (i'm using bash, btw).
I'd recommend rewriting that to use if ... then ... else ... fi
syntax. Pasting stuff together with && and || is only manageable when
the logic is simple and straightforward.
Perhaps (off the top of my head):
echo "Updating rules..."
if /usr/bin/sa-update
then
echo "Done, now checking config syntax..."
if spamassassin --lint
then
echo "Syntax OK, now restarting spamd..."
/etc/init.d/spamassassin restart
echo "Done"
else
echo "Syntax error in update, please manually review!"
fi
else
echo "No updates, exiting."
fi
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
There is no doubt in my mind that millions of lives could have been
saved if the people were not "brainwashed" about gun ownership and
had been well armed. ... Gun haters always want to forget the Warsaw
Ghetto uprising, which is a perfect example of how a ragtag,
half-starved group of Jews took 10 handguns and made asses out of
the Nazis. -- Theodore Haas, Dachau Survivor
-----------------------------------------------------------------------
20 days until Talk Like a Pirate day
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
Sorry for so many emails, but it's sorted - removing "rewrite_subject 1"
from the config does not stop the subject line from being rewritten, so
i'm done!
However, with regards to the cronjob, i'd like some intelligent
reporting if possible, so i've constructed this:
======================================================================
echo "Updating rules..." && /usr/bin/sa-update && echo "Done, now
checking config syntax..." && spamassassin --lint && echo "Done, now
restarting spamd..." && /etc/init.d/spamassassin restart && echo "Done"
======================================================================
However, if there are no updates (and sa-update exits with an exit code
of 1) i would like to print "No updates, exiting" and then exit.
So i modified it to become this:
======================================================================
echo "Updating rules..." && /usr/bin/sa-update && echo "Done, now
checking config syntax..." || echo "No updates, exiting"; exit &&
spamassassin --lint && echo "Done, now restarting spamd..." &&
/etc/init.d/spamassassin restart && echo "Done"
======================================================================
However, whether there are updates or not, the script still exits - do
you know which syntax i need to use here (i'm using bash, btw).
Thanks again,
Richard.
Richard Hobbs wrote:
> Hello,
>
> Don't worry - i know what lint does now, but when i run it, i get this:
>
> mail:/etc/cron.weekly# spamassassin --lint
> [25465] warn: config: failed to parse line, skipping: rewrite_subject 1
> [25465] warn: lint: 1 issues detected, please rerun with debug enabled
> for more information
> mail:/etc/cron.weekly#
>
> But, if i remove the "rewrite_subject 1" line, it stops rewriting the
> subject line (i think)!
>
> Any suggestions?
>
> Thanks again,
> Richard.
>
>
> John D. Hardin wrote:
>> On Wed, 29 Aug 2007, Richard Hobbs wrote:
>>
>>> Would you guys also recommend putting the command "/usr/bin/sa-update"
>>> in the crontab as well?
>>>
>>> If so, should i be using any particular parameters? And how often should
>>> it be run to be worthwhile?
>> I run it weekly...
>>
>> /etc/cron.weekly/spamassassin-update:
>> /usr/bin/sa-update && spamassassin --lint && service spamassassin restart
>>
>> --
>> John Hardin KA7OHZ http://www.impsec.org/~jhardin/
>> jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
>> key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
>> -----------------------------------------------------------------------
>> The difference is that Unix has had thirty years of technical
>> types demanding basic functionality of it. And the Macintosh has
>> had fifteen years of interface fascist users shaping its progress.
>> Windows has the hairpin turns of the Microsoft marketing machine
>> and that's all. -- Red Drag Diva
>> -----------------------------------------------------------------------
>> 21 days until Talk Like a Pirate day
>>
>>
>> _____________________________________________________________________
>> This e-mail has been scanned for viruses by Verizon Business Internet Managed Scanning Services - powered by MessageLabs. For further information visit http://www.verizonbusiness.com/uk
>>
>>
>
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
Don't worry - i know what lint does now, but when i run it, i get this:
mail:/etc/cron.weekly# spamassassin --lint
[25465] warn: config: failed to parse line, skipping: rewrite_subject 1
[25465] warn: lint: 1 issues detected, please rerun with debug enabled
for more information
mail:/etc/cron.weekly#
But, if i remove the "rewrite_subject 1" line, it stops rewriting the
subject line (i think)!
Any suggestions?
Thanks again,
Richard.
John D. Hardin wrote:
> On Wed, 29 Aug 2007, Richard Hobbs wrote:
>
>> Would you guys also recommend putting the command "/usr/bin/sa-update"
>> in the crontab as well?
>>
>> If so, should i be using any particular parameters? And how often should
>> it be run to be worthwhile?
>
> I run it weekly...
>
> /etc/cron.weekly/spamassassin-update:
> /usr/bin/sa-update && spamassassin --lint && service spamassassin restart
>
> --
> John Hardin KA7OHZ http://www.impsec.org/~jhardin/
> jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
> key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
> -----------------------------------------------------------------------
> The difference is that Unix has had thirty years of technical
> types demanding basic functionality of it. And the Macintosh has
> had fifteen years of interface fascist users shaping its progress.
> Windows has the hairpin turns of the Microsoft marketing machine
> and that's all. -- Red Drag Diva
> -----------------------------------------------------------------------
> 21 days until Talk Like a Pirate day
>
>
> _____________________________________________________________________
> This e-mail has been scanned for viruses by Verizon Business Internet Managed Scanning Services - powered by MessageLabs. For further information visit http://www.verizonbusiness.com/uk
>
>
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
What does "spamassassin --lint" do? I can't find the lint option in the
man page...
Thanks again,
Richard.
John D. Hardin wrote:
> On Wed, 29 Aug 2007, Richard Hobbs wrote:
>
>> Would you guys also recommend putting the command "/usr/bin/sa-update"
>> in the crontab as well?
>>
>> If so, should i be using any particular parameters? And how often should
>> it be run to be worthwhile?
>
> I run it weekly...
>
> /etc/cron.weekly/spamassassin-update:
> /usr/bin/sa-update && spamassassin --lint && service spamassassin restart
>
> --
> John Hardin KA7OHZ http://www.impsec.org/~jhardin/
> jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
> key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
> -----------------------------------------------------------------------
> The difference is that Unix has had thirty years of technical
> types demanding basic functionality of it. And the Macintosh has
> had fifteen years of interface fascist users shaping its progress.
> Windows has the hairpin turns of the Microsoft marketing machine
> and that's all. -- Red Drag Diva
> -----------------------------------------------------------------------
> 21 days until Talk Like a Pirate day
>
>
> _____________________________________________________________________
> This e-mail has been scanned for viruses by Verizon Business Internet Managed Scanning Services - powered by MessageLabs. For further information visit http://www.verizonbusiness.com/uk
>
>
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: spamd keeps running at 99% CPU until i kill the process
Posted by "John D. Hardin" <jh...@impsec.org>.
On Wed, 29 Aug 2007, Richard Hobbs wrote:
> Would you guys also recommend putting the command "/usr/bin/sa-update"
> in the crontab as well?
>
> If so, should i be using any particular parameters? And how often should
> it be run to be worthwhile?
I run it weekly...
/etc/cron.weekly/spamassassin-update:
/usr/bin/sa-update && spamassassin --lint && service spamassassin restart
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
The difference is that Unix has had thirty years of technical
types demanding basic functionality of it. And the Macintosh has
had fifteen years of interface fascist users shaping its progress.
Windows has the hairpin turns of the Microsoft marketing machine
and that's all. -- Red Drag Diva
-----------------------------------------------------------------------
21 days until Talk Like a Pirate day
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
Would you guys also recommend putting the command "/usr/bin/sa-update"
in the crontab as well?
If so, should i be using any particular parameters? And how often should
it be run to be worthwhile?
Thanks again,
Richard.
Richard Hobbs wrote:
> Hello,
>
> Thank you - this is now done! :-)
>
> I shall see how it goes...
>
> Richard.
>
>
> Anthony Peacock wrote:
>> Hi,
>>
>> Richard Hobbs wrote:
>>> Hello,
>> <SNIP>
>>
>>>>> Is this all correct?
>>>> Yes. Set up expiration in a cron job, once per day is usually fine.
>>> OK, will do - i assume "sa-learn --force-expire" is the command to run
>>> via cron, right?
>> Yup! This is my crontab line:
>>
>> 00 22 * * * /usr/local/bin/sa-learn --force-expire
>>
>>> Also, how do i disable automatic expiration?
>> Set:
>>
>> bayes_auto_expire 0
>>
>> In your local.cf file.
>>
>>> Thanks again,
>>> Richard.
>>>
>>>
>>>>> Would a suitable alternative be to delete "bayes_seen" and
>>>>> "bayes_toks",
>>>>> then restart spamd? I know i would be deleting everything that had been
>>>>> learned over the last period of time, but starting afresh may not be a
>>>>> bad thing, seeing as the rules in the database are probably 6 months to
>>>>> a year old now (we've not been using spamd for a year or so, because i
>>>>> broke it and had no time to fix it!).
>>>>>
>>>>> Please let me know your thoughts, and also let me know whether deleting
>>>>> both of those files is a good way to go.
>>>> No.
>>>>
>>>>
>>
>
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
Thank you - this is now done! :-)
I shall see how it goes...
Richard.
Anthony Peacock wrote:
> Hi,
>
> Richard Hobbs wrote:
>> Hello,
> <SNIP>
>
>>>>
>>>> Is this all correct?
>>> Yes. Set up expiration in a cron job, once per day is usually fine.
>>
>> OK, will do - i assume "sa-learn --force-expire" is the command to run
>> via cron, right?
>
> Yup! This is my crontab line:
>
> 00 22 * * * /usr/local/bin/sa-learn --force-expire
>
>>
>> Also, how do i disable automatic expiration?
>
> Set:
>
> bayes_auto_expire 0
>
> In your local.cf file.
>
>>
>> Thanks again,
>> Richard.
>>
>>
>>>> Would a suitable alternative be to delete "bayes_seen" and
>>>> "bayes_toks",
>>>> then restart spamd? I know i would be deleting everything that had been
>>>> learned over the last period of time, but starting afresh may not be a
>>>> bad thing, seeing as the rules in the database are probably 6 months to
>>>> a year old now (we've not been using spamd for a year or so, because i
>>>> broke it and had no time to fix it!).
>>>>
>>>> Please let me know your thoughts, and also let me know whether deleting
>>>> both of those files is a good way to go.
>>> No.
>>>
>>>
>>
>
>
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Anthony Peacock <a....@chime.ucl.ac.uk>.
Hi,
Richard Hobbs wrote:
> Hello,
<SNIP>
>>>
>>> Is this all correct?
>> Yes. Set up expiration in a cron job, once per day is usually fine.
>
> OK, will do - i assume "sa-learn --force-expire" is the command to run
> via cron, right?
Yup! This is my crontab line:
00 22 * * * /usr/local/bin/sa-learn --force-expire
>
> Also, how do i disable automatic expiration?
Set:
bayes_auto_expire 0
In your local.cf file.
>
> Thanks again,
> Richard.
>
>
>>> Would a suitable alternative be to delete "bayes_seen" and "bayes_toks",
>>> then restart spamd? I know i would be deleting everything that had been
>>> learned over the last period of time, but starting afresh may not be a
>>> bad thing, seeing as the rules in the database are probably 6 months to
>>> a year old now (we've not been using spamd for a year or so, because i
>>> broke it and had no time to fix it!).
>>>
>>> Please let me know your thoughts, and also let me know whether deleting
>>> both of those files is a good way to go.
>> No.
>>
>>
>
--
Anthony Peacock
CHIME, Royal Free & University College Medical School
WWW: http://www.chime.ucl.ac.uk/~rmhiajp/
"A CAT scan should take less time than a PET scan. For a CAT scan,
they're only looking for one thing, whereas a PET scan could result in
a lot of things." - Carl Princi, 2002/07/19
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
Anthony Peacock wrote:
> Hi,
>
> Richard Hobbs wrote:
>> Hello,
>>
>> John D. Hardin wrote:
>>> On Tue, 28 Aug 2007, Richard Hobbs wrote:
>>>
>>>> Could the size of "bayes_seen" and "bayes_toks" be causing this
>>>> timeout?
>>> Yes.
>>>
>>>> If so, what can i do about this?
>>> Disable automatic Bayes expiry and do a manual expiration run, and
>>> allow it to complete.
>>
>> So what you (and Michael Parker) are saying is that it's not the
>> checking of spam against the tokens that is causing the timeout, it's
>> the automatic expiration of old tokens that "conveniently" gets tagged
>> onto the same operation, right?
>>
>> If so, let me get this straight - an email comes in and goes off to
>> spamd. spamd then checks the message against the tokens to determine
>> whether it's spam or not, then runs an expiry of old tokens (or perhaps
>> it happens the other way around), and only then returns the mail to
>> exim. The expiration of old tokens takes a lot longer than the spam
>> checking and as a result it's timing out.
>>
>> So, if i disable automatic expiration, spamd will only attempt one
>> operation at a time, (checking of spam against the tokens) and should
>> therefore not timeout, correct?
>>
>> But, if i do disable automatic expiration, i will have to remember to do
>> it manually, or via cron.
>>
>> Is this all correct?
>
> Yes. Set up expiration in a cron job, once per day is usually fine.
OK, will do - i assume "sa-learn --force-expire" is the command to run
via cron, right?
Also, how do i disable automatic expiration?
Thanks again,
Richard.
>> Would a suitable alternative be to delete "bayes_seen" and "bayes_toks",
>> then restart spamd? I know i would be deleting everything that had been
>> learned over the last period of time, but starting afresh may not be a
>> bad thing, seeing as the rules in the database are probably 6 months to
>> a year old now (we've not been using spamd for a year or so, because i
>> broke it and had no time to fix it!).
>>
>> Please let me know your thoughts, and also let me know whether deleting
>> both of those files is a good way to go.
>
> No.
>
>
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Anthony Peacock <a....@chime.ucl.ac.uk>.
Hi,
Richard Hobbs wrote:
> Hello,
>
> John D. Hardin wrote:
>> On Tue, 28 Aug 2007, Richard Hobbs wrote:
>>
>>> Could the size of "bayes_seen" and "bayes_toks" be causing this timeout?
>> Yes.
>>
>>> If so, what can i do about this?
>> Disable automatic Bayes expiry and do a manual expiration run, and
>> allow it to complete.
>
> So what you (and Michael Parker) are saying is that it's not the
> checking of spam against the tokens that is causing the timeout, it's
> the automatic expiration of old tokens that "conveniently" gets tagged
> onto the same operation, right?
>
> If so, let me get this straight - an email comes in and goes off to
> spamd. spamd then checks the message against the tokens to determine
> whether it's spam or not, then runs an expiry of old tokens (or perhaps
> it happens the other way around), and only then returns the mail to
> exim. The expiration of old tokens takes a lot longer than the spam
> checking and as a result it's timing out.
>
> So, if i disable automatic expiration, spamd will only attempt one
> operation at a time, (checking of spam against the tokens) and should
> therefore not timeout, correct?
>
> But, if i do disable automatic expiration, i will have to remember to do
> it manually, or via cron.
>
> Is this all correct?
Yes. Set up expiration in a cron job, once per day is usually fine.
>
> Would a suitable alternative be to delete "bayes_seen" and "bayes_toks",
> then restart spamd? I know i would be deleting everything that had been
> learned over the last period of time, but starting afresh may not be a
> bad thing, seeing as the rules in the database are probably 6 months to
> a year old now (we've not been using spamd for a year or so, because i
> broke it and had no time to fix it!).
>
> Please let me know your thoughts, and also let me know whether deleting
> both of those files is a good way to go.
No.
--
Anthony Peacock
CHIME, Royal Free & University College Medical School
WWW: http://www.chime.ucl.ac.uk/~rmhiajp/
"A CAT scan should take less time than a PET scan. For a CAT scan,
they're only looking for one thing, whereas a PET scan could result in
a lot of things." - Carl Princi, 2002/07/19
Re: spamd keeps running at 99% CPU until i kill the process
Posted by "John D. Hardin" <jh...@impsec.org>.
On Tue, 28 Aug 2007, Richard Hobbs wrote:
> Is this all correct?
Spot on.
> Would a suitable alternative be to delete "bayes_seen" and
> "bayes_toks", then restart spamd? I know i would be deleting
> everything that had been learned over the last period of time, but
> starting afresh may not be a bad thing, seeing as the rules in the
> database are probably 6 months to a year old now (we've not been
> using spamd for a year or so, because i broke it and had no time
> to fix it!).
How accurately is Bayes classifying messages? Why zap it entirely if
it's doing a good job?
> Please let me know your thoughts, and also let me know whether
> deleting both of those files is a good way to go.
You can safely delete the files with "expire" in their filename, those
are temporary work files. If you delete the entire Bayes database,
then Bayes won't do anything until it's learned more new spams and
hams.
I would suggest not zapping all of bayes.
Personally, I don't use autolearn, but then I have a very small user
base and can easily manually train.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
It is sadly humorous that those who are the most shrilly vocal
about bemoaning the increasing violations of civil liberties by
the federal government and comparing the president to Hitler are
also those who are working hardest to ensure the citizens of our
nation are disarmed and unable to effectively resist that same
government. Who do these people think will protect them from the
Jackbooted Thugs they are so worried about?
-----------------------------------------------------------------------
Today: Exercise Your Rights day
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
John D. Hardin wrote:
> On Tue, 28 Aug 2007, Richard Hobbs wrote:
>
>> Could the size of "bayes_seen" and "bayes_toks" be causing this timeout?
>
> Yes.
>
>> If so, what can i do about this?
>
> Disable automatic Bayes expiry and do a manual expiration run, and
> allow it to complete.
So what you (and Michael Parker) are saying is that it's not the
checking of spam against the tokens that is causing the timeout, it's
the automatic expiration of old tokens that "conveniently" gets tagged
onto the same operation, right?
If so, let me get this straight - an email comes in and goes off to
spamd. spamd then checks the message against the tokens to determine
whether it's spam or not, then runs an expiry of old tokens (or perhaps
it happens the other way around), and only then returns the mail to
exim. The expiration of old tokens takes a lot longer than the spam
checking and as a result it's timing out.
So, if i disable automatic expiration, spamd will only attempt one
operation at a time, (checking of spam against the tokens) and should
therefore not timeout, correct?
But, if i do disable automatic expiration, i will have to remember to do
it manually, or via cron.
Is this all correct?
Would a suitable alternative be to delete "bayes_seen" and "bayes_toks",
then restart spamd? I know i would be deleting everything that had been
learned over the last period of time, but starting afresh may not be a
bad thing, seeing as the rules in the database are probably 6 months to
a year old now (we've not been using spamd for a year or so, because i
broke it and had no time to fix it!).
Please let me know your thoughts, and also let me know whether deleting
both of those files is a good way to go.
Thanks again,
Richard.
> Once that's done you *can* turn automatic expiry back on to maintain
> it, but you might just get back into the same situation. You may want
> to leave automatic expiry off, and use cron to schedule a manual
> expiry at a low-traffic time of day.
>
> --
> John Hardin KA7OHZ http://www.impsec.org/~jhardin/
> jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
> key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
> -----------------------------------------------------------------------
> Taking my gun away because I *might* shoot someone is like cutting
> my tongue out because I *might* yell "Fire!" in a crowded theater.
> -- Peter Venetoklis
> -----------------------------------------------------------------------
> Today: Exercise Your Rights day
>
>
> _____________________________________________________________________
> This e-mail has been scanned for viruses by Verizon Business Internet Managed Scanning Services - powered by MessageLabs. For further information visit http://www.verizonbusiness.com/uk
>
>
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: spamd keeps running at 99% CPU until i kill the process
Posted by "John D. Hardin" <jh...@impsec.org>.
On Tue, 28 Aug 2007, Richard Hobbs wrote:
> Could the size of "bayes_seen" and "bayes_toks" be causing this timeout?
Yes.
> If so, what can i do about this?
Disable automatic Bayes expiry and do a manual expiration run, and
allow it to complete.
Once that's done you *can* turn automatic expiry back on to maintain
it, but you might just get back into the same situation. You may want
to leave automatic expiry off, and use cron to schedule a manual
expiry at a low-traffic time of day.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Taking my gun away because I *might* shoot someone is like cutting
my tongue out because I *might* yell "Fire!" in a crowded theater.
-- Peter Venetoklis
-----------------------------------------------------------------------
Today: Exercise Your Rights day
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
Could the size of "bayes_seen" and "bayes_toks" be causing this timeout?
======================================================================
mail:/home/spamcheck/.spamassassin# ls -l
total 101060
-rw------- 1 spamcheck spamcheck 2637824 2007-08-28 12:42 auto-whitelist
-rw------- 1 spamcheck spamcheck 6 2007-02-27 16:35
auto-whitelist.mutex
-rw------- 1 spamcheck spamcheck 56 2007-08-28 12:52 bayes.lock
-rw------- 1 spamcheck spamcheck 6 2007-02-27 16:33 bayes.mutex
-rw------- 1 spamcheck spamcheck 10530816 2007-08-28 11:11 bayes_seen
-rw------- 1 spamcheck spamcheck 83738624 2007-08-28 12:52 bayes_toks
-rw------- 1 spamcheck spamcheck 1298432 2007-02-27 18:14
bayes_toks.expire1698
-rw------- 1 spamcheck spamcheck 1327104 2007-02-28 02:17
bayes_toks.expire17474
-rw------- 1 spamcheck spamcheck 2473984 2007-02-28 02:31
bayes_toks.expire17865
-rw------- 1 spamcheck spamcheck 1302528 2007-02-28 02:48
bayes_toks.expire17866
-rw------- 1 spamcheck spamcheck 1335296 2007-02-28 03:17
bayes_toks.expire18715
-rw------- 1 spamcheck spamcheck 2572288 2007-02-28 03:37
bayes_toks.expire19618
-rw------- 1 spamcheck spamcheck 655360 2007-02-28 09:19
bayes_toks.expire28928
-rw------- 1 spamcheck spamcheck 1302528 2007-08-28 10:14
bayes_toks.expire3058
-rw------- 1 spamcheck spamcheck 1302528 2007-08-28 10:34
bayes_toks.expire3059
-rw------- 1 spamcheck spamcheck 1298432 2007-02-27 17:33
bayes_toks.expire31684
-rw------- 1 spamcheck spamcheck 1302528 2007-02-27 18:42
bayes_toks.expire31685
-rw------- 1 spamcheck spamcheck 5017600 2007-08-28 11:08
bayes_toks.expire4625
-rw------- 1 spamcheck spamcheck 5013504 2007-08-28 12:35
bayes_toks.expire7150
-rw------- 1 spamcheck spamcheck 5013504 2007-08-28 12:52
bayes_toks.expire7925
-rw-r--r-- 1 spamcheck spamcheck 1175 2005-07-20 14:23 user_prefs
mail:/home/spamcheck/.spamassassin#
======================================================================
If so, what can i do about this?
Thanks again,
Richard.
Richard Hobbs wrote:
> Hello,
>
> Mark Martinec wrote:
>> Richard,
>>
>>> To add information to this problem, it appears that spamd does
>>> eventually give up after 5 minutes
>> Capture a message causing touble from a MTA queue,
>> and feed it to a command line spamassassin with -t -D options.
>
> I would love to do this, but it's not the same messages all the time -
> in a batch of identical messages, some get through and others cause the
> hanging. It seems to be completely random.
>
> Do you have any other ideas?
>
> Thanks again,
> Richard.
>
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
Mark Martinec wrote:
> Richard,
>
>> To add information to this problem, it appears that spamd does
>> eventually give up after 5 minutes
>
> Capture a message causing touble from a MTA queue,
> and feed it to a command line spamassassin with -t -D options.
I would love to do this, but it's not the same messages all the time -
in a batch of identical messages, some get through and others cause the
hanging. It seems to be completely random.
Do you have any other ideas?
Thanks again,
Richard.
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Mark Martinec <Ma...@ijs.si>.
Richard,
> To add information to this problem, it appears that spamd does
> eventually give up after 5 minutes
Capture a message causing touble from a MTA queue,
and feed it to a command line spamassassin with -t -D options.
Mark
Re: spamd and plugins
Posted by Mark Martinec <Ma...@ijs.si>.
jonathan,
> so given that amavisd is already daemonized... does this suggest that
> there would be minimal gains in moving to spamd called from postfix?
Yes, practically no difference in throughput, possibly even some loss
in throughput due to spamc/spamd being invoked once per recipient,
and amavisd invoking SA once per message. In other words, amavisd
is just like spamd, but uses a different protocol to talk with MTA
(a standard SMTP vs. a proprietary spamc protocol + unix pipe).
Mark
Re: spamd and plugins
Posted by jonathan <je...@ryerson.ca>.
Matthias Leisi wrote:
> Amavisd does not use the spamassassin command line binary, but:
>
> - --- cut ---
> package Amavis::SpamControl;
> [..]
> use Mail::SpamAssassin;
> - --- cut ---
>
> In the logfile of amavisd, you'll also see something like:
>
> | Aug 30 20:26:11 amavis[27896]: Module Mail::SpamAssassin 3.002000
>
so given that amavisd is already daemonized... does this suggest that
there would be minimal gains in moving to spamd called from postfix?
(bearing in mind that amavisd is going to have to stay for virus
filtering regardless).
thanks,
jonathan.
Re: spamd and plugins
Posted by Matthias Leisi <ma...@leisi.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
jonathan schrieb:
> I'm more concerned about the overhead of calling spamassassin over and
> over from within amavis. Some suggest three or four times better
> performance using spamc/d over calling the spamassassin command directly
> (which is (i believe) what amavis does). Being able to handle
Amavisd does not use the spamassassin command line binary, but:
- --- cut ---
package Amavis::SpamControl;
[..]
use Mail::SpamAssassin;
- --- cut ---
In the logfile of amavisd, you'll also see something like:
| Aug 30 20:26:11 amavis[27896]: Module Mail::SpamAssassin 3.002000
> substantially more messages would help us quite a bit as our incoming
> mail volume is rapidly on the rise.
>
> If I've got loadplugin commands in my various custom .cf files, will
> these be properly heeded by spamd?
If the custom .cf file are in a location that is read by your spamd
process, yes.
- -- Matthias
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iD8DBQFG1wwqxbHw2nyi/okRAqq7AKCvhlJA1qdtBKcDDLbLfjbLROsbNQCfYNHk
J/RhHuegL0cH2WRwOTPrknQ=
=hZMI
-----END PGP SIGNATURE-----
Re: spamd and plugins
Posted by jonathan <je...@ryerson.ca>.
Matthias Leisi wrote:
> Amavisd uses the SpamAssassin Perl module directly (ie not really spamc
> or spamd).
>
> For a high-volume site, amavisd has advantages if you want to do
> additional checks besides SA (eg virus checking). If you only want to
> run SA, you could avoid the overhead of amavisd.
>
Ok. I understand this side of things, and we're not going to be able to
ditch amavisd due to our virus filtering setup.
I'm more concerned about the overhead of calling spamassassin over and
over from within amavis. Some suggest three or four times better
performance using spamc/d over calling the spamassassin command directly
(which is (i believe) what amavis does). Being able to handle
substantially more messages would help us quite a bit as our incoming
mail volume is rapidly on the rise.
If I've got loadplugin commands in my various custom .cf files, will
these be properly heeded by spamd?
Re: spamd and plugins
Posted by Matthias Leisi <ma...@leisi.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
jonathan schrieb:
> For a high-volume site, is there any reason to not be running
> spamc/spamd over amavisd/spamassassin? Specifically, can you use all of
Amavisd uses the SpamAssassin Perl module directly (ie not really spamc
or spamd).
For a high-volume site, amavisd has advantages if you want to do
additional checks besides SA (eg virus checking). If you only want to
run SA, you could avoid the overhead of amavisd.
> the various perl plugins (imageinfo, pdfinfo, botnet, etc...)? Are
I run various incarnations of SA (through sendmail-milter, through
amavisd, spamc as a Postfix content_filter), and I can access any plugin
I want in any of these environments. Being able to run a plugin is more
a Perl issue (having the necessary @INC paths for the user running the
"container", filesystem permissions etc) than a SA issue.
- -- Matthias
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
iD8DBQFG1wKVxbHw2nyi/okRAvoiAJ9Dj3d1PyC4aD9t4a5U6SEnK143WwCZAX0h
3PquILgowvAxKq5QlJR89SE=
=88lT
-----END PGP SIGNATURE-----
spamd and plugins
Posted by jonathan <je...@ryerson.ca>.
For a high-volume site, is there any reason to not be running
spamc/spamd over amavisd/spamassassin? Specifically, can you use all of
the various perl plugins (imageinfo, pdfinfo, botnet, etc...)? Are
there any other tradeoffs to this configuration?
thanks,
Jonathan.
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Micke Andersson <mi...@swemic.net>.
Stuart Gall wrote:
>
> On 30 Aug 2007, at 16:55, Micke Andersson wrote:
>
>> Richard Hobbs wrote:
>>> 421 SMTP incoming data timeout - message abandoned
>>>
>>>
>>>
>>>
>> Hi, I have had exactly the same problem as you, with about the very
>> same setup as you!
>> The problem where actually a TCP problem, and not a SpamAssassin problem!
>> From a bunch of TCPDUMPs from my side and a good partner side, we did
>> finaly track down the problem to
>> TCP_WINDOW_SCALING, which is set to 1 (True) on Debian, which seems
>> to give a problem
>> to Exim. My solution for the problem was to set this parameter to 0
>> (false)
>> Easiest done by
>> echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
>>
>> And further on, this seems to be a problem mostly against old
>> versions of Sendmail, when they
>> where sending to us, and had some kind of attachment in the email as
>> well!
>> (I don't really know where the bug is, if it is in Exim or TCP, I
>> found a solution and is pleased with that)
>>
>> Hope this helps you out, a bit of topic though!
>
>
> This may have fixed the problem by simply severely reducing the data
> transmission speed.
> With window scaling off your maximum window size is a mere 64K, hardly
> enough these days.
>
> I have had this problem since upgrading to the latest SA.
> But I had also at the same time taken the opportunity to add a load of
> spam and ham to bayes.
>
> It makes much more sense that spamd is trying to do a bayes expire and
> taking too long.
> sa-learn --force-expire took a good 4 minutes
>
> Setting auto expire off as suggested in this thread has fixed the
> problem for me.
>
> *THEREFORE*
> The actual problem must be that it is taking longer to do the bayes
> expire than spamd's timeout child setting each new spamd child tries
> to do a bayes expire but never has enough time to complete it.
> Other wise one would expect just a single spamd to go 99% not every spamd.
>
> Right?
>
> Perhaps the bayes auto expire code should be moved to the parent
> process to prevent this problem.
>
>>
>>
>>
>
My reflection was purely to the SMTP error "421 SMTP incoming data
timeout - message abandoned"
Which on one of my SMTP server where solved with my above solution!
And that particular error, I have only had it on Exim4 server, none of
my Sendmail nor Postfix installations
suffer from that!
/Micke
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Stuart Gall <st...@otenet.gr>.
On 30 Aug 2007, at 16:55, Micke Andersson wrote:
> Richard Hobbs wrote:
>> Hello,
>>
>> To add information to this problem, it appears that spamd does
>> eventually give up after 5 minutes - which then bounces the
>> message back
>> to the sender stating:
>>
>> 421 SMTP incoming data timeout - message abandoned
>>
>> Obviously, this cannot keep happening, but i don't know how to
>> stop it...
>>
>> Any advice greatly appreciated.
>>
>> Thanks again,
>> Richard.
>>
>>
> Hi, I have had exactly the same problem as you, with about the very
> same setup as you!
> The problem where actually a TCP problem, and not a SpamAssassin
> problem!
> From a bunch of TCPDUMPs from my side and a good partner side, we
> did finaly track down the problem to
> TCP_WINDOW_SCALING, which is set to 1 (True) on Debian, which seems
> to give a problem
> to Exim. My solution for the problem was to set this parameter to 0
> (false)
> Easiest done by
> echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
>
> And further on, this seems to be a problem mostly against old
> versions of Sendmail, when they
> where sending to us, and had some kind of attachment in the email
> as well!
> (I don't really know where the bug is, if it is in Exim or TCP, I
> found a solution and is pleased with that)
>
> Hope this helps you out, a bit of topic though!
This may have fixed the problem by simply severely reducing the data
transmission speed.
With window scaling off your maximum window size is a mere 64K,
hardly enough these days.
I have had this problem since upgrading to the latest SA.
But I had also at the same time taken the opportunity to add a load
of spam and ham to bayes.
It makes much more sense that spamd is trying to do a bayes expire
and taking too long.
sa-learn --force-expire took a good 4 minutes
Setting auto expire off as suggested in this thread has fixed the
problem for me.
THEREFORE
The actual problem must be that it is taking longer to do the bayes
expire than spamd's timeout child setting each new spamd child tries
to do a bayes expire but never has enough time to complete it.
Other wise one would expect just a single spamd to go 99% not every
spamd.
Right?
Perhaps the bayes auto expire code should be moved to the parent
process to prevent this problem.
>
>
>
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Micke Andersson <mi...@swemic.net>.
Richard Hobbs wrote:
> Hello,
>
> To add information to this problem, it appears that spamd does
> eventually give up after 5 minutes - which then bounces the message back
> to the sender stating:
>
> 421 SMTP incoming data timeout - message abandoned
>
> Obviously, this cannot keep happening, but i don't know how to stop it...
>
> Any advice greatly appreciated.
>
> Thanks again,
> Richard.
>
>
Hi, I have had exactly the same problem as you, with about the very same
setup as you!
The problem where actually a TCP problem, and not a SpamAssassin problem!
From a bunch of TCPDUMPs from my side and a good partner side, we did
finaly track down the problem to
TCP_WINDOW_SCALING, which is set to 1 (True) on Debian, which seems to
give a problem
to Exim. My solution for the problem was to set this parameter to 0 (false)
Easiest done by
echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
And further on, this seems to be a problem mostly against old versions
of Sendmail, when they
where sending to us, and had some kind of attachment in the email as well!
(I don't really know where the bug is, if it is in Exim or TCP, I found
a solution and is pleased with that)
Hope this helps you out, a bit of topic though!
Re: spamd keeps running at 99% CPU until i kill the process
Posted by Richard Hobbs <ri...@crl.toshiba.co.uk>.
Hello,
To add information to this problem, it appears that spamd does
eventually give up after 5 minutes - which then bounces the message back
to the sender stating:
421 SMTP incoming data timeout - message abandoned
Obviously, this cannot keep happening, but i don't know how to stop it...
Any advice greatly appreciated.
Thanks again,
Richard.
Richard Hobbs wrote:
> Hello,
>
> We are running SpamAssassin version 3.1.7-deb running on Perl version
> 5.8.4 on a Debian Sarge box with exim4.
>
> I am using the following router:
>
> ======================================================================
> sa_router:
> no_verify
> check_local_user
> # When to scan a message :
> # - it isn't already flagged as spam from Spamassassin
> # - it isn't already scanned
> # - it isn't local
> # - it isn't from one internal domain user to another
> condition = "${if and { \
> {!eq {$received_protocol}{spam-scanned}} \
> {!eq {$received_protocol}{local}} \
> {!eq {$sender_address_domain}{$domain}} \
> } \
> {1}{0}}"
> driver = accept
> transport = sa_spamcheck
> local_parts = /etc/spamassassinUsers
> ======================================================================
>
> and the following transport:
>
> ======================================================================
> sa_spamcheck:
> driver = pipe
> command = /usr/sbin/exim4 -oMr spam-scanned -bS
> use_bsmtp = true
> transport_filter = /usr/bin/spamc
> home_directory = "/tmp"
> current_directory = "/tmp"
> user = spamcheck
> group = spamcheck
> log_output = true
> return_fail_output = true
> return_path_add = false
> message_prefix =
> message_suffix =
> ======================================================================
>
> This always used to work perfectly until i upgraded spamassassin using
> apt-get to the version above (which was done in order to give us access
> to the "sa-update" command).
>
> Now, on some emails spamd eats up 99% CPU and doesn't return until i
> kill it, at which point the email is delivered as non-spam, whether it
> is spam or not.
>
> Does anyone know why this is happening? It's destroying the use of our
> mail server, lol!
>
> Also, FYI, here is a copy of /etc/spamassassin/local.cf:
>
> ======================================================================
> # This is the right place to customize your installation of SpamAssassin.
> #
> # See 'perldoc Mail::SpamAssassin::Conf' for details of what can be
> # tweaked.
> #
> # Only a small subset of options are listed below
> #
> ###########################################################################
>
> rewrite_subject 1
>
> # Add *****SPAM***** to the Subject header of spam e-mails
> #
> rewrite_header Subject ****SPAM**** (_HITS_/_REQD_):
>
>
> # Save spam messages as a message/rfc822 MIME attachment instead of
> # modifying the original message (0: off, 2: use text/plain instead)
> #
> report_safe 1
>
>
> # Set which networks or hosts are considered 'trusted' by your mail
> # server (i.e. not spammers)
> #
> # trusted_networks 212.17.35.
> trusted_networks 192.168.3.
> trusted_networks 193.128.142.
>
>
> # Set file-locking method (flock is not safe over NFS, but is faster)
> #
> # lock_method flock
>
>
> # Set the threshold at which a message is considered spam (default: 5.0)
> #
> required_score 2.5
>
>
> # Use Bayesian classifier (default: 1)
> #
> # use_bayes 1
>
>
> # Bayesian classifier auto-learning (default: 1)
> #
> # bayes_auto_learn 1
>
>
> # Set headers which may provide inappropriate cues to the Bayesian
> # classifier
> #
> # bayes_ignore_header X-Bogosity
> # bayes_ignore_header X-Spam-Flag
> # bayes_ignore_header X-Spam-Status
>
> # WHITELIST
> whitelist_from *@equanet.co.uk
> whitelist_from *@toshiba.co.jp
> whitelist_from *@dsgretail.co.uk
> whitelist_from *@dsgibusiness.com
> whitelist_from *@dhcltd.com
> whitelist_from *@pcwb.com
> whitelist_from *@netapp-web.com
> ======================================================================
>
> Thanks in advance for any assistance! It's greatly appreciated!
>
> Richard.
>
--
Richard Hobbs (Systems Administrator)
Toshiba Research Europe Ltd. - Speech Technology Group
Email: richard.hobbs@crl.toshiba.co.uk
Web: http://www.toshiba-europe.com/research/
Tel: +44 1223 376964 Mobile: +44 7811 803377