You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by a....@ukgrid.net on 2010/06/02 20:25:04 UTC
SpamAssassin is a disaster for me
Hi,
as per the subject Im having severe problems, any help much appreciated.
My installation:
FreeBSD 8.0-p2
Exim 4.71
SpamAssassin 3.3.1
Perl 5.10.1
All packages have been installed from source via FreeBSD ports.
The problem:
Two main issues really
1) Ever since this server was built (as a replacement for another
Exim/FreeBSD mail server) I have been seeing in the main messages file
and the spamassassin log that several spamassassin are dying per hour
with "signal 11".
2) I am seeing spamassassin perl processes that run for a very long
time (like an hour) at 100% CPU which is killing the server (in terms
of performance).
Today I have upgraded from perl 5.8.9 and reinstalled everything mail
related, Exim, SpamAssassin and all associated perl modules. I have
the same issues now as before. I don´t know what to do next. Its also
noteworthy that I have a FreeBSD 7.0 machine running the same
spamassassin version without any of these problems. I installed both
of these systems, I havent knowingly done anything different.
What can I do? Thanks for any suggestions....
Andy.
Re: perf problems (was Re: SpamAssassin is a disaster for me
Posted by Ted Mittelstaedt <te...@ipinc.net>.
On 6/2/2010 12:01 PM, spamassassinlist@saphirtech.fr wrote:
> Hi,
>
>> What can I do? Thanks for any suggestions....
>
> This is what I do when I get performance issues, and long-running
> processes:
>
Here is another performance hint.
When you first setup your server and it's running right, do the following:
cd to wherever you built spamassassin
(in my case
/usr/ports/mail/p5-Mail-SpamAssassin/work/Mail-SpamAssassin-3.3.1)
run the commands:
sh
time spamassassin -t < sample-nonspam.txt > nonspam.out
time spamassassin -t < sample-spam.txt > spam.out
Save this in your server build notes, this is your baseline
and in the future if you start having problems you can run
the same command to see if the slowdown in within SA itself,
or within your MTA.
Ted
perf problems (was Re: SpamAssassin is a disaster for me
Posted by sp...@saphirtech.fr.
Hi,
> What can I do? Thanks for any suggestions....
This is what I do when I get performance issues, and long-running
processes:
1) check for memory issues such as not enough "low" memory using "free -l"
and monitor the machine with top and vmstat for a while to get a general
feeling of what is actually eating up ressources.
2) check for network issues such as:
- badly configured iptables or selinux or whatever security "feature" that
actually prevents the normal operations through sockets
- DNS response times and general connectivity to RBLs
3) try to reproduce manually:
- are you using spamd and a spamc client, or running spamassassin as
command line on every mail ?
Excerpt from my .procmailrc
# Pipe the mail through spamassassin (replace 'spamassassin' with 'spamc'
# if you use the spamc/spamd combination)
# The lock file ensures that only 1 spamassassin invocation happens
# at 1 time, to keep the load down.
:0fw: spamassassin.lock
| spamc -f -s 900000
#| /usr/local/bin/spamassassin
- save any email locally say in mymail.txt and run it again using:
spamassassin -D -t < mymail.txt
and get a general feel about the speed, hangs, etc, then run it again and
save the output for further debugging, read it slowly.
PS: agreed at "you might have picked up another title". As for GFI I guess
this is it: http://www.gfi.com/mes/
HTH,
John
--
... I've seen Sun monitors on fire off the side of the multimedia lab.
I've seen NTU lights glitter in the dark near the Mail Gate.
All these things will be lost in time, like the root partition last
week.
Re: SpamAssassin is a disaster for me
Posted by a....@ukgrid.net.
> Well, before moving servers... What is its file size? See the bug I
> referenced earlier, and the oddities found there.
Yeah, I did have a look at this before. I dont think I have any
unusually large files that would cause a prob, the sizes are:
82M ./auto-whitelist
66K ./bayes_journal
7.6M ./bayes_seen
17M ./bayes_toks
4.0K ./user_prefs
>
> Also interesting would be, if you can read that file with (a) any
> Berkeley DB tool, (b) with Perl, and (c) if you can reproduce the
> issue with that file either on the current system or another.
Ok previously Id tested of DB integrity via sa-learn sync and assumed
if this worked that things were ok. If I run that against my files I
get exit status 0
# /usr/local/bin/sa-learn --dbpath /tmp/.spamassassin.old3 --sync
# echo $?
However if I try running db_dump:
# db_dump185-4.2 -f /tmp/test.db ./bayes_toks
Segmentation fault: 11 (core dumped)
# db_dump185-4.2 -f /tmp/test.db ./bayes_seen
Segmentation fault: 11 (core dumped)
Not so good :S About a month ago I totally deleted all the DB files
on this system as they were even failing the sa-learn sync command. So
these files are recently generated from scratch by SA, not migrated
from an earlier system etc...
Re: SpamAssassin is a disaster for me
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Thu, 2010-06-03 at 14:33 +0100, a.smith@ukgrid.net wrote:
> Quoting Karsten Bräckelmann <gu...@rudersport.de>:
> > That are *not* debug logs. That's standard logging, no debug.
>
> Post 1 of 3 is normal log, post 3 of 3 was from spamassassin with
> debug enabled. Is there some further debugging that can be enabled?
Oops, right. :)
> > Alas, as you mentioned in your reply to Mark, you flamed the BDB files.
> > So nothing to debug left here, can't even have a peek at the files. You
> > don't happen to have kept a copy, do you?
>
> Yes I do have a copy of the BDB files, they arent deleted! just moved.
> Ill need to copy them to another server to do any testing as my main
> priority is getting the live box behaving in a reasonable way.
Well, before moving servers... What is its file size? See the bug I
referenced earlier, and the oddities found there.
Also interesting would be, if you can read that file with (a) any
Berkeley DB tool, (b) with Perl, and (c) if you can reproduce the
issue with that file either on the current system or another.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: SpamAssassin is a disaster for me
Posted by a....@ukgrid.net.
Quoting Karsten Bräckelmann <gu...@rudersport.de>:
> That are *not* debug logs. That's standard logging, no debug.
Post 1 of 3 is normal log, post 3 of 3 was from spamassassin with
debug enabled. Is there some further debugging that can be enabled?
>
> Alas, as you mentioned in your reply to Mark, you flamed the BDB files.
> So nothing to debug left here, can't even have a peek at the files. You
> don't happen to have kept a copy, do you?
Yes I do have a copy of the BDB files, they arent deleted! just moved.
Ill need to copy them to another server to do any testing as my main
priority is getting the live box behaving in a reasonable way.
Re: SpamAssassin is a disaster for me
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Thu, 2010-06-03 at 10:12 +0100, a.smith@ukgrid.net wrote:
> > I first would check bugzilla for similar issues. In this particular
> > case, bug 6127. The comments in that bug should help to get debug logs,
> > which we would need.
>
> I have previously had a look through this bug report and also posted
> some debug logs
That are *not* debug logs. That's standard logging, no debug.
> Let me know if those previously posted provide any useful info
> http://www.gossamer-threads.com/lists/spamassassin/users/153019
Well, the third post hints it probably was Bayes, so Berkeley DB might
be the issue again.
Alas, as you mentioned in your reply to Mark, you flamed the BDB files.
So nothing to debug left here, can't even have a peek at the files. You
don't happen to have kept a copy, do you?
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: SpamAssassin is a disaster for me
Posted by a....@ukgrid.net.
Hi Karsten,
thanks a lot for your reply...
Quoting Karsten Bräckelmann <gu...@rudersport.de>:
>
> I first would check bugzilla for similar issues. In this particular
> case, bug 6127. The comments in that bug should help to get debug logs,
> which we would need.
I have previously had a look through this bug report and also posted
some debug logs
(http://www.gossamer-threads.com/lists/spamassassin/users/153019), Ill
revisit it. First on my agenda is switch to a MySQL backend, so Ill
take a look after that...
>
> It also briefly discusses the topic of a segmentation fault and Perl
> code. And maybe even a straw -- a particular subsystem to try disable
> and see if that helps.
>
> But ultimately, we need debug logs.
Let me know if those previously posted provide any useful info
http://www.gossamer-threads.com/lists/spamassassin/users/153019
>
> Does that include the libs some Perl modules use? Like Berkely DB?
Yes it Included everything dependent on Perl including BDB, everything
that was listed as dependent in the FreeBSD +CONTENTS files anyway...
thanks Andy.
Re: SpamAssassin is a disaster for me
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Wed, 2010-06-02 at 19:25 +0100, a.smith@ukgrid.net wrote:
> as per the subject Im having severe problems, any help much appreciated.
> My installation:
>
> FreeBSD 8.0-p2
> Exim 4.71
> SpamAssassin 3.3.1
> Perl 5.10.1
>
> All packages have been installed from source via FreeBSD ports.
> 1) Ever since this server was built (as a replacement for another
> Exim/FreeBSD mail server) I have been seeing in the main messages file
> and the spamassassin log that several spamassassin are dying per hour
> with "signal 11".
I first would check bugzilla for similar issues. In this particular
case, bug 6127. The comments in that bug should help to get debug logs,
which we would need.
It also briefly discusses the topic of a segmentation fault and Perl
code. And maybe even a straw -- a particular subsystem to try disable
and see if that helps.
But ultimately, we need debug logs.
> 2) I am seeing spamassassin perl processes that run for a very long
> time (like an hour) at 100% CPU which is killing the server (in terms
> of performance).
>
> Today I have upgraded from perl 5.8.9 and reinstalled everything mail
> related, Exim, SpamAssassin and all associated perl modules. I have
Does that include the libs some Perl modules use? Like Berkely DB?
> the same issues now as before. I don´t know what to do next. Its also
> noteworthy that I have a FreeBSD 7.0 machine running the same
> spamassassin version without any of these problems. I installed both
> of these systems, I havent knowingly done anything different.
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: SpamAssassin is a disaster for me
Posted by Ted Mittelstaedt <te...@ipinc.net>.
This weekend I just setup a mailserver running:
FreeBSD 7.3-RELEASE generic kernel
then cvsupped the current ports and make installed
on our software. I'm having no problems at all.
I'll send you more info off list.
This was a clean install. In other words, I FTP'ed the configs and
mail files and home dir contents to another server, then WIPED this with
the FBSD 7.3 install disk, and then threw on RELEASE, then cvsupped
ports, then remade all the utils.
I've never trusted the "in-situ" upgrade procedure, way too much
chance of stale libraries and such lying around and causing problems.
SA is so dependent on Perl and Perl modules, and the Perl modules
have so many dependencies, it could be virtually anything that could
muck up an upgrade like this.
Ted
On 6/2/2010 11:25 AM, a.smith@ukgrid.net wrote:
> Hi,
>
> as per the subject Im having severe problems, any help much appreciated.
> My installation:
>
> FreeBSD 8.0-p2
> Exim 4.71
> SpamAssassin 3.3.1
> Perl 5.10.1
>
> All packages have been installed from source via FreeBSD ports.
>
> The problem:
> Two main issues really
> 1) Ever since this server was built (as a replacement for another
> Exim/FreeBSD mail server) I have been seeing in the main messages file
> and the spamassassin log that several spamassassin are dying per hour
> with "signal 11".
> 2) I am seeing spamassassin perl processes that run for a very long time
> (like an hour) at 100% CPU which is killing the server (in terms of
> performance).
>
> Today I have upgraded from perl 5.8.9 and reinstalled everything mail
> related, Exim, SpamAssassin and all associated perl modules. I have the
> same issues now as before. I don´t know what to do next. Its also
> noteworthy that I have a FreeBSD 7.0 machine running the same
> spamassassin version without any of these problems. I installed both of
> these systems, I havent knowingly done anything different.
>
> What can I do? Thanks for any suggestions....
>
> Andy.
>
>
>
Re: SpamAssassin is a disaster for me
Posted by Ted Mittelstaedt <te...@ipinc.net>.
It's either Ground Fault Interruptor
(http://en.wikipedia.org/wiki/Residual-current_device) commonly used in
kitchens and
bathrooms ;-)
or it's something from this company:
http://www.gfi.com/
Ted
On 6/2/2010 11:38 AM, a.smith@ukgrid.net wrote:
> Well firstly I have said "for me" so Im not trying to trash Spamassasin
> (I seem to be in a minority of people with any severe issues), and
> secondly I have posted some of the same issues previously without such a
> theatrical subject and I received zero replies. No offence was intended
> for those who work on spamassassin, Im just a bit desperate....
>
> PS what is GFI?
>
>
>
Re: SpamAssassin is a disaster for me
Posted by a....@ukgrid.net.
No, Im looking at the spamassassin bayes DB files on CentOS 5.5,
spamassassin installed via yum....
Quoting RW <rw...@googlemail.com>:
> On Mon, 07 Jun 2010 13:14:31 +0100
> a.smith@ukgrid.net wrote:
>
>> I checked this on a dev linux box and on these the flat files are
>> created as Berkeley DB version 8 files. Is this important? And if so
>> how do you define which version to use for spamassassin?
>
>
> SpamAssassain is using GNU gdb rather than Oracle/Sleepycat db.
>
> I presume that accounts for the version difference - assuming you were
> looking at a random linux db file rather that a bayes_* file.
>
Re: SpamAssassin is a disaster for me
Posted by RW <rw...@googlemail.com>.
On Mon, 07 Jun 2010 13:14:31 +0100
a.smith@ukgrid.net wrote:
> I checked this on a dev linux box and on these the flat files are
> created as Berkeley DB version 8 files. Is this important? And if so
> how do you define which version to use for spamassassin?
SpamAssassain is using GNU gdb rather than Oracle/Sleepycat db.
I presume that accounts for the version difference - assuming you were
looking at a random linux db file rather that a bayes_* file.
Re: SpamAssassin is a disaster for me
Posted by a....@ukgrid.net.
Hi,
a few days on and things are still running well with MySQL bayes backend.
If indeed my system stability does continue I wonder if the bayes DB
version may have had something to do with the corruption problems that
seem to have caused me these problems. The flat files are always
created with a very old bayes DB version on my FreeBSD system, as can
be seen here:
# file bayes_*
bayes_seen: Berkeley DB 1.85 (Hash, version 2,
native byte-order)
bayes_toks: Berkeley DB 1.85 (Hash, version 2,
native byte-order)
I checked this on a dev linux box and on these the flat files are
created as Berkeley DB version 8 files. Is this important? And if so
how do you define which version to use for spamassassin? I asked this
question previously but didnt recieve any replies:
http://old.nabble.com/BDB-version-1.85-vs-8-how-to-select-td28475705.html
thanks Andy.
Re: SpamAssassin is a disaster for me
Posted by a....@ukgrid.net.
Hi,
update on this is Ive had the system running using MySQL backend
for Bayes and AWL for the last 4 hours. So far all good, no 100% CPU
runaway processes and no Signal 11 errors. The 100% CPU issue did seem
to be resolved by my initial deleting of the bayes BDB files (prior to
configuring MySQL) so that part is perhaps still unproven whether it
will continue without problems.
The lack of any signal 11 errors is perhaps more interesting, as I
said there have been zero instances and this was something I was
seeing every hour or so previously.
So all in all things are looking a lot better, panic over touch wood!
Ill mail any useful updates in the next days if I come to any firmer
conclusions...
Many thanks to those who have replied on this thread,
Andy.
Re: SpamAssassin is a disaster for me
Posted by a....@ukgrid.net.
Thanks very much for the detailed reply!
Quoting Mark Martinec <Ma...@ijs.si>:
>
> If you can isolate one such message which causes a crash and
> be able to reproduce it from a command line spamassassin, that would
> be ideal. Otherwise, enable debugging and when a process crashes
> check what was the last thing it was doing.
About catching a problem mail so I can reproduce the issue, Im not
sure how to do this as I believe (could well be mistaken!) that at the
time the mail is passed to spamassassin it doesnt exist on the
filesystem. I will look into this tho to check on how I might be able
to do it...
>
> My first suspect would be a file-based database if you happen to
> use it for bayes or awl. Moving bayes and awl to MySQL & InnoDB
> is more stable and faster for many of us.
Ok, I think this will be first on my list then (to change to MySQL
config). And actually thanks for the idea that it could be the DB, Ive
just chucked away all the flat DB files etc and restarted SA. So far
no 100% CPU issues...
>
> Second suspect would be some runaway regexp from third party rules.
AFAIK Im running totally default rules as per FreeBSD ports install.
>
> As you already found out and reported, the crash could also be
> due to some 5.10.1 bug (like the #69973), although that particular
> one was avoided if you use recent versions of perl modules from ports
> (such as the p5-HTML-Parser). I'm running 5.12.1 now on our FreeBSD
> mailer and is stable (just plain install from their source tarball,
> and then deleted and reinstalled all perl modules from ports).
As mentioned I reinstalled perl yesterday, my ports tree is up to date
so I should have latest of everything. The package u mention is at
version: p5-HTML-Parser-3.65. Perl 5.10.1 bug 69973 has been fixed in
the ports tree BTW, I even got a mention as the reporter in fresh
ports ;)
>
> My primary suspect would be bayes auto-expiry. Try turning it off
> (bayes_auto_expire 0) and do expiry runs manually or from cron.
Its already off so it cant be that, assuming this line in the file
/usr/local/etc/mail/spamassassin/local.cf is correct:
bayes_auto_expire 0
>
> A second candidate again could be some third-party regexp rule.
As above, dont think I have any 3rd party regexp rules...
>
> I doubt that 7.0 vs 8.0 would make any difference in this regard.
> Compare the versions of your bdb if using it - the last db47
> from ports seems fine. There were problems with some early 4.*
> versions of bdb.
Im a bit confused about this TBH as SA seems to use version 1.85 DB
files which are not compatible with db4.x. A anyway if I switch to
MySQL I wont have to worry about this....
>
thanks again! Andy.
PS will report back once Ive got MySQL configured...
PPS thats about 10mins with no 100% CPU now, so removing the old DB
files is looking good for fixing that prob (and putting the blame on
the flat file DBs).
Re: SpamAssassin is a disaster for me
Posted by Mark Martinec <Ma...@ijs.si>.
Andy,
> FreeBSD 8.0-p2
> Exim 4.71
> SpamAssassin 3.3.1
> Perl 5.10.1
> All packages have been installed from source via FreeBSD ports.
>
> The problem:
> Two main issues really
> 1) Ever since this server was built (as a replacement for another
> Exim/FreeBSD mail server) I have been seeing in the main messages file
> and the spamassassin log that several spamassassin are dying per hour
> with "signal 11".
If you can isolate one such message which causes a crash and
be able to reproduce it from a command line spamassassin, that would
be ideal. Otherwise, enable debugging and when a process crashes
check what was the last thing it was doing.
My first suspect would be a file-based database if you happen to
use it for bayes or awl. Moving bayes and awl to MySQL & InnoDB
is more stable and faster for many of us.
Second suspect would be some runaway regexp from third party rules.
As you already found out and reported, the crash could also be
due to some 5.10.1 bug (like the #69973), although that particular
one was avoided if you use recent versions of perl modules from ports
(such as the p5-HTML-Parser). I'm running 5.12.1 now on our FreeBSD
mailer and is stable (just plain install from their source tarball,
and then deleted and reinstalled all perl modules from ports).
> 2) I am seeing spamassassin perl processes that run for a very long
> time (like an hour) at 100% CPU which is killing the server (in terms
> of performance).
My primary suspect would be bayes auto-expiry. Try turning it off
(bayes_auto_expire 0) and do expiry runs manually or from cron.
A second candidate again could be some third-party regexp rule.
> Today I have upgraded from perl 5.8.9 and reinstalled everything mail
> related, Exim, SpamAssassin and all associated perl modules. I have
> the same issues now as before. I don´t know what to do next. Its also
> noteworthy that I have a FreeBSD 7.0 machine running the same
> spamassassin version without any of these problems. I installed both
> of these systems, I havent knowingly done anything different.
>
> What can I do? Thanks for any suggestions....
I doubt that 7.0 vs 8.0 would make any difference in this regard.
Compare the versions of your bdb if using it - the last db47
from ports seems fine. There were problems with some early 4.*
versions of bdb.
Mark
Re: SpamAssassin is a disaster for me
Posted by a....@ukgrid.net.
Well firstly I have said "for me" so Im not trying to trash
Spamassasin (I seem to be in a minority of people with any severe
issues), and secondly I have posted some of the same issues previously
without such a theatrical subject and I received zero replies. No
offence was intended for those who work on spamassassin, Im just a bit
desperate....
PS what is GFI?
Re: SpamAssassin is a disaster for me
Posted by Michael Scheidell <sc...@secnap.net>.
On 6/2/10 2:25 PM, a.smith@ukgrid.net wrote:
> Hi,
>
> as per the subject Im having severe problems, any help much
> appreciated.
run GFI on a windows 2007 server and all your troubles will be over.
Next time, try picking a subject line that will immediately piss off
anyone who knows anything spamassassin. that usually helps you get all
the help you need.
--
Michael Scheidell, CTO
Phone: 561-999-5000, x 1259
> *| *SECNAP Network Security Corporation
* Certified SNORT Integrator
* 2008-9 Hot Company Award Winner, World Executive Alliance
* Five-Star Partner Program 2009, VARBusiness
* Best Anti-Spam Product 2008, Network Products Guide
* King of Spam Filters, SC Magazine 2008
______________________________________________________________________
This email has been scanned and certified safe by SpammerTrap(r).
For Information please see http://www.secnap.com/products/spammertrap/
______________________________________________________________________