You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Gustavo Michels <gu...@ig.com.br> on 2004/03/21 17:43:24 UTC
SA hangs for 5 minutes on first message
Hi
I was first asking about SA's performance and I had a tip from the list to use
spamd/spamc instead of the regular spamassassin client. I did that but the
problem still continues. SpamAssassin will take 5 minutes to classify the
first message.
So I turned on debugging and I am attaching the debug log so maybe someone can
help me out. I also thought of filing a bug, but I'd better ask to the list
first.
Here's the log with the message that makes SA wait for 5 minutes with 100% cpu
usage:
spamd[5975]: [debug] bayes: Can't use estimation method for expiry, something
fishy, calculating optimal atime delta (first pass)_
cheers
Gustavo
Re: SA hangs for 5 minutes on first message
Posted by Alan Baxter <at...@baxtersys.com>.
On Sun, 21 Mar 2004 16:32:11 -0300, Gustavo Michels
<gu...@ig.com.br> wrote:
>On Sunday 21 March 2004 15:49, Alan Baxter wrote:
>
>> It looks like you're using a database that you got from somewhere else
>> instead of one that's based on the spam and ham that you've seen since
>> you started using SA. That's not bad per se. Is bayes working
>> effectively for you? It won't expire any tokens until you've accessed
>> at least 112500, and I think it might be several weeks before you reach
>> that number as a single user.
>
>Well, I use Gentoo and all I did was emerge the package. Unless there's a
>sample database with gentoo's package (which btw I don't think there is), I
>am sure this database was created by me.
>
>In the first weeks of use, I used sa-learn --spam everyday since about 40% of
>my spam wasn't being flagged. Nowadays, the accuracy I have is extremely
>good, I rarely get false positives or spam not being flagged. I must say I am
>happy with SA so far, however I started having this 5 minutes hang problem
>(since I use it as a pipe thru filter action in KMail, it would get my mail
>client to hang for 5 minutes everyday) and this wasn't happening in the first
>weeks of use.
That seems about right. It takes about 90 seconds to expire my database
when it reaches 150,000.
It sure looked weird to me that you had a database with over 800,000
tokens. It should have been trying to auto expire every 12 hours ever
since it reached 150,000. On top of that it showed that only 21,000 or
so had been accessed within the past two months. It doesn't look like
you've received much email during that time frame.
>> Your bayes might be more effective if you eliminate all of those unused
>> tokens. You ought to be able to force it to remove all of the tokens
>> you haven't used by putting "bayes_expiry_max_db_size 28000" in your
>> user_prefs. This should cause it to purge all of the tokens that you
>> haven't used the next time an expiry is attempted. Once you've done
>> that you can remove the bayes_expiry_max_db_size line and bayes will
>> grow using only the tokens that you learn from your email. I have a
>> single user installation too, so I don't need auto expiration. I just
>> do a manual expire once every month or so.
>
>Well first I tried feeding the database as Theo Van Dinter suggested. Then I
>rerun the force expire and indeed it worked:
> ...
>
>Then I tried your "bayes_expiry_max_db_size 28000" suggestion, but there seems
>to have a lower limit for the db size:
>
>debug: bayes: expiry check keep size, 75% of max: 21000
>debug: bayes: expiry keep size too small, resetting to 100,000 tokens
Thanks for doing the experiment. :-) Rereading the sa-learn man page, I
see that the minimum keep size is 100,000.
>So I guess I will leave it at 100,000, not use auto-expire and do it like you
>do, once a month. My main problem was KMail being hung for 5 minutes, and
>that's not going to happen anymore.
Cool. Glad SA is working so well for you.
Alan
--
Please avoid sending me Word or PowerPoint attachments.
See http://www.fsf.org/philosophy/no-word-attachments.html
Re: SA hangs for 5 minutes on first message
Posted by Gustavo Michels <gu...@ig.com.br>.
On Sunday 21 March 2004 15:49, Alan Baxter wrote:
> It looks like you're using a database that you got from somewhere else
> instead of one that's based on the spam and ham that you've seen since
> you started using SA. That's not bad per se. Is bayes working
> effectively for you? It won't expire any tokens until you've accessed
> at least 112500, and I think it might be several weeks before you reach
> that number as a single user.
Well, I use Gentoo and all I did was emerge the package. Unless there's a
sample database with gentoo's package (which btw I don't think there is), I
am sure this database was created by me.
In the first weeks of use, I used sa-learn --spam everyday since about 40% of
my spam wasn't being flagged. Nowadays, the accuracy I have is extremely
good, I rarely get false positives or spam not being flagged. I must say I am
happy with SA so far, however I started having this 5 minutes hang problem
(since I use it as a pipe thru filter action in KMail, it would get my mail
client to hang for 5 minutes everyday) and this wasn't happening in the first
weeks of use.
> Your bayes might be more effective if you eliminate all of those unused
> tokens. You ought to be able to force it to remove all of the tokens
> you haven't used by putting "bayes_expiry_max_db_size 28000" in your
> user_prefs. This should cause it to purge all of the tokens that you
> haven't used the next time an expiry is attempted. Once you've done
> that you can remove the bayes_expiry_max_db_size line and bayes will
> grow using only the tokens that you learn from your email. I have a
> single user installation too, so I don't need auto expiration. I just
> do a manual expire once every month or so.
Well first I tried feeding the database as Theo Van Dinter suggested. Then I
rerun the force expire and indeed it worked:
debug: bayes: token count: 1100702, final goal reduction size: 988202
debug: bayes: First pass? Current: 1079895562, Last: 1079891344, atime:
1382400, count: 1789, newdelta: 2502, ratio: 552.376746785914
debug: bayes: Can't use estimation method for expiry, something fishy,
calculating optimal atime delta (first pass)
debug: bayes: atime token reduction
debug: bayes: ======== ===============
debug: bayes: 43200 1083283
debug: bayes: 86400 1083143
debug: bayes: 172800 1082351
debug: bayes: 345600 1078392
debug: bayes: 691200 1067224
debug: bayes: 1382400 901614
debug: bayes: 2764800 879583
debug: bayes: 5529600 859620
debug: bayes: 11059200 0
debug: bayes: 22118400 0
debug: bayes: First pass decided on 1382400 for atime delta
debug: bayes: 6312 untie-ing
debug: bayes: 6312 untie-ing db_toks
debug: bayes: 6312 untie-ing db_seen
debug: bayes: files locked, now unlocking lock
debug: unlock: 6312 unlink /home/gmichels/.spamassassin/bayes.lock
expired old Bayes database entries in 196 seconds
199088 entries kept, 901614 deleted
token frequency: 1-occurence tokens: 93.19%
token frequency: less than 8 occurrences: 5.69%
debug: Syncing complete.
debug: bayes: 6312 untie-ing
Then I tried your "bayes_expiry_max_db_size 28000" suggestion, but there seems
to have a lower limit for the db size:
debug: bayes: expiry check keep size, 75% of max: 21000
debug: bayes: expiry keep size too small, resetting to 100,000 tokens
So I guess I will leave it at 100,000, not use auto-expire and do it like you
do, once a month. My main problem was KMail being hung for 5 minutes, and
that's not going to happen anymore.
cheers
Gustavo
Re: SA hangs for 5 minutes on first message
Posted by Alan Baxter <at...@baxtersys.com>.
On Sun, 21 Mar 2004 14:55:44 -0300, Gustavo Michels
<gu...@ig.com.br> wrote:
>On Sunday 21 March 2004 14:14, Alan Baxter wrote:
>
>> SA is *not* waiting or hanging for five minutes. It's busy analyzing
>> your bayes database to find an appropriate expiry time. Your bayes
>> database is about six times "bayes_journal_max_size" and doesn't look
>> like it has been used much in the last 64 days. As a result, SA can't
>> find an appropriate expire time and attempts to automatically expire the
>> database again at a later time.
>>
>> I suggest you turn off auto expiry by adding "bayes_auto_expire 0" to
>> your user_prefs file and attempt a manual expiry every few days with
>> "sa-learn --force-expire".
>
>Ok, thanks for the clarification, I added the option to my user_prefs so it
>won't be doing that every boot.
>
>I also tried the manual expire with similar results to the log I sent before.
>Is there something wrong with my bayes database? Shouldn't that expire finish
>with a better result than "debug: bayes: couldn't find a good delta atime,
>need more token difference, skipping expire"?
No, it shouldn't. There's a good discussion of bayes expiry in the man
page for sa-learn that I kept in mind when studying your log file. The
debug information "couldn't find a good delta atime, need more token
difference" means that it didn't have enough information to determine
which tokens should be removed from the database. Of the 887218 tokens
in there, only 21636 have been used in the past 64 days. I suppose it
might be reasonable for it to remove all but the 21636 tokens that have
been used, but expire will not remove any tokens if that would cause the
resulting database to have less than 112500. (Unless you reduce
bayes_expiry_max_db_size. See below.)
It looks like you're using a database that you got from somewhere else
instead of one that's based on the spam and ham that you've seen since
you started using SA. That's not bad per se. Is bayes working
effectively for you? It won't expire any tokens until you've accessed
at least 112500, and I think it might be several weeks before you reach
that number as a single user.
Your bayes might be more effective if you eliminate all of those unused
tokens. You ought to be able to force it to remove all of the tokens
you haven't used by putting "bayes_expiry_max_db_size 28000" in your
user_prefs. This should cause it to purge all of the tokens that you
haven't used the next time an expiry is attempted. Once you've done
that you can remove the bayes_expiry_max_db_size line and bayes will
grow using only the tokens that you learn from your email. I have a
single user installation too, so I don't need auto expiration. I just
do a manual expire once every month or so.
Hope this helps,
Alan
P.S. Let me know how well this works if you try it.
--
Please avoid sending me Word or PowerPoint attachments.
See http://www.fsf.org/philosophy/no-word-attachments.html
Re: SA hangs for 5 minutes on first message
Posted by Theo Van Dinter <fe...@kluge.net>.
On Sun, Mar 21, 2004 at 02:55:44PM -0300, Gustavo Michels wrote:
> I also tried the manual expire with similar results to the log I sent before.
> Is there something wrong with my bayes database? Shouldn't that expire finish
> with a better result than "debug: bayes: couldn't find a good delta atime,
> need more token difference, skipping expire"?
The problem, as shown in the debug output, is that you're not learning
enough tokens:
bayes: atime_token reduction_
bayes: ========_===============_
bayes: 43200_887081_
bayes: 86400_887062_
bayes: 172800_885496_
bayes: 345600_883823_
bayes: 691200_880512_
bayes: 1382400_877509_
bayes: 2764800_873664_
bayes: 5529600_865582_
bayes: 11059200_0_
bayes: 22118400_0_
The expiry can't choose any of those atime values, because too many
tokens will get removed. The solution: learn more tokens. :)
--
Randomly Generated Tagline:
"Hoping the problem magically goes away by ignoring it is the 'Microsoft
approach to programming' and should never be allowed." - Linus Torvalds
Re: SA hangs for 5 minutes on first message
Posted by Gustavo Michels <gu...@ig.com.br>.
On Sunday 21 March 2004 14:14, Alan Baxter wrote:
> SA is *not* waiting or hanging for five minutes. It's busy analyzing
> your bayes database to find an appropriate expiry time. Your bayes
> database is about six times "bayes_journal_max_size" and doesn't look
> like it has been used much in the last 64 days. As a result, SA can't
> find an appropriate expire time and attempts to automatically expire the
> database again at a later time.
>
> I suggest you turn off auto expiry by adding "bayes_auto_expire 0" to
> your user_prefs file and attempt a manual expiry every few days with
> "sa-learn --force-expire".
Ok, thanks for the clarification, I added the option to my user_prefs so it
won't be doing that every boot.
I also tried the manual expire with similar results to the log I sent before.
Is there something wrong with my bayes database? Shouldn't that expire finish
with a better result than "debug: bayes: couldn't find a good delta atime,
need more token difference, skipping expire"?
cheers
Gustavo
Re: SA hangs for 5 minutes on first message
Posted by Alan Baxter <at...@baxtersys.com>.
On Sun, 21 Mar 2004 13:43:24 -0300, Gustavo Michels
<gu...@ig.com.br> wrote:
>
>I was first asking about SA's performance and I had a tip from the list to use
>spamd/spamc instead of the regular spamassassin client. I did that but the
>problem still continues. SpamAssassin will take 5 minutes to classify the
>first message.
>
>So I turned on debugging and I am attaching the debug log so maybe someone can
>help me out. I also thought of filing a bug, but I'd better ask to the list
>first.
>
>Here's the log with the message that makes SA wait for 5 minutes with 100% cpu
>usage:
>
>spamd[5975]: [debug] bayes: Can't use estimation method for expiry, something
>fishy, calculating optimal atime delta (first pass)_
SA is *not* waiting or hanging for five minutes. It's busy analyzing
your bayes database to find an appropriate expiry time. Your bayes
database is about six times "bayes_journal_max_size" and doesn't look
like it has been used much in the last 64 days. As a result, SA can't
find an appropriate expire time and attempts to automatically expire the
database again at a later time.
I suggest you turn off auto expiry by adding "bayes_auto_expire 0" to
your user_prefs file and attempt a manual expiry every few days with
"sa-learn --force-expire".
Hope this helps,
Alan
--
Please avoid sending me Word or PowerPoint attachments.
See http://www.fsf.org/philosophy/no-word-attachments.html
Re: SA hangs for 5 minutes on first message
Posted by Duncan Hill <sa...@nacnud.force9.co.uk>.
On Sunday 21 March 2004 16:43, Gustavo Michels wrote:
> Hi
>
> I was first asking about SA's performance and I had a tip from the list to
> use spamd/spamc instead of the regular spamassassin client. I did that but
> the problem still continues. SpamAssassin will take 5 minutes to classify
> the first message.
> spamd[5975]: [debug] bayes: Can't use estimation method for expiry,
> something fishy, calculating optimal atime delta (first pass)_
As a pure guess, spamd is doing a bayes database rebuild/expire on the first
call after a reboot. There is a sa-learn argument that will do this, which
might work. I defer to those who know better though.
Re: SA hangs for 5 minutes on first message
Posted by jdow <jd...@earthlink.net>.
You need to convince spamd to run when the machine is booted. There
should be an init.d entry for spamassassin. Make appropriate links or
other provisions so that it starts up properly when the machine is
rebooted. The way to do this is rather OS dependant.
{^_^}
----- Original Message -----
From: "Gustavo Michels" <gu...@ig.com.br>
To: <sp...@incubator.apache.org>
Sent: Sunday, 2004 March, 21 08:43
Subject: SA hangs for 5 minutes on first message
> Hi
>
> I was first asking about SA's performance and I had a tip from the list to
use
> spamd/spamc instead of the regular spamassassin client. I did that but the
> problem still continues. SpamAssassin will take 5 minutes to classify the
> first message.
>
> So I turned on debugging and I am attaching the debug log so maybe someone
can
> help me out. I also thought of filing a bug, but I'd better ask to the
list
> first.
>
> Here's the log with the message that makes SA wait for 5 minutes with 100%
cpu
> usage:
>
> spamd[5975]: [debug] bayes: Can't use estimation method for expiry,
something
> fishy, calculating optimal atime delta (first pass)_
>
> cheers
> Gustavo
>