Bayes database

[Don Newcomer <ne...@dickinson.edu>]

I'm really confused and it's caused me some serious mail pile-ups today.  I
_thought_ I had Bayes configured so it's only manually trained by me so
locking shouldn't be a problem.  HOWEVER, I keep seeing the modification
dates change for my Bayes database files and lockfiles come and go.  The
biggest problem came when I ended up with a deadlock and I had over 1800
messages backed up when I got back from a meeting.  I'm still getting these
e-mails out because of the normal amount of inbound mail traffic we
experience.

I'm using MailScanner version 4.30.3 and SpamAssassin 2.63.  Here are my
pertinent Bayes entries from local.cf (which is a link to
MailScanner/etc/spam.assassin.prefs.conf for MailScanner):

bayes_path                 /var/spool/spamassassin/bayes
bayes_file_mode            0600
bayes_ignore_header X-Dickinson-MailScanner
bayes_ignore_header X-Dickinson-MailScanner-SpamCheck
bayes_ignore_header X-Dickinson-MailScanner-SpamScore
bayes_ignore_header X-Dickinson-MailScanner-Information
bayes_auto_learn 0

The only thing I changed recently happened yesterday.  I noticed that when
I ran sa-learn, I was creating new Bayes files in /.spamassassin and, since
I had commented the 'bayes_path' entry out, I removed the comment and it
seemed to work fine as long as I told sa-learn where the config file lived.

What am I missing that it's still creating lockfiles?  Here's what
/var/spool/spamassassin looked like a few minutes ago:

-rw-------   1 root     system         26 May 11 16:03 bayes.lock
-rw-------   1 root     system      13899 May 11 16:04 bayes_journal
-rw-------   1 root     system     579653 May 11 16:03 bayes_journal.old
-rw-r--r--   1 root     system   10223616 May 11 15:58 bayes_seen
-rw-------   1 root     system    4849664 May 11 16:04 bayes_toks

Now that lockfile's gone and all looks normal again.  But you can see
clearly that these files ARE being updated on the fly in spite of
"bayes_auto_learn" being turned off.

Any help would be greatly appreciated.  We're at crunch time here and
students are e-mailing papers and the such like and delays like I'm seeing
now are a killer.  Thanks in advance.

Don Newcomer
Senior Manager, Systems
Infrastructure Systems Department
Library and Information Services
Dickinson College
P.O. Box 1773
Carlisle, PA  17013
717-245-1256 (Voice)
717-245-1690 (FAX)
newcomer@dickinson.edu

Re: Bayes database

[Matt Kettler <mk...@evi-inc.com>]

At 04:35 PM 5/11/2004, Don Newcomer wrote:
>Okay, I'll buy that.  I'm still really concerned that I'll hit another
>deadlock situation like happened earlier this afternoon.  I can see it
>generating a lockfile to do an expiry but why would I have had six or seven
>lockfiles unless they were separate expiry runs (every 5 minutes or so it
>appears based on that date changes on the Bayes files).  Why did this
>deadlock happen and how can I prevent it in the future?

Typical problem for MailScanner users.. I bet if you check your logfile 
there are a bunch of messages telling you spamassassin timed out and was 
killed.

MailScanner is very impatient with SpamAssassin and will kill it if it 
takes more than a few seconds to run.

Add this to your spam.assassin.prefs.conf:
#don't do auto-expiry on a MailScanner system
bayes_auto_expire 0

And then run sa-learn --force-expire as a daily cronjob, as the same user 
that owns your bayes database.


>pertinent Bayes entries from local.cf (which is a link to
>MailScanner/etc/spam.assassin.prefs.conf for MailScanner):

I just noticed this.. DO NOT link local.cf to spam.assassin.prefs.conf.

MailScanner won't inhibit local.cf. MailScanner inhibits user_prefs and 
replaces it with spam.assassin.prefs.conf. By symlinking your local.cf onto 
your spam.assassin.prefs.conf, you're forcing MailScanner to reparse the 
exact same file twice, wasting time and resources. (and possibly causing 
crashes if future versions of SA do some kind of weird parallel parsing tricks)

>bayes_path                 /var/spool/spamassassin/bayes
>bayes_file_mode            0600

I also noticed this.. I'd strongly suggest that you NEVER force a global 
bayes in local.cf unless you use mode 666. NEVER. Under ANY condition.

The above line is causing your problems, and it's fundamentally not needed 
when using MailScanner. MailScanner always runs as one user, so do it in 
your spam.assassin.prefs.conf and NOT local.cf.

Symlink or copy spam.assassin.prefs.conf to root's user_prefs so you can do 
training but DO NOT link it to local.cf.

By doing it in local.cf you've fundamentally forced SA to only ever be 
invoked as one user without breaking the whole system. Not a good idea.

Note: the warnings in the SA docs about not honoring bayes_path in 
user_prefs only seems to apply to spamd. sa-learn, spamassassin and 
MailScanner are just fine with it.

Re: Bayes database

[Don Newcomer <ne...@dickinson.edu>]

Okay, I'll buy that.  I'm still really concerned that I'll hit another
deadlock situation like happened earlier this afternoon.  I can see it
generating a lockfile to do an expiry but why would I have had six or seven
lockfiles unless they were separate expiry runs (every 5 minutes or so it
appears based on that date changes on the Bayes files).  Why did this
deadlock happen and how can I prevent it in the future?

                                                Don


On Tue, 11 May 2004, Matt Kettler wrote:

> At 04:07 PM 5/11/2004, Don Newcomer wrote:
> >What am I missing that it's still creating lockfiles?  Here's what
> >/var/spool/spamassassin looked like a few minutes ago:
>
> opportunistic expiry.
>
>
>

Re: Bayes database

[Matt Kettler <mk...@evi-inc.com>]

At 04:07 PM 5/11/2004, Don Newcomer wrote:
>What am I missing that it's still creating lockfiles?  Here's what
>/var/spool/spamassassin looked like a few minutes ago:

opportunistic expiry.