You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by R-Elists <li...@abbacomm.net> on 2010/04/14 17:43:21 UTC
flat file bayes locking issue and difference errors depending on file locking method
greetings :-)
config is centos4 SA 3.3.1 upgraded from SA 3.2.5
having spent the better part of a two days searching as well as trying
different configs and SA restarts
no good results
we do not have a "hardware horsepower" resource starvation issue
this machine does *not* use SQL for Spamassassin at this time
i have tried many different possible SPAMDOPTIONS for SA startup for regular
and round-robin and thrown tons of hardware and software resources at the
issue
in the /home/spamd/.spamassassin directory we have
bayes_journal
bayes_mutex
bayes_seen
bayes_toks
in reference to the error
spamd[30339]: bayes: cannot open bayes databases
/home/spamd/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
what is bayes_mutex ?
is bayes_seen necessary ? (i seem to recall it is not and can be deleted)
if bayes_seen is large, isnt that the file we can delete and it will not
make a difference?
i did back up the database using sa-learn before the upgrade...
should i stop spamd, restore bayes info and then restart spamd ?
other options to preserve bayes?
...or should i stop SA, whack the files, and restart and retrain?
tia
- rh
notes:
when using flock as the file locking in /etc/mail/spamassassin/local.cf we
get
spamd[2489]: bayes: cannot open bayes databases
/home/spamd/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
spamd[2489]: bayes: cannot open bayes databases
/home/spamd/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
when using default SA locking method we get this error
spamd[19334]: bayes: cannot open bayes databases
/home/spamd/.spamassassin/bayes_* R/W: lock failed: File exists
spamd[19337]: bayes: cannot open bayes databases
/home/spamd/.spamassassin/bayes_* R/W: lock failed: File exists
Re: flat file bayes locking issue and difference errors depending on
file locking method
Posted by Alex <my...@gmail.com>.
Hi,
> spamd[30339]: bayes: cannot open bayes databases
> /home/spamd/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
>
> what is bayes_mutex ?
Many years ago Matt wrote this post that describes it:
http://lists.mailscanner.info/pipermail/mailscanner/2004-November/043067.html
"In short, a mutex is a MUTual EXclusion. It's used to lock access to
some piece of data so you don't run into consistency problems where
two different threads are both trying to update the same data and one
winds up stomping on the changes of the other."
> is bayes_seen necessary ? (i seem to recall it is not and can be deleted)
>
> if bayes_seen is large, isnt that the file we can delete and it will not
> make a difference?
Here's a post from Matt quite a while ago where he says that it's okay
to delete it, but doesn't really say what the implications are -- will
you effectively then lose its ability to recognize patterns?
http://markmail.org/message/ju6424xy6r2doslb
> should i stop spamd, restore bayes info and then restart spamd ?
It sounds like you either have multiple copies of SA running at the
same time, or lock files aren't being deleted after the process
closes.
> other options to preserve bayes?
You could always shut down spamd, make a physical copy of it, then
restart. You can also use sa-learn to back it up:
# sa-learn --backup > /var/backup/bayes-backup.sa
> ...or should i stop SA, whack the files, and restart and retrain?
What is the actual problem you're having, outside of the locking
errors? Or did I somehow miss that written in your post...
> spamd[2489]: bayes: cannot open bayes databases
> /home/spamd/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
What's the status of the system? Are you sure the integrity of it
(disks, RAM, etc) are sound? Have you run memtest?
> spamd[19334]: bayes: cannot open bayes databases
> /home/spamd/.spamassassin/bayes_* R/W: lock failed: File exists
Have you stopped spamd, deleted these files, then restarted? They are
just temporary lock files and should be refreshed and deleted as
necessary.
Best,
Alex
Re: flat file bayes locking issue and difference errors depending on
file locking method
Posted by Alex <my...@gmail.com>.
> not happy about losing bayes yet maybe it is time to migrate to SQL
>
> im guessing that SA SQL setup is easy ???
>
> anyone care to chime in?
I'm also interested in this, so thought I would follow up. Can anyone
provide any insight into how difficult it would be to convert from a
local berkeley db to mysql? Is it possible for two hosts to update the
database once that is done?
Are the memory and IO requirements nearly the same? How does
performance compare?
Thanks,
Alex
RE: flat file bayes locking issue and difference errors depending on file locking method
Posted by R-Elists <li...@abbacomm.net>.
> >
> That was going to be my guess, too. You're not swapping, or
> having some other i/o issue are you?
>
> /Jason
>
no sir
i shutdown spamassassin
backed it all up
dusted bayes
started spamassassin
retrained 200 plus of each
seems ok so far...
3.2.5 was working awesome overall yet wanted to be able to move forward with
the current stable dev (so to speak)
not happy about losing bayes yet maybe it is time to migrate to SQL
im guessing that SA SQL setup is easy ???
anyone care to chime in?
- rh
Re: flat file bayes locking issue and difference errors depending
on file locking method
Posted by Jason Bertoch <ja...@i6ix.com>.
On 4/14/2010 4:59 PM, R-Elists wrote:
>> I'd guess that you have a bayes expire running that is either
>> taking too long or not finishing and leaving lock files around.
>>
>> Turn off bayes_auto_expire and use bayes_learn_to_journal.
>> Add a cron job to periodically sa-learn --sync (say hourly)
>> and another cron job to do sa-learn --force-expire
>> (daily/weekly) -jeff
>>
> thank you for the info and your time... :-)
>
> we do have the bayes_auto_expire turned off
>
> and the forced expire is done at off peak hours once a day
>
That was going to be my guess, too. You're not swapping, or having some
other i/o issue are you?
/Jason
RE: flat file bayes locking issue and difference errors depending on file locking method
Posted by R-Elists <li...@abbacomm.net>.
>
> I'd guess that you have a bayes expire running that is either
> taking too long or not finishing and leaving lock files around.
>
> Turn off bayes_auto_expire and use bayes_learn_to_journal.
> Add a cron job to periodically sa-learn --sync (say hourly)
> and another cron job to do sa-learn --force-expire
> (daily/weekly) -jeff
>
thank you for the info and your time... :-)
we do have the bayes_auto_expire turned off
and the forced expire is done at off peak hours once a day
- rh
Re: flat file bayes locking issue and difference errors depending on file locking method
Posted by Jeff Mincy <je...@delphioutpost.com>.
From: "R-Elists" <li...@abbacomm.net>
Date: Wed, 14 Apr 2010 08:43:21 -0700
having spent the better part of a two days searching as well as trying
different configs and SA restarts
we do not have a "hardware horsepower" resource starvation issue
in reference to the error
spamd[30339]: bayes: cannot open bayes databases
/home/spamd/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
I'd guess that you have a bayes expire running that is either taking
too long or not finishing and leaving lock files around.
Turn off bayes_auto_expire and use bayes_learn_to_journal.
Add a cron job to periodically sa-learn --sync (say hourly)
and another cron job to do sa-learn --force-expire (daily/weekly)
-jeff
RE: flat file bayes locking issue and difference errors depending on file locking method
Posted by R-Elists <li...@abbacomm.net>.
>
> notes:
>
> when using flock as the file locking in
> /etc/mail/spamassassin/local.cf we get
>
> spamd[2489]: bayes: cannot open bayes databases
> /home/spamd/.spamassassin/bayes_* R/W: lock failed:
> Interrupted system call
> spamd[2489]: bayes: cannot open bayes databases
> /home/spamd/.spamassassin/bayes_* R/W: lock failed:
> Interrupted system call
>
> when using default SA locking method we get this error
>
> spamd[19334]: bayes: cannot open bayes databases
> /home/spamd/.spamassassin/bayes_* R/W: lock failed: File exists
> spamd[19337]: bayes: cannot open bayes databases
> /home/spamd/.spamassassin/bayes_* R/W: lock failed: File exists
>
>
:-)
apologies for replying to my own post...
....things i forgot to mention and that we are still investigating...
the errors appear to be happening when SA is scanning longer than normal...
ie, a normal scan used to only take a few seconds...
these file locking errors *appear* to be happening when a scan takes 5 to 20
times or more times longer
again, still investigating...
before we upgraded this machine from 3.2.5 to 3.3.1 scan times averaged 2 to
4 seconds per email
now, the average scan time is more like 8 to 12 seconds.
any pointers to newer default knobs and handles and buttons will be
appreciated...
- rh