forcing 2 SA (on 2 hosts) to learn the same spam/ham ?

[Oliver Thalmann <Ol...@hospvd.ch>]

hi,

i have a setup in which i have 2 MX hosts for incoming
internet email, each one with it's own Spamassassin (+amavisd)

over time, those 2 systems have independently learnt
(Bayes autolearn) what is considered spam/not-spam.

Of course, now, those systems sometimes have quite
a different bayesian opinion for the same mail. For
example, some nigerian scam scores a bayes_90 on
one host, while on the other it scores a bayes_50.

i know the question has already been asked before,
but it doesn't look as if there has been a real solution
proposed.

is it possible somehow (has anyone already done it)
to "merge" the 2 databases into 1 "master reference",
which could then be  recopied over the 2 independent
databases, thus reverting to a common "opinion" on
what is bayesian spam and what is not.

alternatively, as SA is learning to a journal, i think
it could be possible to do the following procedure
(your opinion are welcome) :

a) just before SA syncs the journal, it renames it
to bayes_journal.old (this is already done, it seems)

b) make a copy of this old journal file to a spool
directory (and rename the file to a unique name, so
it won't clash with other old journals), before SA
sync it to the db.

c) have a cron job which regularly copies these spooled
journals to the second node

d) on the second node, either have some tool, running via
cron, which can sync these journal files to the local db on it's
own (does such a tool already exist, or do you have any hints
howto write such a tool ?), or else patch SA code to be able
to sync multiple journals (instead of only "bayes_journal.old").

what do you think ?

maybe this point of "common" bayes learning will become
"unimportant" (i don't remember the right word now) in
SA 3.x (or was it 2.70) when the Bayes DB can be stored
in a real SQL database, multiple hosts should the be able to
write to the same database ?

Re: forcing 2 SA (on 2 hosts) to learn the same spam/ham ?

[Robert LeBlanc <rj...@renaissoft.com>]

At 05:32 2004/03/29, Oliver Thalmann wrote:
>Robert LeBlanc wrote:
> > If your users are "confirming" the status of the mail they receive as spam
> > or ham somehow (e.g. with a quarantine management mechanism of some kind),
> > then you don't need to use auto-learning at all.  A script that runs at
> > scheduled intervals can run sa-learn on the confirmed spam and confirmed
> > ham in order to train your Bayes database.  Copy that database (read-only)
> > to all of your nodes, and you're pretty much done.
>
>another idea, not having autolearn enabled on the 2 receiving boxes, and
>somehow pull out the (quarantined by amavisd) mails that else would have
>been autolearned by SA (how ..?), then simultaneously train bayes on both
>nodes with those mails via a cronjob

This is essentially the same idea, except that instead of copying the 
(trained) Bayes database from the master to the slaves, you'd be copying 
all the spam and ham and doing the training on each box.  There are 
efficiency tradeoffs in either case, but both methods should work.


> > >maybe this point of "common" bayes learning will become
> > >"unimportant" (i don't remember the right word now) in
> > >SA 3.x (or was it 2.70) when the Bayes DB can be stored
> > >in a real SQL database, multiple hosts should the be able to
> > >write to the same database ?
> >
> > Yes, that's one of the ideas behind moving the Bayes database to an SQL
> > server--it can be more easily shared across an array of content filters,
> > without having to manage filesystem-level sharing, lockfiles, copying
> > databases, etc.
>
>will it be possible to do the update writes (multiple SA instances) to one
>master db, but do the reads from (replicated & local) slave db's ?

In theory it should be possible, yes.  Your master SA/amavis box would have 
to connect to the master database (for reads and writes), while your slave 
SA/amavis boxes could connect to replicated databases (for reads).


Robert LeBlanc <rj...@renaissoft.com>
Renaissoft, Inc.
Maia Mailguard <http://www.renaissoft.com/maia/> 

Re: forcing 2 SA (on 2 hosts) to learn the same spam/ham ?

[Oliver Thalmann <Ol...@hospvd.ch>]

Robert LeBlanc wrote:

>
> A better solution is to just maintain one "master" database on one of these
> boxes, and make all of the other machines "slaves" that share the master's
> database.  There are at least a couple of different ways to do this:

somehow, in a two box scenario which are supposed to be "twins", i don't
like the idea having one the two boxes being a "master" compared to the
other. because if the master crashes, the slave will be somewhat lost
(slave loses the remote fs mount, etc.), and should have some local
recovery procedure (use a local backup of the bayes database, or such)
in order to continue to work correctly

an independent "db master" node, which relies on rsync/filecopy methods to
synchronise the slaves look a lot more solid an fault-tolerant (if the master
crashes, the slaves just won't get any updated db)

the sql bayes implementation should be very interesting for this (Master-Slave
replication done on the database level)

> (1) You can use NFS, Samba, or some other remote file system to mount the
> master's database on each of the slaves.  In read/write mode, all of the
> boxes can auto-learn, though the reliability of NFS locks may be an issue
> to contend with.  All mistake-based training should take place on the master.

see above (if the master crashes, the slave has to have some kind of recovery
scenario, and if using nfs, the slave, at least it SA processes, be it amavis or
spamd,
will probably hang until nfs times out, i think..., what happens after
the timeout could be interesting)

>
> (2) You can choose to do all of your learning only on the master, and then
> use something like rsync to send read-only copies to the slaves at regular
> intervals.  The slaves should have auto-learning disabled, and all
> mistake-based training should take place on the master.

that would be an idea. In my scenario, this would mean
to place a supplemental SA/amavisd (slightly trimmed, without the DNS
blacklists, for example, as they are not user for the "learning score", if i
remember right) on the mailserver or another independent machine
which would rescan all email that got through, learn it as ham/spam into it's
own master db, then periodically copy the master db to both slaves.

Of course, it could (better solution ?) also be an independent box, which
receives a copy of every mail that got through (postfix's "always bcc" could
be handy), just for scanning/learning purposes

so at least, if this master crashes, both MX still receive and filter emails,
they just won't be up to date with the bayesian stuff (and if the 3rd
instance is the mailserver itself, they won't also be able to relay the
emails, but who cares at this point)

i'll investigate these solutions...

>
> If your users are "confirming" the status of the mail they receive as spam
> or ham somehow (e.g. with a quarantine management mechanism of some kind),
> then you don't need to use auto-learning at all.  A script that runs at
> scheduled intervals can run sa-learn on the confirmed spam and confirmed
> ham in order to train your Bayes database.  Copy that database (read-only)
> to all of your nodes, and you're pretty much done.

another idea, not having autolearn enabled on the 2 receiving boxes, and
somehow pull out the (quarantined by amavisd) mails that else would have
been autolearned by SA (how ..?), then simultaneously train bayes on both
nodes with those mails via a cronjob

>
> >maybe this point of "common" bayes learning will become
> >"unimportant" (i don't remember the right word now) in
> >SA 3.x (or was it 2.70) when the Bayes DB can be stored
> >in a real SQL database, multiple hosts should the be able to
> >write to the same database ?
>
> Yes, that's one of the ideas behind moving the Bayes database to an SQL
> server--it can be more easily shared across an array of content filters,
> without having to manage filesystem-level sharing, lockfiles, copying
> databases, etc.

will it be possible to do the update writes (multiple SA instances) to one
master db, but do the reads from (replicated & local) slave db's ?

Re: forcing 2 SA (on 2 hosts) to learn the same spam/ham ?

[Robert LeBlanc <rj...@renaissoft.com>]

At 14:03 2004/03/27, Oliver Thalmann wrote:

>is it possible somehow (has anyone already done it)
>to "merge" the 2 databases into 1 "master reference",
>which could then be  recopied over the 2 independent
>databases, thus reverting to a common "opinion" on
>what is bayesian spam and what is not.

In a multi-node scenario, where you have an array of content filtering 
boxes, you don't really want to be doing auto-learning on each box 
independently.  As you point out, this leads to each box developing its own 
unique Bayes database, such that one box ends up better able to detect 
particular kinds of spam than the others.  Another problem is that this 
makes mistake-based training more complicated, since you have to do your 
training corrections on the box that made the mistake.

A better solution is to just maintain one "master" database on one of these 
boxes, and make all of the other machines "slaves" that share the master's 
database.  There are at least a couple of different ways to do this:

(1) You can use NFS, Samba, or some other remote file system to mount the 
master's database on each of the slaves.  In read/write mode, all of the 
boxes can auto-learn, though the reliability of NFS locks may be an issue 
to contend with.  All mistake-based training should take place on the master.

(2) You can choose to do all of your learning only on the master, and then 
use something like rsync to send read-only copies to the slaves at regular 
intervals.  The slaves should have auto-learning disabled, and all 
mistake-based training should take place on the master.

If your users are "confirming" the status of the mail they receive as spam 
or ham somehow (e.g. with a quarantine management mechanism of some kind), 
then you don't need to use auto-learning at all.  A script that runs at 
scheduled intervals can run sa-learn on the confirmed spam and confirmed 
ham in order to train your Bayes database.  Copy that database (read-only) 
to all of your nodes, and you're pretty much done.


>maybe this point of "common" bayes learning will become
>"unimportant" (i don't remember the right word now) in
>SA 3.x (or was it 2.70) when the Bayes DB can be stored
>in a real SQL database, multiple hosts should the be able to
>write to the same database ?

Yes, that's one of the ideas behind moving the Bayes database to an SQL 
server--it can be more easily shared across an array of content filters, 
without having to manage filesystem-level sharing, lockfiles, copying 
databases, etc.


Robert LeBlanc <rj...@renaissoft.com>
Renaissoft, Inc.
Maia Mailguard <http://www.renaissoft.com/maia/>