training SpamAssassin without updating bayes*

[Gabriel Wachman <ga...@tufts.edu>]

A colleague and I are writing a paper about a spam filter he developed.
We'd like to compare it against various open source filters, including
SpamAssassin. The methodology we are using is to train the filter on a
set of messages, and then test it on an independent set of messages. The
key is that the filter cannot update itself at all after training.

In my user_prefs:
bayes_auto_learn        0
bayes_learn_during_report       0
bayes_path SOME_PATH

During training I run:
sa-learn --dbpath $WORKDIR --ham $DATADIR/$message_dir
(likewise for spam)

During testing I run:
spamassassin -t -p $PREFSPATH $DATADIR/$message_dir

I'm running several testing and training runs, so for each one I specify
a different database (by setting "SOME_PATH" appropriately and
specifying that "user_prefs" using the -p switch), hence the variables
for certain command-line arguments. The matching testing run for a given
training run must read the bayes_* files from that training run.

During testing, I can see spamassassin create a "bayes_journal" file and
write to it continuously. I understand this is spamassassin's way of
storing its updates to bayes_* temporarily until the updates are merged.
My concern is that it's using bayes_journal in addition to bayes_toks
and bayes_seen during testing, but I just want it to use the bayes_toks
and bayes_seen generating during training.

Can someone tell me how to run spamassassin in testing mode, without
making any updates or doing any learning, but only classifying messages?

Thank you,
Gabriel

Re: training SpamAssassin without updating bayes*

[Theo Van Dinter <fe...@apache.org>]

On Sat, Mar 04, 2006 at 10:50:19PM -0500, Daryl C. W. O'Shea wrote:
> Even with bayes_auto_learn disabled, the tokens' atimes are still 
> updated.  That's the way SpamAssassin works.  That's what helps 
> SpamAssassin's bayes implementation in being effective.

Well, sort of.  The atime updates are *solely* for the use of the
expiration system.  They have no function wrt the effectiveness of the
classifier.  If we didn't care about expiring out tokens, we wouldn't
need to track the atimes.  However, there's no way to disable the atime
updates without going into the code and doing it forcibly.

-- 
Randomly Generated Tagline:
"I'm convinced that the body metal for this car was supplied by Reynold's
 Aluminum." - Unknown about the Renault LeCar

Re: training SpamAssassin without updating bayes*

[mouss <us...@free.fr>]

Gabriel Wachman a écrit :
> 
> Yes. I know it may sound strange from some people's perspective, but
> there are good reasons we need to do it this way. We are comparing
> several spam filters; in order to make claims about the performance of
> any of the filters we need to evaulate a _fixed_ classifier on a test
> set. If the classifier is not fixed, then our confidence intervals go
> out the window. It actually helps SpamAssassin if we can do this because
> if we can't, we need to mention in the paper that any results from
> SpamAssassin are not statistically robust since it changes its
> classifier during training. Since SpamAssassin is so widely used and in
> my experience performs very well, we would really like to include
> results from it without any such caveats.
> 
> I hope that helps explain the situation. Regardless, our testing
> methodology is really not up for discussion, I just want to know if
> there is an easy way to do what we want.
> 

All adaptive filters change learn. consider auto learn as an additionnal
Bayesian feature. SA uses non textual attributes collected by its rules.
If you remove the ability to use these attributes, you are removing a
possibly helpful feature (whether this feature works well or not,
whether it is theoritically sound, ... are other questions).

Or do you want to compare SA's "heuristic" rules only (disabling both
Bayes and auto_learn) to other filters? If so, just disable these. Of
course, I assume you'll signal this in your report:)

Re: training SpamAssassin without updating bayes*

[Gabriel Wachman <ga...@tufts.edu>]

Daryl C. W. O'Shea wrote:
> On 04/03/06 09:56 PM, Gabriel Wachman wrote:
> 
>> A colleague and I are writing a paper about a spam filter he developed.
>> We'd like to compare it against various open source filters, including
>> SpamAssassin. The methodology we are using is to train the filter on a
>> set of messages, and then test it on an independent set of messages. The
>> key is that the filter cannot update itself at all after training.
> 
> 
> That's the key?!

Yes. I know it may sound strange from some people's perspective, but 
there are good reasons we need to do it this way. We are comparing 
several spam filters; in order to make claims about the performance of 
any of the filters we need to evaulate a _fixed_ classifier on a test 
set. If the classifier is not fixed, then our confidence intervals go 
out the window. It actually helps SpamAssassin if we can do this because 
if we can't, we need to mention in the paper that any results from 
SpamAssassin are not statistically robust since it changes its 
classifier during training. Since SpamAssassin is so widely used and in 
my experience performs very well, we would really like to include 
results from it without any such caveats.

I hope that helps explain the situation. Regardless, our testing 
methodology is really not up for discussion, I just want to know if 
there is an easy way to do what we want.

Thank you all for your replies.

Gabriel

Re: training SpamAssassin without updating bayes*

["Daryl C. W. O'Shea" <sp...@dostech.ca>]

On 04/03/06 09:56 PM, Gabriel Wachman wrote:
> A colleague and I are writing a paper about a spam filter he developed.
> We'd like to compare it against various open source filters, including
> SpamAssassin. The methodology we are using is to train the filter on a
> set of messages, and then test it on an independent set of messages. The
> key is that the filter cannot update itself at all after training.

That's the key?!

The reality is SpamAssassin CAN update itself after initial training. 
Doing one set of tests with bayes_auto_learn disabled is fine, but not 
also doing a set of tests with it enabled is warping reality.

<snip config>

> During testing, I can see spamassassin create a "bayes_journal" file and
> write to it continuously. I understand this is spamassassin's way of
> storing its updates to bayes_* temporarily until the updates are merged.
> My concern is that it's using bayes_journal in addition to bayes_toks
> and bayes_seen during testing, but I just want it to use the bayes_toks
> and bayes_seen generating during training.

Even with bayes_auto_learn disabled, the tokens' atimes are still 
updated.  That's the way SpamAssassin works.  That's what helps 
SpamAssassin's bayes implementation in being effective.


> Can someone tell me how to run spamassassin in testing mode, without
> making any updates or doing any learning, but only classifying messages?

Make the files read-only or hack at the code, or break both it's legs 
and left arm. :)

Just be sure to make sure you're clear in your paper that you've 
intentionally hampered SpamAssassin's ability to classify mail.


Daryl

Re: training SpamAssassin without updating bayes*

[jdow <jd...@earthlink.net>]

From: "mouss" <us...@free.fr>

> Gabriel Wachman a écrit :
>> A colleague and I are writing a paper about a spam filter he developed.
>> We'd like to compare it against various open source filters, including
>> SpamAssassin. The methodology we are using is to train the filter on a
>> set of messages, and then test it on an independent set of messages. The
>> key is that the filter cannot update itself at all after training.
>>
>> In my user_prefs:
>> bayes_auto_learn        0
>> bayes_learn_during_report       0
>> bayes_path SOME_PATH
>
> why disable auto learning? Are you sure what you want isn't:
>
> - run in supervised mode (human to correct the decisions) on N messages
> (the training set)
>
> and then
>
> - run in unsupervised mode (no human correction) on M other messages
> (The validation set)

Um, the cynic in me suggests we're seeing test conditions being tuned
to make a specific outcome appear.

{o.o} 

Re: training SpamAssassin without updating bayes*

[mouss <us...@free.fr>]

Gabriel Wachman a écrit :
> A colleague and I are writing a paper about a spam filter he developed.
> We'd like to compare it against various open source filters, including
> SpamAssassin. The methodology we are using is to train the filter on a
> set of messages, and then test it on an independent set of messages. The
> key is that the filter cannot update itself at all after training.
> 
> In my user_prefs:
> bayes_auto_learn        0
> bayes_learn_during_report       0
> bayes_path SOME_PATH

why disable auto learning? Are you sure what you want isn't:

- run in supervised mode (human to correct the decisions) on N messages
(the training set)

and then

- run in unsupervised mode (no human correction) on M other messages
(The validation set)

Re: training SpamAssassin without updating bayes*

[Theo Van Dinter <fe...@apache.org>]

On Sat, Mar 04, 2006 at 09:56:14PM -0500, Gabriel Wachman wrote:
> During training I run:
> sa-learn --dbpath $WORKDIR --ham $DATADIR/$message_dir
> (likewise for spam)
> 
> During testing I run:
> spamassassin -t -p $PREFSPATH $DATADIR/$message_dir

You may want to look into mass-check.  It's much better for this type
of thing.

> During testing, I can see spamassassin create a "bayes_journal" file and
> write to it continuously. I understand this is spamassassin's way of

Yes.

> storing its updates to bayes_* temporarily until the updates are merged.

Yes.

> My concern is that it's using bayes_journal in addition to bayes_toks
> and bayes_seen during testing, but I just want it to use the bayes_toks
> and bayes_seen generating during training.

If you disable auto_learn, and you don't use sa-learn afterwards, there
won't be any other learning.  The journal updates (by default) are only
atime values used for expiry.  You can't disable them, though you can
disable the journal sync if you want to ("bayes_journal_max_size 0").

Hope this helps. :)

-- 
Randomly Generated Tagline:
"I spaced out on lingo."         - Prof. Wills

Re: training SpamAssassin without updating bayes*

[Loren Wilton <lw...@earthlink.net>]

> During testing, I can see spamassassin create a "bayes_journal" file and
> write to it continuously. I understand this is spamassassin's way of

If the journal is only growing it isn't being learned from.

Typically at some point if auto-learn were enabled one of the spam mail runs
would take some extra time and update the bayes store.  Since you have
auto-learn disabled, this isn't happening.  Typically in a real-world
situation, the admin would have set up a cron job to run sa-learn and do the
update.


I would strongly suggest in your paper when comparing your Bayes technique
to SA's, that you EMPHASIZE that your comparison was ONLY on Bayes ability,
and not the overall ability of the product.  A very large part of SA is the
ability to classify spam based on rules, and quite a lot of that benefit can
come from running net tests.

So if your thing is only a Bayes tool and it is twice as good as SA's Bayes
module, it would be somewhat disingenious to claim "our spam tool is twice
as good as SpamAssassin", rather than "our tool is twice as good as the
Bayes module of SpamAssassin".

        Loren