Re: SpamAssassin tested by lwn.net

[Justin Mason <jm...@jmason.org>]

Matt Kettler writes:
> Michael Monnerie wrote:
> > http://lwn.net/SubscriberLink/173910/e7bf95a7cb044637/
> >
> > They are wondering why bayes_99 is not given 5 points by default, as it 
> > seems to have no FP.
> 
> Statisticaly speaking, 1% of BAYES_99 hits should be nonspam.In reality,
> it does a lot better than that.
> 
> However, in the SA 3.1.0 set3 mass checks it still managed to match
> about 21 messages in the nonspam test set:
> 
> OVERALL%   SPAM%     HAM%     S/O    RANK   SCORE  NAME
>  176869   123778    53091    0.700   0.00    0.00  (all messages)
> 60.712  86.7351   0.0396    1.000   0.90    3.50  BAYES_99
> 
> SA's scores aren't based on human assumptions about how the rules
> behave. They are based on real-world testing and a perceptron
> score-fitting system that accounts not only for the hit-rate of the
> rule, but also for the combinations of rules that it tends to match
> with. Often the reality is a lot more complex than you think.

It's important to note that, without good training, BAYES_99 may indeed
fire regularly on nonspam mail -- that's the danger with user-trained
rules.  In the *default* scenario, therefore, a score of 3.5 is reasonably
optimal.   However, if good training is supplied, it's a good plan to
increase the BAYES_99 score to 5.0, or even more.  (I think we might
mention that somewhere in the documentation -- I hope. ;)

Also, it's worth noting that "BAYES_99" doesn't really refer to a 1%
probability.   SpamAssassin uses the Fisher Inverse Chi-Square Procedure
described at http://garyrob.blogs.com/whychi90.pdf , and as a result these
are no longer true probability values -- so don't expect to see
probabilistic distributions.

Great articles btw.  The grumpy editor has outdone himself ;)
[I've posted this as a comment on the story already btw.]

--j.