Re: Scoring [was: How to report FPs to Spamcop?]

[Robert Menschel <Ro...@Menschel.net>]

Hello Al,

Tuesday, April 13, 2004, 1:36:25 PM, you wrote:

>> The whole point of SpamAssassin's score based approach is that no
>> single test is enough to push a message over the threshold, 
 
AD> Second: We don't necessarily share the SA scoring philosophy. (Here's
AD> where this becomes a different thread.) 

AD> It seems there are a couple of fundamental flaws in the SA no single
AD> rule philosophy.

Flaws, no.  Opinions about it, yes.

AD> 1. If a rule has a strong history of only hitting SPAM - why not
AD> score it at least at threshold until it produces fp's? Not every
AD> message will hit that rule, it will only be effective against the
AD> subset that do hit it.

Goal:  No FPs. Zero, zip, zilch. Nada, nothin', nohow.  'Tis better to
receive spam than to not receive ham.

Yes, by all means increase scores; but unless a rule is GUARANTEED! to
hit only spam, keep that score under the required_hits threshold.

Example:  I have a number of obfuscation rules, such as
body      RM_bwo_Free    /(?!\bfree\b)(?!\bf\.r\.e\.e\b)(?:\bf|\B(?:\xC5\xBF|\xC6\x92|\xD2[\x92-\x93])) ...
describe  RM_bwo_Free    masked spam word(s)
score     RM_bwo_Free    8.000 # type=obfu - 1234s/0h of 100793 corpus (82099s/18694h) 02/21/04

I run at a required_hits threshold of 9 -- this spam-only rule, which
hits over 1k spam and never any ham, scores 89% of that threshold.

Why?  Notice that second exclusion: (?!\bf\.r\.e\.e\b)
That wasn't in the rule two months ago, not until I received ham with
that w.o.r.d in it (from my f.a.t.h.e.r, wouldn't you know -- ever try
explaining to your father that you didn't receive his email because it
got flagged as spam?).

AD> 2. Email score distribution. What we are trying to do is identify 2
AD> discrete species of email among the email population - HAM and SPAM.
AD> Bell curve territory - although skewed distribution is OK. Ideally
AD> the HAM will be almost all clustered around 0 and tail way off
AD> approaching the hit threshold. The key part is tailing off to near
AD> zero as it approaches the threshold to avoid fp's. SPAM would bell
AD> curve somewhere far enough above threshold that it tails off
AD> approaching threshold. Again the key here is to approach zero as it
AD> approaches the threshold to avoid fn's.

That would be ideal.

AD> What actually happens is somewhat different: HAM clustered at 0 and
AD> -100 with whitelists. SPAM kind of spreads out above the threshold
AD> with another cluster around 100 with blacklists.) The problem is that
AD> HAM and SPAM #'s do not tail off as they approach the threshold. That
AD> means there is a bunch of HAM that is only 1 rule away from being
AD> scored as SPAM - fp's and a bunch of SPAM that is only 1 less rule
AD> away from being scored as HAM.

Actually, here I have very, very few ham that score above 50% of
threshold, I'm guessing around 10%, dropping to under 0.1% within 1.0 of
threshold, and very, very few spam that score under threshold (99.8% of
all spam is correctly flagged here). I believe my SA system runs close
to your ideal (allowing for a goodly number of 100+ scores due to
blacklists, and 1% -100 scores due to whitelists).

AD> Maybe Bayes scores things well enough to pull the distributions
AD> apart, I'm skeptical - especially with the increasing frequency of
AD> Bayes poison. I would like to see the email score distribution of
AD> some Bayes systems.

IMO (I've said it before, and I'll say it again): Bayes poison doesn't
exist. With the increasing frequency of Bayes fodder, I'm finding that
Bayes catches more and more and more of the spam without needing new
rules all the time.

I'd love to give you stats to back up my impessions, but other than the
percentages of FPs (1 FP every 2-3 months) and FNs (0.2%), I don't have
any that are reliable.

AD> Anyway, I was going through the rules according to how many messages
AD> they hit, looking for potential score adjustments that would
AD> eliminate false negatives. The RCVD_IN_BL_SPAMCOP hits a lot of
AD> messages for us and would be a good candidate for score increase. It
AD> does get some fp's which I could either deal with by trying to get
AD> the sender(s) off Spamcop, or by whitelisting.

I've found SA's mass-check capabilities to be a big help here. I run a
comprehensive mass-check against my comprehensive ruleset here, and
rework scores accordingly. I do this every 2 months or so.

Bob Menschel