You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Joe Flowers <fl...@social.chass.ncsu.edu> on 2005/07/10 14:13:46 UTC
update on floating dividing score between spam and ham messages
I don't know if this will help anyone or not, but I wanted to report
back just in case.
In early April, I completely unhinged the dividing line between what SA
score is used to mark a message as spam or ham (5.00 = default). This
allows the system and this dividing line to drift "freely" to anywhere
that SA will allow, without bound. This anti-spam setup has worked
consistently much much better the whole time than in any previous
implementation that we have done and with very little maintenance. We
are very happy with it and are looking forward to implementing future SA
versions in the same fashion.
I'm not exactly sure the following numbers represent the whole time
since April, but they should be pretty close.
We've had 360,922 spam messages and 396,983 ham messages with a
normalized average spam score of 6.8714134 and a normalized average ham
score of -2.1532284. I have the divding line "set" at 30% of the
distance between the average ham score and average spam score (30% above
the average ham score). So, the dividing line is currently floating
around 0.55416414.
Apart from the default SA install, the only thing I have changed is
1. Turned off auto-learn <--- I think this is very important.
2. Set SA to ignore our custom spam score tag in the message headers.
We are currently running SA v3.02.
From time to time, but not very often (a couple of times every two
weeks or so), I do feed bayes (sa-learn) with a few messages that are
misplaced. I don't know the stats, but we have very few false positives,
so I'm mostly feeding bayes with the false negatives which consist of
the new/different message tricks that the spammers are using.
Everyone here has been very happy with the results. It's been much much
better than any implementation in the past.
Many thanks to the SA developers! Rock on!
Joe
Re: update on floating dividing score between spam and ham messages
Posted by Joe Flowers <fl...@social.chass.ncsu.edu>.
Loren Wilton wrote:
>This is quite interesting, and seems reasonably obvious that with the right
>sort of mail (at least, maybe with any mail) this shoudl work better, since
>it self tunes to your conditions. It does of course assume a reasonable
>fp/fn rate to start, but SA is generally pretty good about that.
>
>
Yes, and that's the assumption - that SA is good at what it does and has
a "practical" desert of more than infinitesimal width somewhere/anywhere
between real spam and real ham.
>How have you implemented the floating score? A change to SA of some sort?
>
>
I have a small C program which is a very small subset of spamd. I don't
use spamc or spamd. I just make calls directly into Perl (to Spam
Assassin since SA is a Perl module) from the C program. The C program is
a multithreaded socket program that talks to SA and to our mail servers.
>Or do you just have something that monitors the current ratio and every so
>often restarts SA with a new score threshhold?
>
>
The averages are recalculated/updated after every message.
The "dividing" line floats without bound, but the "30%" is "set" as a
command line input argument to the C program and was based on trial and
error. For example, 50% let way too much spam through. Lower than 30%
*seems* like it might produce too many false positives - but has not
been tested and it seems reasonably likely that 25% could have just as
many FPs as 30%. i.e., an electron could jump across into the next
energy band without ever being in the middle no-man's land - giving rise
to a way to measure SA's imperfectness? These percentages are stabs in
the dark about what the distributions of spam and ham really look like.
What is intriguing to me is: why can't this "30%" be a function of
something else (maybe like weekend mail versus weekday mail, night
versus day, etc.) that would increase the overall anti-spam
effectiveness and not just set and left at "30%"? (For example, I
noticed that we seem to get a higher spam to ham ratio during the
weekend than during the week.)
Joe
Re: update on floating dividing score between spam and ham messages
Posted by Loren Wilton <lw...@earthlink.net>.
This is quite interesting, and seems reasonably obvious that with the right
sort of mail (at least, maybe with any mail) this shoudl work better, since
it self tunes to your conditions. It does of course assume a reasonable
fp/fn rate to start, but SA is generally pretty good about that.
How have you implemented the floating score? A change to SA of some sort?
Or do you just have something that monitors the current ratio and every so
often restarts SA with a new score threshhold?
Loren
Re: update on floating dividing score between spam and ham messages
Posted by Joe Flowers <fl...@social.chass.ncsu.edu>.
jdow wrote:
> The greater the separation <choke> the
> better the results for a decision point between them.
....
> But anything you can do that widens the
> typical score distribution between ham and spam is a good thing.
Amen!!!!
Re: update on floating dividing score between spam and ham messages
Posted by jdow <jd...@earthlink.net>.
From: "Matt Kettler" <mk...@evi-inc.com>
> Joe Flowers wrote:
> > I don't know if this will help anyone or not, but I wanted to report
> > back just in case.
> >
> > In early April, I completely unhinged the dividing line between what SA
> > score is used to mark a message as spam or ham (5.00 = default). This
> > allows the system and this dividing line to drift "freely" to anywhere
> > that SA will allow, without bound. This anti-spam setup has worked
> > consistently much much better the whole time than in any previous
> > implementation that we have done and with very little maintenance. We
> > are very happy with it and are looking forward to implementing future SA
> > versions in the same fashion.
> >
> > I'm not exactly sure the following numbers represent the whole time
> > since April, but they should be pretty close.
> >
> > We've had 360,922 spam messages and 396,983 ham messages with a
> > normalized average spam score of 6.8714134 and a normalized average ham
> > score of -2.1532284. I have the divding line "set" at 30% of the
> > distance between the average ham score and average spam score (30% above
> > the average ham score). So, the dividing line is currently floating
> > around 0.55416414.
>
>
> The only problem I see with this approach is that it treats false
positives and
> false negatives as being equally bad.
>
> In general, you're adjusting the score bias so the number of FP's and FNs
are
> approximately equal. Although STATISTICS*.txt would suggest that this
boundary
> occurs somewhere near 2.0, your own local biases could change this
considerably.
>
>
> SA's normal scoreset is evolved with the concept that it's better to have
99
> false negatives than 1 false positive. The concept here is most people use
> scripts to move their spam into a separate folder, or auto delete it. With
that
> going on, a FP is potentially lost valid email, whereas a FN is a minor
> inconvenience.
Operating experience here seems to indicate that the SA score evolution
is not optimum. What you want to do is create a <cough> brassiere curve
for the markups for ham and spam. The greater the separation <choke> the
better the results for a decision point between them. The bias to
prevent false negatives probably means you do not want the decision
point right in the center. But anything you can do that widens the
typical score distribution between ham and spam is a good thing. It makes
the decision point less sensitive to set and the overall error rates
lower. I think this is part of the reason I have so much success on a
box vastly overloaded with SARE and other rules. The good rules pile
one on the other until it's VERY clear what is ham and what is spam.
(It surely would be nice if there were some really good indications of
"not spam". However, nothing has ever appeared other than absence of
hits on spam-sign.)
{^_^}
Re: update on floating dividing score between spam and ham messages
Posted by Kai Schaetzl <ma...@conactive.com>.
Loren Wilton wrote on Mon, 11 Jul 2005 11:30:07 -0700:
> Which of course means that by picking the ratio value you can pick pretty
> much any fp/fn ratio you want.
Only if the distribution was equal.
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
IE-Center: http://ie5.de & http://msie.winware.org
Re: update on floating dividing score between spam and ham messages
Posted by Loren Wilton <lw...@earthlink.net>.
> > score of -2.1532284. I have the divding line "set" at 30% of the
> > distance between the average ham score and average spam score (30% above
> > the average ham score). So, the dividing line is currently floating
> > around 0.55416414.
>
>
> The only problem I see with this approach is that it treats false
positives and
> false negatives as being equally bad.
Matt, isn't he actually treating an FP as ~2x as bad as an FN? He has the
divider set to 30%, so is biassed in one direction or the other.
Which of course means that by picking the ratio value you can pick pretty
much any fp/fn ratio you want.
Loren
Re: update on floating dividing score between spam and ham messages
Posted by Kai Schaetzl <ma...@conactive.com>.
Joe Flowers wrote on Tue, 12 Jul 2005 11:55:36 -0400:
> >That's bad, really bad
> >detection ...
> >
> >
>
> No. It's good, really good detection.
Sorry, I don't want to be rude by repeating myself, but if your average spam
score is something like 6-something the *detection* *is* bad. Maybe not the
end result but the pure spam detection. And that's also the reason why you had
to try and find a method which lowers the threshold without giving you too
much false positives. If your spam would score high enough you simply wouldn't
need to do that. That's btw exactly what you said yourself:
> > But anything you can do that widens the
> > typical score distribution between ham and spam is a good thing.
>
> Amen!!!!
> For lack of a better term in mind, I used "normalized". If the score of
> a message is more than 30 points (or 25, I'm not going to waste time
> looking back at the code) away from the nearest average, then I set the
> score for the message back to 30 points away from the nearest average.
Ok, I see you want to avoid peaks, makes sense.
> It sounds like you have put in a lot of time to become an expert in the
> traditional wisdom of SA and to tune it accordingly.
Not more than others here. Not really too much time.
And, I assume you
> spend a lot of time keeping it tuned and dealing with SA upgrades.
Not at all. I have once carefully crafted a combination of my own rules plus
SARE rules some time ago, trained it a lot of spam and ham at first and now
let it just run, SARE updates are done automatically by rulesdujour. I haven't
put much attention to it for probably a year now. Just some upgrading to SA
3.1.* recently and maybe choosing a different SARE ruleset here and there.
I'm
> glad you have that time.... But, my situation is different and I agree
> with some of the crtitics of SA - that it requires or almost requires an
> expert to tune it properly and to keep it tuned properly
You indeed need some time to understand how it all works together, but then
you don't need to apply too much care anymore, really. Of course, you should
stay up-to-date with releases and the rulesets you use, but that's not a daily
business at all.
>
> And again, you are wrong. It is a very good setup (the proof is in the
> pudding)
As I said earlier if you look closer at the pudding I'm sure that your false
positive rate is much higher than ours. Do you have a proven figure of your FP
rate?
To make it clear: I don't want to say that you have bad results from your
setup. But I'm quite convinced that your FP rate could be much better if you
tried to widen the gap between the ham and spam by applying more rules that
are able to classify spam and maybe by finetuning a few rules scores (f.i. if
you Bayes_99 is reliable you should boost it to 3 or 4, it's overly low in the
3.0 setups). Your peaks are only 8 score points away from each other. Ours are
more than 20 points away from each other and the vale between them is really
low. Which means even if I slide the threshold for one absolute score point
from 5 to 6 or down to 4 I won't get a much different detection rate because
there's so few messages scoring in that range. I'm sure that many on this list
would have similar results if they did that. But I suppose you can't do that
because your gap is simply too small. What happens if you move your threshold
one score point down or up?
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
IE-Center: http://ie5.de & http://msie.winware.org
Re: update on floating dividing score between spam and ham messages
Posted by Joe Flowers <fl...@social.chass.ncsu.edu>.
Kai Schaetzl wrote:
>Joe Flowers wrote on Mon, 11 Jul 2005 12:09:29 -0400:
>
>
>
>That's bad, really bad
>detection ...
>
>
No. It's good, really good detection.
>You should improve that instead of trying to find a
>barrier which gives you the best FP:FN ratio.
>
I'm not trying to find the best FP:FN ratio and I never said I was.
> It may indeed give you the
>best ratio with your bad setup
>
>
No. It's a good setup and I won't tell our users that they should stop
complimenting us on it's success because the results speak for
themselves and back up those compliments. And yes, it does work very
good in the short and long run - you are purely speculating that it is not.
>BTW: what does "normalized" exactly mean in this context?
>
>
For lack of a better term in mind, I used "normalized". If the score of
a message is more than 30 points (or 25, I'm not going to waste time
looking back at the code) away from the nearest average, then I set the
score for the message back to 30 points away from the nearest average.
It's to help the difting averages from changes rapidly and abruptly. It
does not prevent them from drifting anywhere however.
It sounds like you have put in a lot of time to become an expert in the
traditional wisdom of SA and to tune it accordingly. And, I assume you
spend a lot of time keeping it tuned and dealing with SA upgrades. I'm
glad you have that time.... But, my situation is different and I agree
with some of the crtitics of SA - that it requires or almost requires an
expert to tune it properly and to keep it tuned properly - at least with
the traditional wisdom of how it "should" be set up. My goal is not to
become a SA expert, it would be nice if it happened along the way
somewhere down the road, but it is not a goal. Also, I have no plans of
hiring an SA expert unless it comes as a bonus along with the more
important things we are doing here. It will not be a deciding factor.
It's nice, but it is not primary.
And again, you are wrong. It is a very good setup (the proof is in the
pudding) and the only thing I see being changed in the future is an
upgrade to the latest version of SA.
Joe
Re: update on floating dividing score between spam and ham messages
Posted by Kai Schaetzl <ma...@conactive.com>.
Kai Schaetzl wrote on Mon, 11 Jul 2005 22:31:29 +0200:
> With the default of 5 we get almost none, not even one per day.
That was about FPs. Wrong. We don't get *any* FPs. We do not get even one
*FN* per day.
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
IE-Center: http://ie5.de & http://msie.winware.org
Re: update on floating dividing score between spam and ham messages
Posted by Kai Schaetzl <ma...@conactive.com>.
Joe Flowers wrote on Mon, 11 Jul 2005 12:09:29 -0400:
> We are very glad and happy about this concept and implementation.
Well, the big question is: How many of your spam messages score between
the default 5 and your "floating score"? If it is many there's obviously
something wrong with your setup: your spam is not scoring high enough.
Additionally, it means that your Bayes auto-learn will feed less spam to
learn than it could because your overall spam score is way too low. Our
average spam score is indeed around -2 as yours is. And it's a very high
peak, -2 mails are more than any other ham mails combined. However, our
spam score peak is *way* higher than yours is: it "flattens" over 18 and
30, so the average is somewhere around 25 or so. (I deduced that from
looking at the raw figures not by calculating a median or average.) I
consider your average spam score of 6 as *extremely* bad from a detection
standpoint.
With a score of 0.5 I would get a *considerable* amount of ham scored as
spam. With the default of 5 we get almost none, not even one per day. I
doubt that your rate of FPs is nearly non-existant with a spam threshold
of 0.5. There *must* be a considerable rate of FPs, you just don't hear
about it.
I think the general approach on this list is to make spam score as spammy
as possible. That's what we do as well. Instead of driving spam to the sky
you are trying to find some non-existing "barrier" which may indeed float
because tomorrow's messages score different than yesterday's. It does not
float at all in the long run. And it exists *only* in the long run. It may
throw off next day's detection quite heavily, since there's no guarantee
spam and ham look the same next day or even float around that point. It's
not even a statistical figure, you deliberately set it to 30%, probably
because you get too much spam if you set it higher. That's bad, really bad
detection ...
If much of your spam is lower than 5 than the spam detection rate of your
SA is quite bad. You should improve that instead of trying to find a
barrier which gives you the best FP:FN ratio. It may indeed give you the
best ratio with your bad setup but not the lowest FP rate and probably not
the best ratio compared with a setup that drives spam to the sky.
I see your approach as an interesting way of optimizing the threshold when
you don't get optimal scores. But you would be better off to optimize the
scores.
BTW: what does "normalized" exactly mean in this context?
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
IE-Center: http://ie5.de & http://msie.winware.org
Re: update on floating dividing score between spam and ham messages
Posted by Kelson <ke...@speed.net>.
Joe Flowers wrote:
> BTW, if anyone knows a command line program that can easy run thu a
> bunch of mbox files and tell how many messages are in them, I will
> report back how many ham and how many spam messages that I have fed to
> bayes. It's far from perfect, but it may offer some interesting info
> regarding the 100:1 (fn:fp) ratio.
I usually do this:
grep -c "^From " filename
It's not perfect, since it's theoretically possible for someone to have
a line in their message that starts with
From (to provide an example -- see if your mbox-generating program
escapes that line!), but it's usually enough.
--
Kelson Vibber
SpeedGate Communications <www.speed.net>
Re: update on floating dividing score between spam and ham messages
Posted by Kris Deugau <kd...@vianet.ca>.
jdow wrote:
> A few weeks ago I'd have said "Easy, Ducky!" Then I ran into DoveCot
> that uses an indexed almost "mbox" file. There is no way to do it
> other than "good guess". However, for a traditional UNIX mbox file
> you should be able to nail it perfectly simply looking for the "From"
> feature. The dirt stupid "mail" utility looks for a blank line
> followed by a line that starts with "From". All other lines that
> start with From are supposed to be escaped to ensure accurate
> detection. DoveCot skips this blank like feature sometimes. "mail"
> does not like this. I have not yet seen any indication that SA is
> upset with this, however.
Just to be pedantic, it's actually (IIRC) a double newline followed by
"From " (note the space! It's important.) Many mail-handling apps will
actually parse the From-space "header" in more detail, "just in case".
grep "^From " <mboxfile>|wc -l typically gives an accurate count;
procmail at least is bright enough to escape message body lines such
that they don't break this.
-kgd
--
Get your mouse off of there! You don't know where that email has been!
Re: update on floating dividing score between spam and ham messages
Posted by jdow <jd...@earthlink.net>.
A few weeks ago I'd have said "Easy, Ducky!" Then I ran into DoveCot
that uses an indexed almost "mbox" file. There is no way to do it other
than "good guess". However, for a traditional UNIX mbox file you should
be able to nail it perfectly simply looking for the "From" feature. The
dirt stupid "mail" utility looks for a blank line followed by a line
that starts with "From". All other lines that start with From are supposed
to be escaped to ensure accurate detection. DoveCot skips this blank like
feature sometimes. "mail" does not like this. I have not yet seen any
indication that SA is upset with this, however.
{^_^}
----- Original Message -----
From: "Joe Flowers" <fl...@social.chass.ncsu.edu>
> Matt:
>
> I know you know a lot more about this than I do, but for what it's
> worth, you're impressions/intuitions are very close to mine.
> Originally back in April, I started off using the "average of the
> means", but that let through way too much spam.
>
> So, what I have now is it set to 30% above the average spam score, which
> is 20% below the "average of the means".
> The assumption being that the optimal spot is somewhere between the two
> averages.
>
> Also, that nastly drop off that produces a lot of FPs is in my intuition
> too and as of yet, we haven't run into it.
>
> Now, if the two curves could be slid apart wider so that there is a big
> deadzone,... Although, without upgrading to a newer version of SA, I
> don't see how I can expect much better results.
>
> BTW, if anyone knows a command line program that can easy run thu a
> bunch of mbox files and tell how many messages are in them, I will
> report back how many ham and how many spam messages that I have fed to
> bayes. It's far from perfect, but it may offer some interesting info
> regarding the 100:1 (fn:fp) ratio.
>
> Joe
>
>
> Matt Kettler wrote:
>
> >Joe Flowers wrote:
> >
> >
> >>Matt Kettler wrote:
> >>
> >>
> >>
> >>>The only problem I see with this approach is that it treats false
> >>>positives and
> >>>false negatives as being equally bad.
> >>>
> >>>
> >>>
> >>>
> >>We do get many more false negatives than false positives, even though we
> >>don't get false positives very often - they are rare.
> >>We certainly don't get 1 fp for every fn.
> >>
> >>
> >>
> >>>In general, you're adjusting the score bias so the number of FP's and
> >>>FNs are
> >>>approximately equal.
> >>>
> >>>
> >>This is not what we are seeing in practice. It's not even close to
50-50.
> >>
> >>
> >>
> >
> >Based on JM's comments about the score distribution for hams being
non-linear,
> >this makes sense. If the distribution was linear for both you'd get 50/50
by
> >dividing the score between the two means.
> >
> >Since the ham is going to have a pretty sharp drop-off somewhere slightly
above
> >it's mean your split score approach won't be as bad as 1:1, but it's also
likely
> >to not be as good as 100:1 which the 5.0 threshold should get you.
> >
> >It's an interesting concept, and it would be very interesting to graph
out FP vs
> >FN rates against thresholds.
> >
> >This graph from JM's post is real data:
> >http://spamassassin.apache.org/presentations/HEANet_2002/img12.html
> >
> >But it doesn't go below 5.0. It would be interesting to see how those
curves
> >continue as you approach 0.
> >
> >This graph is a good conceptual one in the "normal" sense of numbers:
> >http://taint.org/xfer/2005/score-dist-doodle.gif
> >
> >That graph would suggest that somewhere below 5.0 there is a threshold at
which
> >the ham FP rate gets MUCH worse in a very sudden way. However, there's no
score
> >associated. I'd venture to guess that your "average of the means" is
going to
> >wind up picking something near, but just above that threshold.
> >
> >That's a bit of an intuitive guess, but also it has some roots in
reality. The
> >average score of a ham message on a curve like that is going to wind up
being
> >somewhere in the middle of that nasty drop off. By biasing just above
that you
> >should bring yourself into the second part of the curve, where decreases
in
> >score have a somewhat modest impact on FP rate.
> >
> >
> >
>
Re: Bayes Questions
Posted by "Daniel J. Cody" <dc...@uwm.edu>.
Andrew,
Andrew Ott wrote:
> Also is there any way to see the count of spam and ham messages that are in
> the bayes database, I can't seem to find any info on that. I want to make
> sure there are a lot in there before I turn the bayes rules on.
If you run spamassassin --lint -D you should see a line that says
something like:
debug: bayes corpus size: nspam = 10000, nham = 5000
nspam is the number of spam messages, nham is the number of hams it has
learned.
HTH
Dan
Re: Bayes Questions
Posted by Kai Schaetzl <ma...@conactive.com>.
Herb Martin wrote on Tue, 12 Jul 2005 12:35:55 -0500:
> I checked magic, performed learn with each of ham/spam,
> saw positive sign that 'M of N messages were learned' and
> yet on viewing magic subsequently the numbers for both
> nspam and nham had not changed.
You probably learned with a different user. SA can have site-wide or
user-specific Bayes. Depending on that you have to be careful which user
you learn to.
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
IE-Center: http://ie5.de & http://msie.winware.org
RE: Bayes Questions
Posted by Herb Martin <He...@learnquick.com>.
> From: Kai Schaetzl [mailto:maillists@conactive.com]
> > Andrew Ott wrote on Mon, 11 Jul 2005 17:37:42 -0600:
>
> > Also is there any way to see the count of spam and ham
> messages that
> > are in the bayes database, I can't seem to find any info on
> that. I
> > want to make sure there are a lot in there before I turn
> the bayes rules on.
>
> sa-learn --dump magic
I checked magic, performed learn with each of ham/spam,
saw positive sign that 'M of N messages were learned' and
yet on viewing magic subsequently the numbers for both
nspam and nham had not changed.
Is this expected behavior? What should I do different.
I did 'sync' and yet the magic values still remained unchanged.
--
Herb
Re: Bayes Questions
Posted by Kai Schaetzl <ma...@conactive.com>.
Andrew Ott wrote on Mon, 11 Jul 2005 17:37:42 -0600:
> Also is there any way to see the count of spam and ham messages that are in
> the bayes database, I can't seem to find any info on that. I want to make
> sure there are a lot in there before I turn the bayes rules on.
sa-learn --dump magic
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
IE-Center: http://ie5.de & http://msie.winware.org
Re: Bayes Questions
Posted by Matt Kettler <mk...@evi-inc.com>.
Andrew Ott wrote:
> For those of you running large sites ( we have about 12,000 users, with
> 210,000 messages a day) what do you have for a bayes_expiry_max_db_size?
Sorry, I'm not a large site, can't answer that.
> Also is there any way to see the count of spam and ham messages that are in
> the bayes database, I can't seem to find any info on that. I want to make
> sure there are a lot in there before I turn the bayes rules on.
sa-learn --dump magic
Bayes Questions
Posted by Andrew Ott <ao...@actcom.net>.
For those of you running large sites ( we have about 12,000 users, with
210,000 messages a day) what do you have for a bayes_expiry_max_db_size?
Also is there any way to see the count of spam and ham messages that are in
the bayes database, I can't seem to find any info on that. I want to make
sure there are a lot in there before I turn the bayes rules on.
Thank you.
Andrew
Re: update on floating dividing score between spam and ham messages
Posted by Joe Flowers <fl...@social.chass.ncsu.edu>.
> BTW, if anyone knows a command line program that can easy run thu a
bunch of mbox files and tell how many messages are in them, I will report
> back how many ham and how many spam messages that I have fed to bayes.
Well, I thought this might give some good stats on the FP:FN ratio, but
I forgot I manually fed Bayes at the very beginning of the SA 3.02
install to get it kick-started immediately. So, counting those messages
won't give anything accurate :( Initially, I thought I was feeding Bayes
just the FPs and FNs, but I forgot about the initial feeding.
Re: update on floating dividing score between spam and ham messages
Posted by Joe Flowers <fl...@social.chass.ncsu.edu>.
Matt:
I know you know a lot more about this than I do, but for what it's
worth, you're impressions/intuitions are very close to mine.
Originally back in April, I started off using the "average of the
means", but that let through way too much spam.
So, what I have now is it set to 30% above the average spam score, which
is 20% below the "average of the means".
The assumption being that the optimal spot is somewhere between the two
averages.
Also, that nastly drop off that produces a lot of FPs is in my intuition
too and as of yet, we haven't run into it.
Now, if the two curves could be slid apart wider so that there is a big
deadzone,... Although, without upgrading to a newer version of SA, I
don't see how I can expect much better results.
BTW, if anyone knows a command line program that can easy run thu a
bunch of mbox files and tell how many messages are in them, I will
report back how many ham and how many spam messages that I have fed to
bayes. It's far from perfect, but it may offer some interesting info
regarding the 100:1 (fn:fp) ratio.
Joe
Matt Kettler wrote:
>Joe Flowers wrote:
>
>
>>Matt Kettler wrote:
>>
>>
>>
>>>The only problem I see with this approach is that it treats false
>>>positives and
>>>false negatives as being equally bad.
>>>
>>>
>>>
>>>
>>We do get many more false negatives than false positives, even though we
>>don't get false positives very often - they are rare.
>>We certainly don't get 1 fp for every fn.
>>
>>
>>
>>>In general, you're adjusting the score bias so the number of FP's and
>>>FNs are
>>>approximately equal.
>>>
>>>
>>This is not what we are seeing in practice. It's not even close to 50-50.
>>
>>
>>
>
>Based on JM's comments about the score distribution for hams being non-linear,
>this makes sense. If the distribution was linear for both you'd get 50/50 by
>dividing the score between the two means.
>
>Since the ham is going to have a pretty sharp drop-off somewhere slightly above
>it's mean your split score approach won't be as bad as 1:1, but it's also likely
>to not be as good as 100:1 which the 5.0 threshold should get you.
>
>It's an interesting concept, and it would be very interesting to graph out FP vs
>FN rates against thresholds.
>
>This graph from JM's post is real data:
>http://spamassassin.apache.org/presentations/HEANet_2002/img12.html
>
>But it doesn't go below 5.0. It would be interesting to see how those curves
>continue as you approach 0.
>
>This graph is a good conceptual one in the "normal" sense of numbers:
>http://taint.org/xfer/2005/score-dist-doodle.gif
>
>That graph would suggest that somewhere below 5.0 there is a threshold at which
>the ham FP rate gets MUCH worse in a very sudden way. However, there's no score
>associated. I'd venture to guess that your "average of the means" is going to
>wind up picking something near, but just above that threshold.
>
>That's a bit of an intuitive guess, but also it has some roots in reality. The
>average score of a ham message on a curve like that is going to wind up being
>somewhere in the middle of that nasty drop off. By biasing just above that you
>should bring yourself into the second part of the curve, where decreases in
>score have a somewhat modest impact on FP rate.
>
>
>
Re: update on floating dividing score between spam and ham messages
Posted by Matt Kettler <mk...@evi-inc.com>.
Joe Flowers wrote:
> Matt Kettler wrote:
>
>> The only problem I see with this approach is that it treats false
>> positives and
>> false negatives as being equally bad.
>>
>>
>
> We do get many more false negatives than false positives, even though we
> don't get false positives very often - they are rare.
> We certainly don't get 1 fp for every fn.
>
>> In general, you're adjusting the score bias so the number of FP's and
>> FNs are
>> approximately equal.
>
>
> This is not what we are seeing in practice. It's not even close to 50-50.
>
Based on JM's comments about the score distribution for hams being non-linear,
this makes sense. If the distribution was linear for both you'd get 50/50 by
dividing the score between the two means.
Since the ham is going to have a pretty sharp drop-off somewhere slightly above
it's mean your split score approach won't be as bad as 1:1, but it's also likely
to not be as good as 100:1 which the 5.0 threshold should get you.
It's an interesting concept, and it would be very interesting to graph out FP vs
FN rates against thresholds.
This graph from JM's post is real data:
http://spamassassin.apache.org/presentations/HEANet_2002/img12.html
But it doesn't go below 5.0. It would be interesting to see how those curves
continue as you approach 0.
This graph is a good conceptual one in the "normal" sense of numbers:
http://taint.org/xfer/2005/score-dist-doodle.gif
That graph would suggest that somewhere below 5.0 there is a threshold at which
the ham FP rate gets MUCH worse in a very sudden way. However, there's no score
associated. I'd venture to guess that your "average of the means" is going to
wind up picking something near, but just above that threshold.
That's a bit of an intuitive guess, but also it has some roots in reality. The
average score of a ham message on a curve like that is going to wind up being
somewhere in the middle of that nasty drop off. By biasing just above that you
should bring yourself into the second part of the curve, where decreases in
score have a somewhat modest impact on FP rate.
Re: update on floating dividing score between spam and ham messages
Posted by Joe Flowers <fl...@social.chass.ncsu.edu>.
Matt Kettler wrote:
>The only problem I see with this approach is that it treats false positives and
>false negatives as being equally bad.
>
>
We do get many more false negatives than false positives, even though we
don't get false positives very often - they are rare.
We certainly don't get 1 fp for every fn.
>In general, you're adjusting the score bias so the number of FP's and FNs are
>approximately equal.
>
This is not what we are seeing in practice. It's not even close to 50-50.
>Although STATISTICS*.txt would suggest that this boundary
>occurs somewhere near 2.0, your own local biases could change this considerably.
>
>
>SA's normal scoreset is evolved with the concept that it's better to have 99
>false negatives than 1 false positive.
>
We are very glad and happy about this concept and implementation.
>The concept here is most people use
>scripts to move their spam into a separate folder, or auto delete it. With that
>going on, a FP is potentially lost valid email, whereas a FN is a minor
>inconvenience.
>
>
Yes.... We work hard to inform our users and to actively solicit their
feedback on how the system is working and to lookout for the system
misplacing emails, especially valid ones. I know it's still not perfect....
>For any site that considers FPs to be "not too bad" because all mail is manually
>examined anyway, lowering the score threshold may be a workable thing.
>
>However, other sites that auto-delete such messages may have considerable
>problems if they lower the threshold
>
>
YES!
Re: update on floating dividing score between spam and ham messages
Posted by Matt Kettler <mk...@evi-inc.com>.
Joe Flowers wrote:
> I don't know if this will help anyone or not, but I wanted to report
> back just in case.
>
> In early April, I completely unhinged the dividing line between what SA
> score is used to mark a message as spam or ham (5.00 = default). This
> allows the system and this dividing line to drift "freely" to anywhere
> that SA will allow, without bound. This anti-spam setup has worked
> consistently much much better the whole time than in any previous
> implementation that we have done and with very little maintenance. We
> are very happy with it and are looking forward to implementing future SA
> versions in the same fashion.
>
> I'm not exactly sure the following numbers represent the whole time
> since April, but they should be pretty close.
>
> We've had 360,922 spam messages and 396,983 ham messages with a
> normalized average spam score of 6.8714134 and a normalized average ham
> score of -2.1532284. I have the divding line "set" at 30% of the
> distance between the average ham score and average spam score (30% above
> the average ham score). So, the dividing line is currently floating
> around 0.55416414.
The only problem I see with this approach is that it treats false positives and
false negatives as being equally bad.
In general, you're adjusting the score bias so the number of FP's and FNs are
approximately equal. Although STATISTICS*.txt would suggest that this boundary
occurs somewhere near 2.0, your own local biases could change this considerably.
SA's normal scoreset is evolved with the concept that it's better to have 99
false negatives than 1 false positive. The concept here is most people use
scripts to move their spam into a separate folder, or auto delete it. With that
going on, a FP is potentially lost valid email, whereas a FN is a minor
inconvenience.
For any site that considers FPs to be "not too bad" because all mail is manually
examined anyway, lowering the score threshold may be a workable thing.
However, other sites that auto-delete such messages may have considerable
problems if they lower the threshold.