You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by RW <rw...@googlemail.com> on 2009/01/28 21:03:48 UTC
Autolearning from rules rather than score
I was wondering if it's possible to control autolearning based on rules.
I'm scoring DSPAM into Spamassassin, and since DSPAM autolearns
everything, it would be sensible to synchronise the learning, so when I
correct dspam the same set of mails need to be corrected in
Spamassassin.
Re: Autolearning from rules rather than score
Posted by Kai Schaetzl <ma...@conactive.com>.
Rw wrote on Thu, 29 Jan 2009 03:23:47 +0000:
> > > However, thinking about it a bit more, I think that the only real
> > > problem is that ham that scores between 0.1 and 5.0
> > > wont be learned as ham, and I can fix that by moving the autolearn
> > > threshold to up to 4.9.
> >
> > Eek! No, this is wrong and gives me the creeps.
>
> That's probably because you're not seeing the big picture yet.
There is no big picture. Learning everything below the spam threshold as
ham means learning the occasional FP as well. If you want to tell us you
don't have any FPs ... Well, why then do you need to train that heavy?
Kai
--
Kai Schätzl, Berlin, Germany
Get your web at Conactive Internet Services: http://www.conactive.com
Re: Autolearning from rules rather than score
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
On 29.01.09 03:23, RW wrote:
> I meant have Bayes learn from the DSPAM header rules that I quoted.
> What does the plugin actually do that simply piping mail though DSPAM
> before SA doesn't?
you want SA BAYES filter to learn from what will DSPAM filter tell it?
You can do that with procmail/maildrop/whatever filters, but I object
against it - any false positives or negatives will have bad effect on your
spam filters' effectiveness
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
My mind is like a steel trap - rusty and illegal in 37 states.
Re: Autolearning from rules rather than score
Posted by RW <rw...@googlemail.com>.
On Thu, 29 Jan 2009 00:08:57 +0100
Karsten Bräckelmann <gu...@rudersport.de> wrote:
> On Wed, 2009-01-28 at 22:36 +0000, RW wrote:
> > On Wed, 28 Jan 2009 22:02:59 +0100
> > Karsten Bräckelmann <gu...@rudersport.de> wrote:
> > > By that you mean... Using the DSPAM plugin for SA? And the rule
> > > you want to base auto-learning upon is the DSPAM plugin one?
> >
> > No, is there any point?
>
> Err, then I don't understand the "auto-learning from rules" in your
> Subject. What do you mean by that?
I meant have Bayes learn from the DSPAM header rules that I quoted.
What does the plugin actually do that simply piping mail though DSPAM
before SA doesn't?
> > However, thinking about it a bit more, I think that the only real
> > problem is that ham that scores between 0.1 and 5.0
> > wont be learned as ham, and I can fix that by moving the autolearn
> > threshold to up to 4.9.
>
> Eek! No, this is wrong and gives me the creeps.
That's probably because you're not seeing the big picture yet.
> As I've mentioned before (hey, see your quote :), certain rules like
> Bayes will NOT be taken into account for the threshold. Also, scores
> used for auto-learning evaluation are using a non-Bayes score set. See
> the docs.
> http://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Plugin_AutoLearnThreshold.html
>
> That means that a mail scoring your threshold PLUS the BAYES_00 score
> can be learned as ham. Possibly even much higher, since auto-learn
> uses score set 0 or 1... Think about this for a moment.
That perfectly OK, because it will neccessarily get caught as spam and
I'll reclassify it manually as a matter of course.
> AFAIK, there is no clean way of tricking SA into learning *everything*
> above and below a given threshold.
>
> Also, a certain gray area is better not learned automatically.
> Seriously. False learning *immediately* will have an impact on further
> results. Whereas learning after a manual re-view is slower, but not
> affected by bootstrapping even more FNs and FPs out of its own ass.
Don't forget that the -2.5 score of DSPAM does contribute to
autolearning but BAYES_* doesn't, so it's not going to runaway in the
FP direction. And anyway very few emails score in the region -15 to
+20, with the rules I quoted. It's just that I think they may be
particularly important to learn.
For the most part I keep the Bayes rules just for the meta rule:
meta DS_HAM_FULL DS_HAM && (BAYES_00 || BAYES_05)
score DS_HAM_FULL -15.0
Which gives almost all ham a score of around -20, but doesn't prevent
SA from catching spams that DSPAM misses.
DSPAM produces yes/no results, and works on an "innocent until proven
guilty" basis, so it occasionally classifies as ham when it's really
just uncertain. If I simply scored DS_HAM at -20, I'd effectively
prevent SA for deciding anything other than which Junk folder the spam
go into.
DSPAM is a *much* better statistical filter overall, but I've noticed
that however good spammers get at obfuscating spam, simple Bayesian
filters remains highly effective at identifying almost all of my ham
beyond reasonable doubt - unidentified email clusters around 0.5 not
0.0 as it does with DSPAM. This makes DSPAM and SA's Bayesian filter
complementary. If the latter actually identifies spam it's no more than
a bonus.
Re: Autolearning from rules rather than score
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Wed, 2009-01-28 at 22:36 +0000, RW wrote:
> On Wed, 28 Jan 2009 22:02:59 +0100
> Karsten Bräckelmann <gu...@rudersport.de> wrote:
> > > On Wed, 28 Jan 2009, RW wrote:
> > >
> > > > I was wondering if it's possible to control autolearning based
> > > > on rules.
> >
> > No. And even tweaking the various thresholds will not help, since
> > auto-learning is based on the score *without* Bayes, etc.
> >
> > > > I'm scoring DSPAM into Spamassassin, and since DSPAM autolearns
> >
> > By that you mean... Using the DSPAM plugin for SA? And the rule you
> > want to base auto-learning upon is the DSPAM plugin one?
>
> No, is there any point?
Err, then I don't understand the "auto-learning from rules" in your
Subject. What do you mean by that?
> I just pass it though dspam and then score like this:
[...]
> I combine this with some sieve rules that file into Junk and Junk.high
> folders at the scores 5 and 30. Junk.high is effectively discarded. I
> check the Junk folder and move everything to the training folders,
> along with any spam that gets through. Additionally a sieve rule
> autofiles anything over 30 that dspam didn't get into the learn-spam
> folder.
>
> That means that every single mail misclassified by dspam's
> autolearning will get reclassified, but it doesn't imply the same for
> Bayes unless Bayes autolearns in line with dspam.
You got a special handling of mails dspam missed. So you are right, it
won't do the same for SA Bayes, unless you get in some equivalent
special handling for SA...
> However, thinking about it a bit more, I think that the only real
> problem is that ham that scores between 0.1 and 5.0
> wont be learned as ham, and I can fix that by moving the autolearn
> threshold to up to 4.9.
Eek! No, this is wrong and gives me the creeps.
As I've mentioned before (hey, see your quote :), certain rules like
Bayes will NOT be taken into account for the threshold. Also, scores
used for auto-learning evaluation are using a non-Bayes score set. See
the docs.
http://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Plugin_AutoLearnThreshold.html
That means that a mail scoring your threshold PLUS the BAYES_00 score
can be learned as ham. Possibly even much higher, since auto-learn uses
score set 0 or 1... Think about this for a moment.
AFAIK, there is no clean way of tricking SA into learning *everything*
above and below a given threshold.
Also, a certain gray area is better not learned automatically.
Seriously. False learning *immediately* will have an impact on further
results. Whereas learning after a manual re-view is slower, but not
affected by bootstrapping even more FNs and FPs out of its own ass.
> BTW am I correct in assumimg that my dspam header rules
> in /usr/local/etc/mail/spamassassin/local.cf will contribute to
> autolearning.
Yes. Unless you set tflags noautolearn for your rules. See the above
doc, and section Rule Definitions here:
http://spamassassin.apache.org/full/3.2.x/doc/Mail_SpamAssassin_Conf.html
guenther
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Autolearning from rules rather than score
Posted by RW <rw...@googlemail.com>.
On Thu, 29 Jan 2009 10:32:05 +0100
Matus UHLAR - fantomas <uh...@fantomas.sk> wrote:
> On 28.01.09 22:36, RW wrote:
> > I just pass it though dspam and then score like this:
> >
> > header DS_HAM X-DSPAM-Result =~ /^(Innocent|Whitelisted)/
> > header DS_SPAM X-DSPAM-Result =~ /^Spam/
> > meta DS_HAM_FULL DS_HAM && (BAYES_00 || BAYES_05)
> >
> > score DS_HAM -2.5
> > score DS_SPAM 21.0
> > score DS_HAM_FULL -15.0
>
> don't you trust dspam too much?
No, it's 9 points below the 30 point threshold, and DSPAM FP's once in
a blue moon.
>
> > score BAYES_00 -2.5
> > score BAYES_05 -1.5
>
> Why do you do this?
> 1. you are assigning scores for BAYES even if BAYES is turned off
Are any of the BAYES_* rules hit if BAYES is turned-off?
> 2. BAYES_00 has score -2.599 when network rules are on, you are
> lowering effectiveness -
Not really, BAYES_00 is left independently scored just to allow a little
extra safety-margin for the unlikely combination DS_SPAM + BAYES_00,
aside from this combination BAYES_00 will always trip the DS_HAM_FULL
meta rule. It's set so that BAYES_00 + DS_HAM + DS_HAM_FULL = -20 which
eliminates the most serious problem in SA whereby a large legitimate
mail sometimes accumulates a lot of textual hits, and runs up a huge
score. The precise value doesn't matter all that much, but -20 allows
me to see at a glance how many other points I'm getting.
I don't regard the default BAYES scores to be all that sensible
because the scoring algorithm tolerates a very high level of FP's. I
think most people that use sa-learn properly would see better results
if they dropped BAYES_00 to -10 or lower. Personally I've never seen a
single spam hit BAYES_00.
> > However, thinking about it a bit more, I think that the only real
> > problem is that ham that scores between 0.1 and 5.0
> > wont be learned as ham, and I can fix that by moving the autolearn
> > threshold to up to 4.9.
>
> So you are willing to feed _anything_ to BAYES filters, which means
> that without manual intervention, FP and FN rate will both quickly
> increase
That's a bit alarmist, worst case is that the 4.9 threshold causes a
small number of spams that are aren't recognised by dspam to be
*temporarily* learned as ham in SA until I correct it. That sounds
pretty good to me, if a new type of mail starts to arrive, that's not
recognised by DSPAM, I'd like SA to behave like DSPAM does and
give it the benefit of the doubt.
The normal way to train DSPAM is to let it autolearn everything, and
correct it when it's wrong. Once DSPAM has learned from reasonable
sized corpora it can stand a few percent miss-classifications in new
mail because they're washed-out by correct classifications. It's only
if you stop correcting that you have a real problem, and even then it
will simply catch less spam, due to the strong bias against FP's.
Re: Autolearning from rules rather than score
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
On 28.01.09 22:36, RW wrote:
> I just pass it though dspam and then score like this:
>
> header DS_HAM X-DSPAM-Result =~ /^(Innocent|Whitelisted)/
> header DS_SPAM X-DSPAM-Result =~ /^Spam/
> meta DS_HAM_FULL DS_HAM && (BAYES_00 || BAYES_05)
>
> score DS_HAM -2.5
> score DS_SPAM 21.0
> score DS_HAM_FULL -15.0
don't you trust dspam too much?
> score BAYES_00 -2.5
> score BAYES_05 -1.5
Why do you do this?
1. you are assigning scores for BAYES even if BAYES is turned off
2. BAYES_00 has score -2.599 when network rules are on, you are lowering
effectiveness - if you want this, you shouldn't assign -1.5 for BAYES_05
> However, thinking about it a bit more, I think that the only real
> problem is that ham that scores between 0.1 and 5.0
> wont be learned as ham, and I can fix that by moving the autolearn
> threshold to up to 4.9.
So you are willing to feed _anything_ to BAYES filters, which means that
without manual intervention, FP and FN rate will both quickly increase
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
If Barbie is so popular, why do you have to buy her friends?
Re: Autolearning from rules rather than score
Posted by RW <rw...@googlemail.com>.
On Wed, 28 Jan 2009 22:02:59 +0100
Karsten Bräckelmann <gu...@rudersport.de> wrote:
> On Wed, 2009-01-28 at 12:16 -0800, John Hardin wrote:
> > On Wed, 28 Jan 2009, RW wrote:
> >
> > > I was wondering if it's possible to control autolearning based
> > > on rules.
>
> No. And even tweaking the various thresholds will not help, since
> auto-learning is based on the score *without* Bayes, etc.
>
> > > I'm scoring DSPAM into Spamassassin, and since DSPAM autolearns
>
> By that you mean... Using the DSPAM plugin for SA? And the rule you
> want to base auto-learning upon is the DSPAM plugin one?
No, is there any point?
I just pass it though dspam and then score like this:
header DS_HAM X-DSPAM-Result =~ /^(Innocent|Whitelisted)/
header DS_SPAM X-DSPAM-Result =~ /^Spam/
meta DS_HAM_FULL DS_HAM && (BAYES_00 || BAYES_05)
score DS_HAM -2.5
score DS_SPAM 21.0
score DS_HAM_FULL -15.0
score BAYES_00 -2.5
score BAYES_05 -1.5
I combine this with some sieve rules that file into Junk and Junk.high
folders at the scores 5 and 30. Junk.high is effectively discarded. I
check the Junk folder and move everything to the training folders,
along with any spam that gets through. Additionally a sieve rule
autofiles anything over 30 that dspam didn't get into the learn-spam
folder.
That means that every single mail misclassified by dspam's
autolearning will get reclassified, but it doesn't imply the same for
Bayes unless Bayes autolearns in line with dspam.
However, thinking about it a bit more, I think that the only real
problem is that ham that scores between 0.1 and 5.0
wont be learned as ham, and I can fix that by moving the autolearn
threshold to up to 4.9.
BTW am I correct in assumimg that my dspam header rules
in /usr/local/etc/mail/spamassassin/local.cf will contribute to
autolearning.
Re: Autolearning from rules rather than score
Posted by Karsten Bräckelmann <gu...@rudersport.de>.
On Wed, 2009-01-28 at 12:16 -0800, John Hardin wrote:
> On Wed, 28 Jan 2009, RW wrote:
>
> > I was wondering if it's possible to control autolearning based on rules.
No. And even tweaking the various thresholds will not help, since
auto-learning is based on the score *without* Bayes, etc.
> > I'm scoring DSPAM into Spamassassin, and since DSPAM autolearns
By that you mean... Using the DSPAM plugin for SA? And the rule you want
to base auto-learning upon is the DSPAM plugin one?
> > everything, it would be sensible to synchronise the learning, so when I
> > correct dspam the same set of mails need to be corrected in
> > Spamassassin.
>
> "correcting" implies a manual process, does it not? If you have a corpus
> of mislearned messages you're feeding to DSPAM to correct it, can you not
> do the same thing via sa-learn to retrain SA?
SA will not learn any message twice, unless the classification changed.
In that case it isn't necessary to forget the previous wrong learned
tokens either. So, yeah, I guess 'RW' could...
--
char *t="\10pse\0r\0dtu\0.@ghno\x4e\xc8\x79\xf4\xab\x51\x8a\x10\xf4\xf4\xc4";
main(){ char h,m=h=*t++,*x=t+2*h,c,i,l=*x,s=0; for (i=0;i<l;i++){ i%8? c<<=1:
(c=*++x); c&128 && (s+=h); if (!(h>>=1)||!t[s+h]){ putchar(t[s]);h=m;s=0; }}}
Re: Autolearning from rules rather than score
Posted by John Hardin <jh...@impsec.org>.
On Wed, 28 Jan 2009, RW wrote:
> I was wondering if it's possible to control autolearning based on rules.
>
> I'm scoring DSPAM into Spamassassin, and since DSPAM autolearns
> everything, it would be sensible to synchronise the learning, so when I
> correct dspam the same set of mails need to be corrected in
> Spamassassin.
"correcting" implies a manual process, does it not? If you have a corpus
of mislearned messages you're feeding to DSPAM to correct it, can you not
do the same thing via sa-learn to retrain SA?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
It's easy to be noble with other people's money.
-- John McKay, _The Welfare State:
No Mercy for the Middle Class_
-----------------------------------------------------------------------
Today: the 23rd anniversary of the loss of STS-51L Challenger