You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Matias Lopez Bergero <ml...@udesa.edu.ar> on 2005/02/10 23:06:04 UTC
bayesian filter training
Hi
Just a question,
It is worth to train the bayes filter with messages already detected and
flagged as spam by spamassassin? That would do any good?
BR,
Matías.
Re: bayesian filter training
Posted by Thomas Arend <ml...@arend-whv.info>.
Am Freitag, 11. Februar 2005 17:58 schrieb Matt Kettler:
> At 11:15 AM 2/11/2005, Matías López Bergero wrote:
> >The sa-learn man page says that for a good training of the Bayesian
> >filter, you need to train it with equal amounts of spam and ham, or more
> >ham if is possible. So if I sa-learn the spam folder, the spam tokens
> >are going to grow a lot compared to ham tokens.
> >Here are my training efforts:
> >
[..]
> >This possible increase in the spam data would have adverse effects on
> >the bayes filter classifying the spam or ham messages??
>
> The manpage is suggesting an ideal situation.. Really, you can be pretty
> wildly off and bayes will work reasonably well.
>
> Training a lot more spam than ham makes bayes more likely to misclassify a
> nonspam email as spam, but really, I'm VERY off balance and I've not had
> any problems with this at all. The difference between spam and nonspam here
> is just too great. Even a massive imbalance isn't causing FPs.
>
> Look at my stats:
>
> 0.000 0 2 0 non-token data: bayes db version
> 0.000 0 565896 0 non-token data: nspam
> 0.000 0 24693 0 non-token data: nham
> 0.000 0 180900 0 non-token data: ntokens
>
> My spam training outnumbers my ham training by 22:1. That's pretty far off
> from the ideal 1:1 or 1:1+. I've got more FN problems than FP problems with
> my bayes DB, but I rarely have problems with either.
>
> Also, I trust Dan and the SADevs to know bayes better than I do, and they
> have tested and have found 1:1 to work best.. However, from a mathematics
> perspective I'm still not sure why that works best. My original impressions
> were the ratio should match your real-world ratio... Sometime when I have
> some spare time I intend to test this myself so I can better understand it.
Hot agree, mathematical your database should represent the real ratio. Best
way is to train SA with every message. If you train less spam then you will
get a to low probability for spam. You will get less False Positive in this
case. My spam / ham ratio is 1 : 40 and it works fine.
Thomas
--
icq:133073900
http://www.t-arend.de
Re: bayesian filter training
Posted by Matías López Bergero <ml...@udesa.edu.ar>.
Matt Kettler wrote:
> At 11:15 AM 2/11/2005, Matías López Bergero wrote:
>
>> The sa-learn man page says that for a good training of the Bayesian
>> filter, you need to train it with equal amounts of spam and ham, or more
>> ham if is possible. So if I sa-learn the spam folder, the spam tokens
>> are going to grow a lot compared to ham tokens.
>>
>> This possible increase in the spam data would have adverse effects on
>> the bayes filter classifying the spam or ham messages??
>
> The manpage is suggesting an ideal situation.. Really, you can be pretty
> wildly off and bayes will work reasonably well.
>
> Training a lot more spam than ham makes bayes more likely to misclassify
> a nonspam email as spam, but really, I'm VERY off balance and I've not
> had any problems with this at all. The difference between spam and
> nonspam here is just too great. Even a massive imbalance isn't causing FPs.
>
> Look at my stats:
>
> 0.000 0 2 0 non-token data: bayes db version
> 0.000 0 565896 0 non-token data: nspam
> 0.000 0 24693 0 non-token data: nham
> 0.000 0 180900 0 non-token data: ntokens
>
> My spam training outnumbers my ham training by 22:1. That's pretty far
> off from the ideal 1:1 or 1:1+. I've got more FN problems than FP
> problems with my bayes DB, but I rarely have problems with either.
Wow!
Thank you very much for answer Matt! I'm going to sa-learn those spam
messages right now.
I was to worried about the spam/ham data balance, now it's going to be
much easier to train the Bayesian filter :)
Also I'm storing the non detected spam messages and the ham data, and
from time to time I re-run a sa-learn on those files. The sa-learn does
not report any learning but, I have read some where, probably the
sa-learn man page, that this is a good thing to do, because it will help
to reinforcing the bayes info and Bayesian filter decisions. Did I get
this right?
BR,
Matías.
Re: bayesian filter training
Posted by Matt Kettler <mk...@evi-inc.com>.
At 11:15 AM 2/11/2005, Matías López Bergero wrote:
>The sa-learn man page says that for a good training of the Bayesian
>filter, you need to train it with equal amounts of spam and ham, or more
>ham if is possible. So if I sa-learn the spam folder, the spam tokens
>are going to grow a lot compared to ham tokens.
>Here are my training efforts:
>
>[root@anubis root]# sa-learn --dump magic | head -4
>0.000 0 3 0 non-token data: bayes db version
>0.000 0 1932 0 non-token data: nspam
>0.000 0 1973 0 non-token data: nham
>0.000 0 170590 0 non-token data: ntokens
>[root@anubis root]#
>
>This possible increase in the spam data would have adverse effects on
>the bayes filter classifying the spam or ham messages??
The manpage is suggesting an ideal situation.. Really, you can be pretty
wildly off and bayes will work reasonably well.
Training a lot more spam than ham makes bayes more likely to misclassify a
nonspam email as spam, but really, I'm VERY off balance and I've not had
any problems with this at all. The difference between spam and nonspam here
is just too great. Even a massive imbalance isn't causing FPs.
Look at my stats:
0.000 0 2 0 non-token data: bayes db version
0.000 0 565896 0 non-token data: nspam
0.000 0 24693 0 non-token data: nham
0.000 0 180900 0 non-token data: ntokens
My spam training outnumbers my ham training by 22:1. That's pretty far off
from the ideal 1:1 or 1:1+. I've got more FN problems than FP problems with
my bayes DB, but I rarely have problems with either.
Also, I trust Dan and the SADevs to know bayes better than I do, and they
have tested and have found 1:1 to work best.. However, from a mathematics
perspective I'm still not sure why that works best. My original impressions
were the ratio should match your real-world ratio... Sometime when I have
some spare time I intend to test this myself so I can better understand it.
Re: bayesian filter training
Posted by Matías López Bergero <ml...@udesa.edu.ar>.
Matt Kettler wrote:
> At 05:06 PM 2/10/2005, Matias Lopez Bergero wrote:
>
>> It is worth to train the bayes filter with messages already detected
>> and flagged as spam by spamassassin? That would do any good?
>
> Yes. And even if they are already flagged as BAYES_99 it is still
> worthwhile.
Many thanks for the explanation Matt :)
I think that this are good news.
A couple of weeks ago I started storing the spam flagged messages by SA.
Currently, I have like 20400 messages stored, I'm planing to sa-learn
them, but now I got another question ;)
The sa-learn man page says that for a good training of the Bayesian
filter, you need to train it with equal amounts of spam and ham, or more
ham if is possible. So if I sa-learn the spam folder, the spam tokens
are going to grow a lot compared to ham tokens.
Here are my training efforts:
[root@anubis root]# sa-learn --dump magic | head -4
0.000 0 3 0 non-token data: bayes db version
0.000 0 1932 0 non-token data: nspam
0.000 0 1973 0 non-token data: nham
0.000 0 170590 0 non-token data: ntokens
[root@anubis root]#
This possible increase in the spam data would have adverse effects on
the bayes filter classifying the spam or ham messages??
BR,
Matías.
Re: bayesian filter training
Posted by Matias Lopez Bergero <ml...@udesa.edu.ar>.
Robert Menschel wrote:
> Hello Matias,
>
> Friday, February 11, 2005, 5:32:10 AM, you wrote:
>
> MLB> The sa-learn man page says that for a good training of the
> MLB> Bayesian filter, you need to train it with equal amounts of spam
> MLB> and ham, or more ham if is possible. So if I sa-learn the spam
> MLB> folder, the spam tokens are going to grow a lot compared to ham
> MLB> tokens.
>
> IMO, if you manually train ONLY spam into the system, then yes, you
> may end up with Bayes problems. Emphasis: may. It might work just
> fine.
>
> You don't need to worry about training Bayes with equal amounts of
> spam and ham -- my ratio has varied from 10:1 to 15:1 spam:ham, with
> no problem.
>
> But it's important to feed ham into the system as well. I would
> hesitate exceeding a 100:1 ratio, unless your actual spam load exceeds
> 100:1.
I'm running a bayes site-wide db now, and I'm seeing a lot of ham
appended to the db by the auto learn.
I think that this is a very good thing, and it's helping me to keep a
good ham:spam radio :)
Since: Feb 13 04:03:57
learned ham: 1671
Learned spam: 1560
:-D
BR,
Matías.
Re: bayesian filter training
Posted by Matias Lopez Bergero <ml...@udesa.edu.ar>.
Robert Menschel wrote:
> Hello Matias,
>
> Friday, February 11, 2005, 5:32:10 AM, you wrote:
>
> MLB> A couple of weeks ago I started storing the spam flagged messages
> MLB> by SA. Currently, I have like 20400 messages stored, I'm planing
> MLB> to sa-learn them, but now I got another question ;)
>
> MLB> The sa-learn man page says that for a good training of the
> MLB> Bayesian filter, you need to train it with equal amounts of spam
> MLB> and ham, or more ham if is possible. So if I sa-learn the spam
> MLB> folder, the spam tokens are going to grow a lot compared to ham
> MLB> tokens.
>
> IMO, if you manually train ONLY spam into the system, then yes, you
> may end up with Bayes problems. Emphasis: may. It might work just
> fine.
>
> You don't need to worry about training Bayes with equal amounts of
> spam and ham -- my ratio has varied from 10:1 to 15:1 spam:ham, with
> no problem.
>
> But it's important to feed ham into the system as well. I would
> hesitate exceeding a 100:1 ratio, unless your actual spam load exceeds
> 100:1.
>
> How much non-spam are you able to capture, verify, and sa-learn?
Last week I enable a flag in milter-spamc to redirect the non-spam mail
to some email address.
I'm going to use a few hundred of messages to maintain the radio that
you suggest. I only need to verify the messages one by one in order to
not train the Bayesian filter in the wrong way :-P
Thanks Bob :)
BR,
Matías.
Re[2]: bayesian filter training
Posted by Robert Menschel <Ro...@Menschel.net>.
Hello Matias,
Friday, February 11, 2005, 5:32:10 AM, you wrote:
MLB> A couple of weeks ago I started storing the spam flagged messages
MLB> by SA. Currently, I have like 20400 messages stored, I'm planing
MLB> to sa-learn them, but now I got another question ;)
MLB> The sa-learn man page says that for a good training of the
MLB> Bayesian filter, you need to train it with equal amounts of spam
MLB> and ham, or more ham if is possible. So if I sa-learn the spam
MLB> folder, the spam tokens are going to grow a lot compared to ham
MLB> tokens.
IMO, if you manually train ONLY spam into the system, then yes, you
may end up with Bayes problems. Emphasis: may. It might work just
fine.
You don't need to worry about training Bayes with equal amounts of
spam and ham -- my ratio has varied from 10:1 to 15:1 spam:ham, with
no problem.
But it's important to feed ham into the system as well. I would
hesitate exceeding a 100:1 ratio, unless your actual spam load exceeds
100:1.
How much non-spam are you able to capture, verify, and sa-learn?
Bob Menschel
Re: bayesian filter training
Posted by Matias Lopez Bergero <ml...@udesa.edu.ar>.
Matt Kettler wrote:
> At 05:06 PM 2/10/2005, Matias Lopez Bergero wrote:
>
>> It is worth to train the bayes filter with messages already detected
>> and flagged as spam by spamassassin? That would do any good?
>
> Yes. And even if they are already flagged as BAYES_99 it is still
> worthwhile.
Many thanks for the explanation Matt :)
I think that this are good news.
A couple of weeks ago I started storing the spam flagged messages by SA.
Currently, I have like 20400 messages stored, I'm planing to sa-learn
them, but now I got another question ;)
The sa-learn man page says that for a good training of the Bayesian
filter, you need to train it with equal amounts of spam and ham, or more
ham if is possible. So if I sa-learn the spam folder, the spam tokens
are going to grow a lot compared to ham tokens.
Here are my training efforts:
[root@anubis root]# sa-learn --dump magic | head -4
0.000 0 3 0 non-token data: bayes db version
0.000 0 1932 0 non-token data: nspam
0.000 0 1973 0 non-token data: nham
0.000 0 170590 0 non-token data: ntokens
[root@anubis root]#
This possible increase in the spam data would have adverse effects on
the bayes filter classifying the spam or ham messages??
BR,
Matías.
Re: bayesian filter training
Posted by Peter Marshall <pe...@caris.com>.
Jason, I accidently deleted your reply before I could read it ... would
you mind re-sending it ... my apologies ...
Peter
Matt Kettler wrote:
> At 05:06 PM 2/10/2005, Matias Lopez Bergero wrote:
>
>> Just a question,
>> It is worth to train the bayes filter with messages already detected
>> and flagged as spam by spamassassin? That would do any good?
>
>
> Yes. And even if they are already flagged as BAYES_99 it is still
> worthwhile.
>
>
> The reason why is that bayes does not learn that a message is spam or
> not. Bayes learns that a given set of words and tokens were seen in
> spam. A given spam message might be scored as spam and might already
> score high on the bayes scale, but it can still contain valuable new
> words to learn from. In particular the constant mutations of ways of
> spelling drug names provides a constant stream of fresh new spam
> indicators to for bayes learn about. Learning about these helps it
> identify future spam messages that might not otherwise look very
> spam-like, and offers you some protection from false negatives caused by
> spam mutations.
>
>
> The only time it's not worthwhile is if the message was already learned
> as spam (ie: by the autolearner).. but in that case SA will just ignore
> you. You're wasting some cpu time, but you won't damage or corrupt
> anything.
>
Re[2]: bayesian filter training
Posted by Robert Menschel <Ro...@Menschel.net>.
Hello Peter,
Friday, February 11, 2005, 4:17:33 AM, you wrote:
PM> but would that not mean that the bayes filter will learn the headers
PM> that spam assassin adds as spam .. and then after a while only start
PM> classing mail that already has the spam headers as bayes_99 ?
No, since the sa-learn process knows to ignore the SpamAssassin
headers.
I feed EVERYTHING into Bayes, once it's been manually
verified/classified.
Bob Menschel
PM> I really do not know, I am just asking.
PM> Peter
PM> Matt Kettler wrote:
>> At 05:06 PM 2/10/2005, Matias Lopez Bergero wrote:
>>
>>> Just a question,
>>> It is worth to train the bayes filter with messages already detected
>>> and flagged as spam by spamassassin? That would do any good?
>>
>>
>> Yes. And even if they are already flagged as BAYES_99 it is still
>> worthwhile.
>>
>>
>> The reason why is that bayes does not learn that a message is spam or
>> not. Bayes learns that a given set of words and tokens were seen in
>> spam. A given spam message might be scored as spam and might already
>> score high on the bayes scale, but it can still contain valuable new
>> words to learn from. In particular the constant mutations of ways of
>> spelling drug names provides a constant stream of fresh new spam
>> indicators to for bayes learn about. Learning about these helps it
>> identify future spam messages that might not otherwise look very
>> spam-like, and offers you some protection from false negatives caused by
>> spam mutations.
>>
>>
>> The only time it's not worthwhile is if the message was already learned
>> as spam (ie: by the autolearner).. but in that case SA will just ignore
>> you. You're wasting some cpu time, but you won't damage or corrupt
>> anything.
Re: bayesian filter training
Posted by Matt Kettler <mk...@evi-inc.com>.
At 07:17 AM 2/11/2005, Peter Marshall wrote:
>but would that not mean that the bayes filter will learn the headers that
>spam assassin adds as spam .. and then after a while only start classing
>mail that already has the spam headers as bayes_99 ?
>
>I really do not know, I am just asking.
Erm.. no. Bayes doesn't even examine SA's own headers, so it won't learn
from that.. ever.
SA's bayes would also be absurdly broken if something like that could shake
up the classification.. Bayes is VERY robust. It's not uncommon for
messages to hit over 30 different bayes tokens. The state of just one token
is pretty much a non-issue for bayes. One addball won't overthrow a dozen
others.
Re: bayesian filter training
Posted by Thomas Arend <ml...@arend-whv.info>.
Am Freitag, 11. Februar 2005 13:17 schrieb Peter Marshall:
> but would that not mean that the bayes filter will learn the headers
> that spam assassin adds as spam .. and then after a while only start
> classing mail that already has the spam headers as bayes_99 ?
>
> I really do not know, I am just asking.
Before learning the headers added by SA are removed. sa-learn does a
spamassassin -d implizit.
see man sa-learn:
SpamAssassin remembers which mail messages it has learnt
already, and will not re-learn those messages again,
unless you use the --forget option. Messages learnt as
spam will have SpamAssassin markup removed, on the fly.
RTFM
Thomas
[..]
--
icq:133073900
http://www.t-arend.de
RE: bayesian filter training
Posted by Sander Holthaus - Orange XL <in...@orangexl.com>.
> but would that not mean that the bayes filter will learn the
> headers that spam assassin adds as spam .. and then after a
> while only start classing mail that already has the spam
> headers as bayes_99 ?
You can set bayes to ignore certain header. See the documentation for more
info. I would think it does not learn X-Spam-* headers by default. Also, I
would try to add other spam-related headers as well, as those can only be
abused by spammers IMHO.
Kind Regards,
Sander Holthaus
Re: bayesian filter training
Posted by Peter Marshall <pe...@caris.com>.
but would that not mean that the bayes filter will learn the headers
that spam assassin adds as spam .. and then after a while only start
classing mail that already has the spam headers as bayes_99 ?
I really do not know, I am just asking.
Peter
Matt Kettler wrote:
> At 05:06 PM 2/10/2005, Matias Lopez Bergero wrote:
>
>> Just a question,
>> It is worth to train the bayes filter with messages already detected
>> and flagged as spam by spamassassin? That would do any good?
>
>
> Yes. And even if they are already flagged as BAYES_99 it is still
> worthwhile.
>
>
> The reason why is that bayes does not learn that a message is spam or
> not. Bayes learns that a given set of words and tokens were seen in
> spam. A given spam message might be scored as spam and might already
> score high on the bayes scale, but it can still contain valuable new
> words to learn from. In particular the constant mutations of ways of
> spelling drug names provides a constant stream of fresh new spam
> indicators to for bayes learn about. Learning about these helps it
> identify future spam messages that might not otherwise look very
> spam-like, and offers you some protection from false negatives caused by
> spam mutations.
>
>
> The only time it's not worthwhile is if the message was already learned
> as spam (ie: by the autolearner).. but in that case SA will just ignore
> you. You're wasting some cpu time, but you won't damage or corrupt
> anything.
>
Re: bayesian filter training
Posted by Matt Kettler <mk...@evi-inc.com>.
At 05:06 PM 2/10/2005, Matias Lopez Bergero wrote:
>Just a question,
>It is worth to train the bayes filter with messages already detected and
>flagged as spam by spamassassin? That would do any good?
Yes. And even if they are already flagged as BAYES_99 it is still worthwhile.
The reason why is that bayes does not learn that a message is spam or not.
Bayes learns that a given set of words and tokens were seen in spam. A
given spam message might be scored as spam and might already score high on
the bayes scale, but it can still contain valuable new words to learn from.
In particular the constant mutations of ways of spelling drug names
provides a constant stream of fresh new spam indicators to for bayes learn
about. Learning about these helps it identify future spam messages that
might not otherwise look very spam-like, and offers you some protection
from false negatives caused by spam mutations.
The only time it's not worthwhile is if the message was already learned as
spam (ie: by the autolearner).. but in that case SA will just ignore you.
You're wasting some cpu time, but you won't damage or corrupt anything.