You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by tonjg <to...@freeuk.com> on 2010/03/18 00:47:17 UTC
bayes, numbers of tokens and performance
I've done a bayes learn on 500 spams and 400 hams, my spam hit threshold is 5
and I'm getting a success rate of about 40% in identifying spam, and that's
after doing an sa-update too. I was hoping to get better results than this.
How many spam and ham tokens does SA normally need before it really starts
to work well ie: 90%+ success rate identifying spam?
--
View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27940005.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: bayes, numbers of tokens and performance
Posted by Bowie Bailey <Bo...@BUC.com>.
tonjg wrote:
> Kai Schaetzl wrote:
>
>> So, how many tokens do you have in your db now?
>>
>
> I hope this command gives the correct answer...
> # sa-learn --dump magic
> 0.000 0 3 0 non-token data: bayes db version
> 0.000 0 514 0 non-token data: nspam
> 0.000 0 402 0 non-token data: nham
> 0.000 0 54301 0 non-token data: ntokens
As long as you are running sa-learn as the same user SA uses when it
checks mail, this looks ok.
As has been asked before, post a sample missed spam or two to a pastebin
and then post the links here. That way, we will be able to see what
rules are hitting, test the spam on our systems to see what happens, and
give you some suggestions. With a hit-rate as low as you are reporting,
there has to be something broken, but until we can see what your system
is doing, we can't give you much constructive advice.
You may also want to add the line:
add_header all Report _REPORT_
to your local.cf. This will add the full SA header report to all emails
and make it easier to see which rules are hitting on the missed spams.
--
Bowie
Re: bayes, numbers of tokens and performance
Posted by tonjg <to...@freeuk.com>.
Kai Schaetzl wrote:
> I've been running
> with 2 million with no problems.
based on this I see you're right that my db's are tiny and not enough for
the success rate I'm aiming for.
--
View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27946954.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: bayes, numbers of tokens and performance
Posted by Kai Schaetzl <ma...@conactive.com>.
Tonjg wrote on Thu, 18 Mar 2010 05:17:21 -0700 (PDT):
> I hope this command gives the correct answer...
> # sa-learn --dump magic
> 0.000 0 3 0 non-token data: bayes db version
> 0.000 0 514 0 non-token data: nspam
> 0.000 0 402 0 non-token data: nham
> 0.000 0 54301 0 non-token data: ntokens
Yes, that's not very much. I don't have such "tiny" dbs, so I don't know
how well this works. As a gut feeling I'd say you want at least 200.000
tokens. Fresh tokens, not tokens from last year's spam and ham.
My systems usually have at least half a million tokens. I've been running
with 2 million with no problems.
Kai
--
Get your web at Conactive Internet Services: http://www.conactive.com
Re: bayes, numbers of tokens and performance
Posted by tonjg <to...@freeuk.com>.
Kai Schaetzl wrote:
>
> So, how many tokens do you have in your db now?
I hope this command gives the correct answer...
# sa-learn --dump magic
0.000 0 3 0 non-token data: bayes db version
0.000 0 514 0 non-token data: nspam
0.000 0 402 0 non-token data: nham
0.000 0 54301 0 non-token data: ntokens
--
View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27944771.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: bayes, numbers of tokens and performance
Posted by Kai Schaetzl <ma...@conactive.com>.
So, how many tokens do you have in your db now?
Kai
--
Get your web at Conactive Internet Services: http://www.conactive.com
Re: bayes, numbers of tokens and performance
Posted by Kevin Parris <KP...@ed.sc.gov>.
It doesn't really work that way. Bayes is just one part of the picture and in order to get good results you have to turn the full toolkit loose on the problem; I'm not sure Bayes by itself should be expected to achieve 95% recognition anyway. The main flaw in your current plan is that once you re-activate the BLs then your Bayes content will begin to get stale - and effectiveness is likely then to decline over time. Bayes tends to work better when trained continuously on current traffic. Rather than stop using other tools, just to get some spam to train with, perhaps you should focus more on training Bayes more actively with the spam that gets through otherwise.
You're not likely ever to detect ALL the spam traffic, no matter what combination of tools you deploy - there will always be clever spammers working on ways to bypass them.
>>> tonjg <to...@freeuk.com> 03/18/10 11:04 AM >>>
Matus UHLAR - fantomas wrote:
>
>> DNS available?
>> no
>
> well, why? DNS helps very much for catching spam. all blacklists use DNS
> (afaik)
sorry, when you said dns I didn't know you were referring to the dnsbl's. I
know the black lists are excellent for filtering spam but I've got those
switched off so I can actually accumulate some spam for the sa-learn. I
figured if I get spamassassin working really well first (ie: a 95% success
rate) I would then switch the bl's back on and use both.
Re: bayes, numbers of tokens and performance
Posted by tonjg <to...@freeuk.com>.
Matus UHLAR - fantomas wrote:
>
>> DNS available?
>> no
>
> well, why? DNS helps very much for catching spam. all blacklists use DNS
> (afaik)
sorry, when you said dns I didn't know you were referring to the dnsbl's. I
know the black lists are excellent for filtering spam but I've got those
switched off so I can actually accumulate some spam for the sa-learn. I
figured if I get spamassassin working really well first (ie: a 95% success
rate) I would then switch the bl's back on and use both.
--
View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27946914.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: bayes, numbers of tokens and performance
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
> Mikael Syska wrote:
> > Does it help when you sa-learn the spams ? Does it change the BAYES_
> > score for that mail ?
On 18.03.10 05:20, tonjg wrote:
> I'm going to do another sa-learn when I hit 100 more spams and I'll see then
> if it makes a difference.
learn whenever possible, mostly on mismatched spams and hams.
you can pipe individual mail to "spamassassin -r" and "spamasssassin -k"
to do that.
> In the meantime I've lowered my hit threshold to 4.
so you get more false positives.
> DNS available?
> no
well, why? DNS helps very much for catching spam. all blacklists use DNS
(afaik)
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Chernobyl was an Windows 95 beta test site.
Re: bayes, numbers of tokens and performance
Posted by Mikael Syska <mi...@syska.dk>.
Hi
On Thu, Mar 18, 2010 at 1:20 PM, tonjg <to...@freeuk.com> wrote:
>
>
> Mikael Syska wrote:
>>
>> Does it help when you sa-learn the spams ? Does it change the BAYES_
>> score for that mail ?
> I'm going to do another sa-learn when I hit 100 more spams and I'll see then
> if it makes a difference. In the meantime I've lowered my hit threshold to
> 4.
>
>
> DNS available?
> no
no DNS ... then RBL etc wont work, and maybe thats why is scores so
awful.... do enable DNS.
Still ... you havent told us anything really ... what rules does some
of the mails hit that are catched? and what rules does mails hit that
arent catched ...
This seems like its your first SA installation, or ?
If, theres alot to read about how SA score mails.
> --
> View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27944806.html
> Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
>
>
muh
Mikael Syska
Re: bayes, numbers of tokens and performance
Posted by tonjg <to...@freeuk.com>.
Jason Bertoch-2 wrote:
> You should really try to determine why your system isn't
> performing well first.
ok I've changed it back to 5
--
View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27947096.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: bayes, numbers of tokens and performance
Posted by Jason Bertoch <ja...@i6ix.com>.
On 2010/03/18 10:56 AM, tonjg wrote:
>
> Kai Schaetzl wrote:
>> Don't do that.
>
> why not?
>
Rule scores are generated based on a default required_score of 5.
Fiddling with the required_score should be the _last_ thing you do, if
at all. You should really try to determine why your system isn't
performing well first.
--
/Jason
Re: bayes, numbers of tokens and performance
Posted by tonjg <to...@freeuk.com>.
Kai Schaetzl wrote:
> Don't do that.
why not?
--
View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27946788.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: bayes, numbers of tokens and performance
Posted by Kai Schaetzl <ma...@conactive.com>.
Tonjg wrote on Thu, 18 Mar 2010 05:20:45 -0700 (PDT):
> I'm going to do another sa-learn when I hit 100 more spams and I'll see then
> if it makes a difference. In the meantime I've lowered my hit threshold to
> 4.
Don't do that.
Kai
--
Get your web at Conactive Internet Services: http://www.conactive.com
Re: bayes, numbers of tokens and performance
Posted by tonjg <to...@freeuk.com>.
Mikael Syska wrote:
>
> Does it help when you sa-learn the spams ? Does it change the BAYES_
> score for that mail ?
I'm going to do another sa-learn when I hit 100 more spams and I'll see then
if it makes a difference. In the meantime I've lowered my hit threshold to
4.
DNS available?
no
--
View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27944806.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: bayes, numbers of tokens and performance
Posted by Mikael Syska <mi...@syska.dk>.
Hi
On Thu, Mar 18, 2010 at 1:35 AM, tonjg <to...@freeuk.com> wrote:
>
> it's not 'a' spam I'm referring to, it lots of different spams getting
> through to my inbox with only 40% of them being identified as spam.
Yes, that's awful, I would be killed if that many spams got though our
filters ....
What rules does the mails hit? Not even a standard system with default
rules, when sa-update have been run have such a bad catch rate ....
unless you maybe only got hit by very rare spam mails.
Does it help when you sa-learn the spams ? Does it change the BAYES_
score for that mail ?
So there must to something that is not configured right ... so please
post what rules it hit, that way its easier for us to help ...
DNS available?
>This strikes me as a poor success rate. What do you mean by 'crappy mail bayes' ?
> the bayes learn was done on spam and ham that ended up in my inbox.
Needs to be all kinds of spam and ham ... but I guess thats in place.
> --
> View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27940352.html
> Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
>
>
muh
Mikael Syska
Re: bayes, numbers of tokens and performance
Posted by tonjg <to...@freeuk.com>.
it's not 'a' spam I'm referring to, it lots of different spams getting
through to my inbox with only 40% of them being identified as spam. This
strikes me as a poor success rate. What do you mean by 'crappy mail bayes' ?
the bayes learn was done on spam and ham that ended up in my inbox.
--
View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27940352.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: bayes, numbers of tokens and performance
Posted by Mikael Syska <mi...@syska.dk>.
Hi,
What score does the spam have .... post it with full header on
nomorepasting.com or similar ... then we can try what it hits on out
setups ...
On Thu, Mar 18, 2010 at 12:47 AM, tonjg <to...@freeuk.com> wrote:
>
> I've done a bayes learn on 500 spams and 400 hams, my spam hit threshold is 5
> and I'm getting a success rate of about 40% in identifying spam, and that's
> after doing an sa-update too.
Bayes alone with stardard score on 3.2.5 wont get it higher than 3.5
with BAYES_99 ...
And since yours is 5.0 ... you must be scoring on something else too ...
> I was hoping to get better results than this.
> How many spam and ham tokens does SA normally need before it really starts
> to work well ie: 90%+ success rate identifying spam?
It should be working now ... unless its crappy mail bayes ... its
needs to reflect what kind of spam and ham you are getting ...
> --
> View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27940005.html
> Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
>
>
mvh
Mikael Syska
http://ifyoudo.net
Re: bayes, numbers of tokens and performance
Posted by tonjg <to...@freeuk.com>.
Matus UHLAR - fantomas wrote:
>
> do you have network checks enabled? Do you have network plugins (razor,
> pyzor, dcc, uribl) loaded? Do you have other plugins (like textcat)
> loaded?
no, I am unfamilar with these plugins.
which version of SA do you have installed?
version 3.2.5-1.el4.rf from rpmforge
--
View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27944717.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: bayes, numbers of tokens and performance
Posted by Bowie Bailey <Bo...@BUC.com>.
tonjg wrote:
> Yet Another Ninja wrote:
>
>> Did you also install the plugins?
>> These two are not delivered with SA.
>>
>
> I thought they were. In my system I've got:
> /var/lib/spamassassin/3.002005/updates_spamassassin_org/25_razor2.cf
> /usr/share/spamassassin/25_razor2.cf
> and
> /var/lib/spamassassin/3.002005/updates_spamassassin_org/25_pyzor.cf
> /usr/share/spamassassin/25_pyzor.cf
The rules needed for SA to work with Razor and Pyzor are included, but
you have to install the plugins yourself.
http://razor.sourceforge.net/
http://sourceforge.net/apps/trac/pyzor/
They are both pretty easy to setup.
--
Bowie
Re: bayes, numbers of tokens and performance
Posted by tonjg <to...@freeuk.com>.
Yet Another Ninja wrote:
>
> Did you also install the plugins?
> These two are not delivered with SA.
>
I thought they were. In my system I've got:
/var/lib/spamassassin/3.002005/updates_spamassassin_org/25_razor2.cf
/usr/share/spamassassin/25_razor2.cf
and
/var/lib/spamassassin/3.002005/updates_spamassassin_org/25_pyzor.cf
/usr/share/spamassassin/25_pyzor.cf
--
View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27947813.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: bayes, numbers of tokens and performance
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
> On 2010-03-18 16:36, tonjg wrote:
>> update: after doing some reading on google I found init.pre and added:
>> loadplugin Mail::SpamAssassin::Plugin::Razor2
>> and
>> loadplugin Mail::SpamAssassin::Plugin::Pyzor
>> and restarted spamassassin.
On 18.03.10 16:44, Yet Another Ninja wrote:
> Did you also install the plugins?
> These two are not delivered with SA.
not the plugins, the programs. Plugins are in SA distribution, but they
still need the programs...
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
You have the right to remain silent. Anything you say will be misquoted,
then used against you.
Re: bayes, numbers of tokens and performance
Posted by Yet Another Ninja <sa...@alexb.ch>.
On 2010-03-18 16:36, tonjg wrote:
> update: after doing some reading on google I found init.pre and added:
> loadplugin Mail::SpamAssassin::Plugin::Razor2
> and
> loadplugin Mail::SpamAssassin::Plugin::Pyzor
> and restarted spamassassin.
>
Did you also install the plugins?
These two are not delivered with SA.
Re: bayes, numbers of tokens and performance
Posted by tonjg <to...@freeuk.com>.
update: after doing some reading on google I found init.pre and added:
loadplugin Mail::SpamAssassin::Plugin::Razor2
and
loadplugin Mail::SpamAssassin::Plugin::Pyzor
and restarted spamassassin.
--
View this message in context: http://old.nabble.com/bayes%2C-numbers-of-tokens-and-performance-tp27940005p27947325.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: bayes, numbers of tokens and performance
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
On 17.03.10 16:47, tonjg wrote:
> I've done a bayes learn on 500 spams and 400 hams, my spam hit threshold is 5
> and I'm getting a success rate of about 40% in identifying spam, and that's
> after doing an sa-update too. I was hoping to get better results than this.
> How many spam and ham tokens does SA normally need before it really starts
> to work well ie: 90%+ success rate identifying spam?
do you have network checks enabled? Do you have network plugins (razor,
pyzor, dcc, uribl) loaded? Do you have other plugins (like textcat) loaded?
which version of SA do you have installed?
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Micro$oft random number generator: 0, 0, 0, 4.33e+67, 0, 0, 0...