You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spamassassin.apache.org by Marc Perkel <ma...@perkel.com> on 2004/12/27 15:45:53 UTC
A Feature I've always wanted - Test for multiple hits on same rule
This may be something that is too CPU intensive but maybe it could work.
I've always wanted to be able to see if a message triggered a rule
several times and if so - had a higher score depending on the number of
times the rule was triggered. Might even cap it at 3 times and do
something like this.
One hit = Score
Two hits = score * 1.20
three hits = score * 1.30
Any thoughts on this?
Re: A Feature I've always wanted - Test for multiple hits on same rule
Posted by Loren Wilton <lw...@earthlink.net>.
> Any thoughts on this?
For certain rules I think it would be a great idea.
The trick is not so much running the rule globally, as it is getting the hit
count to use in the score generation. I don't think (although I may be
wrong) that Perl can tell you how many times a regex hit in a global match.
You'ld probably have to run it in a loop by hand somehow.
Of course a second minor trick would be telling SA that you wanted this done
to the rule, and what the multiplier should be for each additional hit. You
might also want to give a maximum hit count that you care about to keep the
re from being run over-much in really bad cases.
One case I have in my rules is counting the number of recipients at the same
domain in the message. No score for <3, and increasing scores for more hits
up to about 8, which is a guaranteed spam for me. Not a rule for everyone,
but illustrative of what I think might want to be done.
Loren
Re: A Feature I've always wanted - Test for multiple hits on same
rule
Posted by Marc Perkel <ma...@perkel.com>.
Thanks Loren for supporting the concept. Yes - some sort of thing like
this should be implemented. I leave it to the smart people to figure out
how.
Loren Wilton wrote:
>>I'd have to take this into account when optimising the scores. Then,
>>since the scores would be optimised for multiple hits, spammers would
>>only have to reduce the number of hits to evade SpamAssassin.
>>
>>
>
>This strikes me as more of an implementation problem than an argument
>against the concept.
>
>The fact is that spammers do do some things in repeated manners in their
>spams, and those repeats are good detections of spam. They could remove the
>repeated things, but the spam might be less effective if they did. Or
>alternately, they might not know how to remove the repeated things, in which
>case the rule remains effective.
>
>The fact that a spammer may be able to adapt to a spam catcher and work
>around it in no way degrades the usefulness of the spam catcher in general.
>It merely degrades it in specific, for those spammers able to work around
>it.
>
>Off the top of my head, I'd guess that between SARE, Fred, and I, we
>probably have 30+ rules that are basically escalating scores for multiple
>hits. They work well. And spammers aren't working around them; the rules
>are still effective.
>
>The only problem at the moment is you have to write those escalating rules
>as a series of rules with increasing numbers of a repeated pattern, and then
>adjust the separate scores to sum correctly. Overall it is more work for
>the person generating rules, and it is more runtime effort for SA to run
>multiple sets of what is essentially the same rule.
>
>Having syntax for this sort of thing would eliminate both of those
>annoyances.
>
> Loren
>
>
>
>
Re: A Feature I've always wanted - Test for multiple hits on same rule
Posted by Loren Wilton <lw...@earthlink.net>.
> I'd have to take this into account when optimising the scores. Then,
> since the scores would be optimised for multiple hits, spammers would
> only have to reduce the number of hits to evade SpamAssassin.
This strikes me as more of an implementation problem than an argument
against the concept.
The fact is that spammers do do some things in repeated manners in their
spams, and those repeats are good detections of spam. They could remove the
repeated things, but the spam might be less effective if they did. Or
alternately, they might not know how to remove the repeated things, in which
case the rule remains effective.
The fact that a spammer may be able to adapt to a spam catcher and work
around it in no way degrades the usefulness of the spam catcher in general.
It merely degrades it in specific, for those spammers able to work around
it.
Off the top of my head, I'd guess that between SARE, Fred, and I, we
probably have 30+ rules that are basically escalating scores for multiple
hits. They work well. And spammers aren't working around them; the rules
are still effective.
The only problem at the moment is you have to write those escalating rules
as a series of rules with increasing numbers of a repeated pattern, and then
adjust the separate scores to sum correctly. Overall it is more work for
the person generating rules, and it is more runtime effort for SA to run
multiple sets of what is essentially the same rule.
Having syntax for this sort of thing would eliminate both of those
annoyances.
Loren
Re: A Feature I've always wanted - Test for multiple hits on same rule
Posted by Loren Wilton <lw...@earthlink.net>.
> If I don't take it into account when optimizing the scores, then the
> increased scores will cause more false positive errors.
What you probably need to take into account is the cumulative score for the
rule in the test corpus. Which of course you do for all rules. The only
oddity you would have to account for is you would have to know the effective
multipler that was used on the base score in the corpus, so that you could
set the base score correctly. I think that is a piece of data that you
probably don't have currently for rules, since it is implicitly 1.0.
Of course, if you wanted to tune the multipler and min/max hits values for
the rule it becomes vastly harder. I think that might be best left to some
other hunk of corpus-analysis software that generates some statistics for
multiple-hit rules and lets a human make the final decisions. There would
be few enough of this sort of rule to make that feasible. Once those values
were set by a human it would be easy enough to factor in the effective
multiplier on the test corpus when setting the score for the rule.
Loren
Re: A Feature I've always wanted - Test for multiple hits on same rule
Posted by Henry Stern <he...@stern.ca>.
If I don't take it into account when optimizing the scores, then the
increased scores will cause more false positive errors.
Henry
Marc Perkel wrote:
> I'm not sure you would have to take this into account - but it would be
> interesting to test the results. I would want just a couple of system
> settable commands like:
>
> TwoHitFactor = 1.2
> ThreeHitFactor = 1.3
>
> If unset 1.0 is assumed.
>
> Is this something that can be tested? I think it would be worth a try.
>
> Henry Stern wrote:
>
>> I'd have to take this into account when optimising the scores. Then,
>> since the scores would be optimised for multiple hits, spammers would
>> only have to reduce the number of hits to evade SpamAssassin.
>>
>> It's the same reason why we use a Bernoulli event model in Bayes.
>>
>> Henry
>>
>> Marc Perkel wrote:
>>
>>> This may be something that is too CPU intensive but maybe it could work.
>>>
>>> I've always wanted to be able to see if a message triggered a rule
>>> several times and if so - had a higher score depending on the number of
>>> times the rule was triggered. Might even cap it at 3 times and do
>>> something like this.
>>>
>>> One hit = Score
>>> Two hits = score * 1.20
>>> three hits = score * 1.30
>>>
>>> Any thoughts on this?
>>
>>
RE: A Feature I've always wanted - Test for multiple hits on same rule
Posted by Gary Funck <ga...@intrepid.com>.
> -----Original Message-----
> From: Marc Perkel [mailto:marc@perkel.com]
> Sent: Monday, December 27, 2004 7:35 AM
> To: Spamassassin Dev List
> Subject: Re: A Feature I've always wanted - Test for multiple hits on
> same rule
>
>
> I'm not sure you would have to take this into account - but it would be
> interesting to test the results. I would want just a couple of system
> settable commands like:
>
> TwoHitFactor = 1.2
> ThreeHitFactor = 1.3
>
This isn't germane to the scoring algorithm per se, but perhaps more having
to do with fine-tuning the Bayesian algorithm, somewhat along the lines you
outline above:
http://www.andrew.cmu.edu/user/dgovinda/pdf/multinomial-aaaiws98.pdf
Abstract
Recent approaches to text classifcation have used two
different first-order probabilistic models for classification
both of which make the naive Bayes assumption.
Some use a multi-variate Bernoulli model, that is, a
Bayesian Network with no dependencies between words
and binary word features, e.g. Larkey and Croft,
Koller and Sahami
Others use a multinomial
model, that is, a uni-gram language model with integer
word counts, e.g. Lewis and Gale, Mitchell.
This paper aims to clarify the confusion by describing
the differences and details of these two models and by
empirically comparing their classification performance
on five text corpora. We find that the multi-variate
Bernoulli performs well with small vocabulary sizes,
but that the multinomial performs usually performs
even better at larger vocabulary sizes providing on
average a 27% reduction in error over the multi-variate
Bernoulli model at any vocabulary size.
Re: A Feature I've always wanted - Test for multiple hits on same
rule
Posted by Marc Perkel <ma...@perkel.com>.
I'm not sure you would have to take this into account - but it would be
interesting to test the results. I would want just a couple of system
settable commands like:
TwoHitFactor = 1.2
ThreeHitFactor = 1.3
If unset 1.0 is assumed.
Is this something that can be tested? I think it would be worth a try.
Henry Stern wrote:
> I'd have to take this into account when optimising the scores. Then,
> since the scores would be optimised for multiple hits, spammers would
> only have to reduce the number of hits to evade SpamAssassin.
>
> It's the same reason why we use a Bernoulli event model in Bayes.
>
> Henry
>
> Marc Perkel wrote:
>
>> This may be something that is too CPU intensive but maybe it could work.
>>
>> I've always wanted to be able to see if a message triggered a rule
>> several times and if so - had a higher score depending on the number of
>> times the rule was triggered. Might even cap it at 3 times and do
>> something like this.
>>
>> One hit = Score
>> Two hits = score * 1.20
>> three hits = score * 1.30
>>
>> Any thoughts on this?
>
Re: A Feature I've always wanted - Test for multiple hits on same rule
Posted by Henry Stern <he...@stern.ca>.
I'd have to take this into account when optimising the scores. Then,
since the scores would be optimised for multiple hits, spammers would
only have to reduce the number of hits to evade SpamAssassin.
It's the same reason why we use a Bernoulli event model in Bayes.
Henry
Marc Perkel wrote:
> This may be something that is too CPU intensive but maybe it could work.
>
> I've always wanted to be able to see if a message triggered a rule
> several times and if so - had a higher score depending on the number of
> times the rule was triggered. Might even cap it at 3 times and do
> something like this.
>
> One hit = Score
> Two hits = score * 1.20
> three hits = score * 1.30
>
> Any thoughts on this?