You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Robert Fitzpatrick <li...@webtent.net> on 2007/04/19 15:18:12 UTC

Rules report

I've seen some others on the list here show reports of the different
rules and how much they hit. How can I produce these reports? And is it
possible to produce a report like this by domain name?

-- 
Robert


Re: Rules report

Posted by Robert Fitzpatrick <li...@webtent.net>.
On Thu, 2007-04-19 at 15:03 +0100, Chris Lear wrote:
> * Matt Kettler wrote (19/04/07 14:49):
<snip>
> > If you want to know how accurate a particular rule is, by comparing the
> > spam vs nonspam hit rates, those stats are useless, because of the bias.
> > You need a manually sorted corpus to get this kind of information.
> > 
> > If you want to see which rules are getting used a lot, vs those that are
> > rarely getting used, these stats are quite useful.
> > 
> > If you want a "top x rules" list, sa-stats can do that for you:
> > 
> > http://www.rulesemporium.com/programs/sa-stats.txt
> 
> http://www.rulesemporium.com/programs/sa-stats-1.0.txt is probably a bit 
> better in this case.
> 
> > 
> > It will parse a spamd logfile and report the most-frequently used spam
> > and nonspam rules (and you can configure how many it will list for each)
> 
> The 1.0 version can do per-domain and per-user info, given a 3.1 log.

Yes, this is all I'm after, but we use Amavisd-new to pass off to SA,
not spamd. The amavisd logs don't seem to show that information. Will it
work? Or is there a way to do this with amavisd?

-- 
Robert


Re: Rules report

Posted by Chris Lear <ch...@laculine.com>.
* Matt Kettler wrote (19/04/07 14:49):
> Matt Kettler wrote:
>> If you try to build it off a live feed and use SA's marking as the spam
>> criteria, your statistics are useless. Any rule with a high enough score
>> would get "perfect" results.. all the mail it matched would be spam, and
>> no nonspam. You have, essentially, created a "self fulfilling prophecy".
>> The higher-scoring a rule is, the more likely messages that match it
>> will be tagged as spam, even if they're not really spam.
>>   
> Self correction. Such stats aren't "useless", it depends on what you
> want out of them.
> 
> If you want to know how accurate a particular rule is, by comparing the
> spam vs nonspam hit rates, those stats are useless, because of the bias.
> You need a manually sorted corpus to get this kind of information.
> 
> If you want to see which rules are getting used a lot, vs those that are
> rarely getting used, these stats are quite useful.
> 
> If you want a "top x rules" list, sa-stats can do that for you:
> 
> http://www.rulesemporium.com/programs/sa-stats.txt

http://www.rulesemporium.com/programs/sa-stats-1.0.txt is probably a bit 
better in this case.

> 
> It will parse a spamd logfile and report the most-frequently used spam
> and nonspam rules (and you can configure how many it will list for each)

The 1.0 version can do per-domain and per-user info, given a 3.1 log.

Chris

Re: Rules report

Posted by Matt Kettler <mk...@verizon.net>.
Matt Kettler wrote:
> If you try to build it off a live feed and use SA's marking as the spam
> criteria, your statistics are useless. Any rule with a high enough score
> would get "perfect" results.. all the mail it matched would be spam, and
> no nonspam. You have, essentially, created a "self fulfilling prophecy".
> The higher-scoring a rule is, the more likely messages that match it
> will be tagged as spam, even if they're not really spam.
>   
Self correction. Such stats aren't "useless", it depends on what you
want out of them.

If you want to know how accurate a particular rule is, by comparing the
spam vs nonspam hit rates, those stats are useless, because of the bias.
You need a manually sorted corpus to get this kind of information.

If you want to see which rules are getting used a lot, vs those that are
rarely getting used, these stats are quite useful.

If you want a "top x rules" list, sa-stats can do that for you:

http://www.rulesemporium.com/programs/sa-stats.txt

It will parse a spamd logfile and report the most-frequently used spam
and nonspam rules (and you can configure how many it will list for each)

>
>   
>


Re: Rules report

Posted by Matt Kettler <mk...@verizon.net>.
Robert Fitzpatrick wrote:
> I've seen some others on the list here show reports of the different
> rules and how much they hit. 
Most of them are quoting the ones out of the official ruleset mass-check
results. Those are in the tarball under the rules directory as
STATISTICS*.txt
> How can I produce these reports?
Generate a hand sorted set (corpus) of spam and nonspam messages and
then feed them into the mass-check tool.

Note: this is really a developer tool, so its use should be considered
"advanced"

See also:
http://wiki.apache.org/spamassassin/MassCheck


>  And is it
> possible to produce a report like this by domain name?
>   
By domain? Sure, create a separate corpus for each.

That said, it sounds like you're thinking of using this to monitor your
live mail feeds.

It's impossible to produce these reports accurately on live email. You
must have a hand-sorted set of spam and nonspam to work with, that way
the tool knows for sure when a rule is matching spam, or nonspam.

If you try to build it off a live feed and use SA's marking as the spam
criteria, your statistics are useless. Any rule with a high enough score
would get "perfect" results.. all the mail it matched would be spam, and
no nonspam. You have, essentially, created a "self fulfilling prophecy".
The higher-scoring a rule is, the more likely messages that match it
will be tagged as spam, even if they're not really spam.