question about testing new rulesets

[Rob McEwen <ro...@invaluement.com>]

RE: question about testing new rulesets

Is it possible to do the following when testing out a new ruleset:

(1) score that rule at 0.01 (of course this is possible... but then also...)

(2) copy the original source file that was "fed" to SA to a separate 
directory if (a) the new rule being tested triggered ...AND... (b) if 
that message ended up scoring "below threshold" and was therefore NOT 
considered spam.

This would allow someone to audit those messages which would ONLY have 
been blocked had that new ruleset been giving a higher score. Analysis 
on such messages could then be done to see how many of these are FNs and 
how many of these are FPs.

I'm thinking that, if SA can delete and re-write the source file with a 
new header, it seems like it could also copy the message to a different 
folder, under certain conditions?

Thanks!

-- 
Rob McEwen
http://dnsbl.invaluement.com/
rob@invaluement.com

Re: question about testing new rulesets

["Daryl C. W. O'Shea" <sp...@dostech.ca>]

On 03/10/2008 5:13 PM, Rob McEwen wrote:
> RE: question about testing new rulesets
> 
> Is it possible to do the following when testing out a new ruleset:
> 
> (1) score that rule at 0.01 (of course this is possible... but then
> also...)
> 
> (2) copy the original source file that was "fed" to SA to a separate
> directory if (a) the new rule being tested triggered ...AND... (b) if
> that message ended up scoring "below threshold" and was therefore NOT
> considered spam.

Yep.  The plugin you would need to do that is pretty trivial.

> This would allow someone to audit those messages which would ONLY have
> been blocked had that new ruleset been giving a higher score. Analysis
> on such messages could then be done to see how many of these are FNs and
> how many of these are FPs.
> 
> I'm thinking that, if SA can delete and re-write the source file with a

SA doesn't do that, but it's irrelevant I suppose.

Daryl

Re: question about testing new rulesets

[Martin Gregorie <ma...@gregorie.org>]

On Fri, 2008-10-03 at 17:13 -0400, Rob McEwen wrote:
> RE: question about testing new rulesets
> 
> Is it possible to do the following when testing out a new ruleset:
> 
As an alternative, what about something like this:

Accumulate a set of rule testing messages for regression testing and
write a script that submits the lot via spamc. The script grabs spamc
output, compares it with a set of expected results and reports match
failures in sufficient detail to identify the test message(s) and
rule(s) that don't give the expected results.

The regression test script should be easy enough to write using Perl or
a combination of a shell script to run the tests and an awk script to do
the comparisons.


Martin