new rule->sa-update speedup idea (was Re: spam attacks - so and so wrote about a stock )

[Justin Mason <jm...@jmason.org>]

Theo Van Dinter writes:
> On Wed, Oct 18, 2006 at 09:06:15AM -0700, Ken A wrote:
> > Any thoughts on how to best address this issue, other than every SA
> > admin on the planet writing their own rules every evening? I'd note that
> > these are not identical spams, but are somewhat limited in their
> > 'variety', and they are hitting DCC now.
> 
> Welcome to anti-spam rule development. :)   These are no different than
> anything before...  People need to be paying attention to what's being
> sent/received.  Rules need to be written and tested.  The good ones can
> be distributed.
> 
> Via sa-update, new rules can be distributed very quickly, so it's really
> about the time necessary to develop and test new rules, which generally
> speaking comes down to manpower (there's some technology involved as well,
> but that's addressable).

Actually, on this point, I had an idea.

Currently, we have this worst-case scenario with sa-update:

  - day 1: 0930 UTC: developer writes good rule, checks it into
    rulesrc/sandbox/dev/20_whatever.cf
  - day 1: [nothing interesting happens until next nightly-mass-check tag]
  - day 2: 0900 UTC: SVN is tagged; all mass-checkers check out of SVN and
    start mass-checks
  - day 2: [allow time for mass-checks]
  - day 3: 0830 UTC: updatesd runs "build/mkupdates/run_nightly", collates
    mass-check results, adds rule to "rules/active.list"
  - day 3: 0850 UTC: new update is published, containing the rule

So, in other words, the worst-case scenario is that it'd take just under 2
days to get the rule into a packaged, released update.

I was thinking we could have a fast-reaction mode for just-created rules:

  - day 1: 0930 UTC: developer writes good rule, checks it into
    rulesrc/sandbox/dev/20_whatever.cf
  - day 1: 0931 UTC: bbmass preflight mass-check runs; new rule gets 1.0
    S/O on that limited corpus set
  - day 2: 0830 UTC: updatesd runs "build/mkupdates/run_nightly", collates
    mass-check results, adds rule to "rules/active.list"; also, adds
    newly-added rules that scored 1.0 S/O on the most recent preflight
    mass-check, and are not appearing in the nightly mass-check results
    yet
  - day 2: 0850 UTC: new update is published, containing the rule

in other words, reducing the worst-case scenario to just under 1 day. (If
we were to increase frequency of update publishing in the future, that
would then reduce that further, if necessary.)

Rules that got promoted based on "being new" and having a 1.0 S/O in the
preflight mass-checks would then only *stay* promoted if they then passed
the normal, existing promotion criteria -- so a rule that was good
"enough" to get into the update due to a 1.0 S/O, but had FPs on the
larger test set, would fall out anyway after 1 day.

--j.

Re: new rule->sa-update speedup idea (was Re: spam attacks - so and so wrote about a stock )

[Duncan Findlay <du...@debian.org>]

On Wed, Oct 18, 2006 at 06:07:01PM +0100, Justin Mason wrote:

> Theo Van Dinter writes:
> in other words, reducing the worst-case scenario to just under 1 day. (If
> we were to increase frequency of update publishing in the future, that
> would then reduce that further, if necessary.)

> Rules that got promoted based on "being new" and having a 1.0 S/O in the
> preflight mass-checks would then only *stay* promoted if they then passed
> the normal, existing promotion criteria -- so a rule that was good
> "enough" to get into the update due to a 1.0 S/O, but had FPs on the
> larger test set, would fall out anyway after 1 day.


I think I'd want to see a spam% restriction on there
too. Unfortunately, this probably wont help, since (correct me if I'm
wrong) the preflight mass-checks are old messages, not brand new ones,
right? This would mean they wouldn't get a good S/O ratio anyways.

-- 
Duncan Findlay

Re: new rule->sa-update speedup idea (was Re: spam attacks - so and so wrote about a stock )

[Duncan Findlay <du...@debian.org>]

On Wed, Oct 18, 2006 at 06:07:01PM +0100, Justin Mason wrote:

> Theo Van Dinter writes:
> in other words, reducing the worst-case scenario to just under 1 day. (If
> we were to increase frequency of update publishing in the future, that
> would then reduce that further, if necessary.)

> Rules that got promoted based on "being new" and having a 1.0 S/O in the
> preflight mass-checks would then only *stay* promoted if they then passed
> the normal, existing promotion criteria -- so a rule that was good
> "enough" to get into the update due to a 1.0 S/O, but had FPs on the
> larger test set, would fall out anyway after 1 day.


I think I'd want to see a spam% restriction on there
too. Unfortunately, this probably wont help, since (correct me if I'm
wrong) the preflight mass-checks are old messages, not brand new ones,
right? This would mean they wouldn't get a good S/O ratio anyways.

-- 
Duncan Findlay