[Spamassassin Wiki] Update of "RulesProjPromotion" by JustinMason

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Spamassassin Wiki" for change notification.

The following page has been changed by JustinMason:
http://wiki.apache.org/spamassassin/RulesProjPromotion

The comment on the change is:
refactor page, other was getting too big

New page:
= Rules Project: Promotion of Rules =

from the sandboxes to the core ruleset, that is.

(''this page split from RulesProjSandboxes, part of RulesProjPlan'')

Getting rules from the sandbox, into the distribution:

  * each user gets their own sandbox as discussed on RulesProjSandboxes
  * checked-in rules in the sandboxes are mass-checked in the nightly mass-checks
  * to migrate a rule from "sandbox" (dev) to "core" (production) ruleset uses C-T-R; ie. votes are not required in advance
  * also C-T-R to migrate from "sandbox" to "extra" ruleset

Rules that get promoted from a "sandbox" to "core" should pass the following criteria:

  * pass "--lint"!
  * S/O ratio of 0.95 or greater (or 0.05 or less for nice rules)
  * > 0.25% of target type hit (e.g. spam for non-nice rules)
  * < 1.00% of non-target type hit (e.g. ham for non-nice rules)

These numbers are really just ball-park figures and should be fine-tuned as we go. (DuncanFindlay)

We can automate those criteria pretty easily. We can also vote for rules that don't pass those criteria, but we think should be put into core for some reason. 

Future criteria:

  * not too slow ;)   TODO: need an automated way to measure that
  * TODO: criteria for overlap with existing rules? see 'overlap criteria' below.

== Moving files out of trunk into the new rules project ==

JustinMason: If we're going to start pulling rules from sandboxes into core/ in
the above fashion, but we leave the current ruleset intact in the
core as well, things will get messy.
I propose we move the current core ruleset into a sandbox, called
'rules/sandbox/legacy/'.  The good rules that pass the above selection
criteria, get promoted as any other rules from other sandboxes do, into the new
'core/'; the old, stale rules (of which we have a few), will not get back into
core.

DanielQuinlan: vetoed.  Instead: code-tied rules stay with main tree in current
rules directory, with the exception of 25_replace.cf which is really just
another way to write body/header rules.   Basically, the static stuff that is
tied to code does not move to the rules project.

In more detail -- files that DO NOT move to rules project:

{{{
      25_accessdb.cf    (plugins in core code)
      25_antivirus.cf
      25_dcc.cf
      25_domainkeys.cf
      25_hashcash.cf
      25_pyzor.cf
      25_razor2.cf
      25_spf.cf
      25_textcat.cf
      25_uribl.cf
      60_awl.cf
      60_whitelist_subject.cf
      20_dnsbl_tests.cf (eval tests in EvalTests.pm)
      20_html_tests.cf (rawbody ones can move to ROOT/rules/core/)
      20_net_tests.cf
      23_bayes.cf
      60_whitelist.cf
      init.pre          (Misc non-cf files)
      local.cf
      name-triplets.txt
      regression_tests.cf
      triplets.txt
      user_prefs.template
      v310.pre
}}}

Files that DO get moved:

{{{
   25_body_tests_es.cf -> ROOT/rules/lang/es/
   25_body_tests_pl.cf -> ROOT/rules/lang/pl/
   30_text_de.cf       -> ROOT/rules/lang/de/
   30_text_fr.cf       -> ROOT/rules/lang/fr/
   30_text_it.cf       -> ROOT/rules/lang/it/
   30_text_nl.cf       -> ROOT/rules/lang/nl/
   30_text_pl.cf       -> ROOT/rules/lang/pl/
   30_text_pt_br.cf    -> ROOT/rules/lang/pt_br/

   20_advance_fee.cf   -> ROOT/rules/core/
   20_drugs.cf         -> ROOT/rules/core/
   20_p**n.cf          -> ROOT/rules/core/    [wikicensorship!]

   10_misc.cf           -> ROOT/rules/core/
   20_anti_ratware.cf   -> ROOT/rules/core/
   20_body_tests.cf     -> ROOT/rules/core/
   20_compensate.cf     -> ROOT/rules/core/
   20_fake_helo_tests.cf -> ROOT/rules/core/
   20_head_tests.cf     -> ROOT/rules/core/
   20_meta_tests.cf     -> ROOT/rules/core/
   20_phrases.cf        -> ROOT/rules/core/
   20_ratware.cf        -> ROOT/rules/core/
   20_uri_tests.cf      -> ROOT/rules/core/
   25_replace.cf (odd case, but will change a lot) -> ROOT/rules/core/
   50_scores.cf         -> ROOT/rules/core/
   60_whitelist_spf.cf  -> ROOT/rules/core/
}}}

Files that get deleted: 20_anti_ratware.cf: it's empty.

JustinMason: ok, that looks good -- except for one thing.  We still have the problem that ROOT/rules/core/ is going to be a mix of legacy files and auto-promoted rules.  What do we do about that problem?

== Algorithm for auto-promotion ==

JustinMason: Aside from the criteria, we also need an idea of how the config file lines get from sandbox to core.  Here's my proposal.

For each sandbox directory:
 * iterate through all files in the dir
 * if a config line refers to a rule name (e.g. "header", "describe", "tflags"), then:
   * apply the criteria from 'Rule Promotion'.  if the rule passes:
     * output the line
   * else:
     * ignore the line and produce no output
 * if the config line doesn't refer to a rule name, output the line.
 * send that output to a file in ROOT/rules/core/ , named according to the sandbox directory's name.  e.g. lines from all files matching ROOT/rules/sandbox/jmason/*.cf would be output to ROOT/rules/core/25_jmason.cf