Re: initial rules organization ideas

[Justin Mason <jm...@jmason.org>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


Loren Wilton writes:
> Agree in general, but possibly...
> 
> > 2. code-tied rules stay with main tree in current rules directory with
> >    the exception of 25_replace.cf which is really just another way to
> >    write body/header rules (basically, the static stuff that is tied to
> >    code does not move to the rules project)
> >
> >    a. plugins:
> >
> >       25_accessdb.cf
> >       25_antivirus.cf
> >       25_dcc.cf
> >       25_domainkeys.cf
> >       25_hashcash.cf
> >       25_pyzor.cf
> >       25_razor2.cf
> >       25_spf.cf
> >       25_textcat.cf
> >       25_uribl.cf
> >       60_awl.cf
> >       60_whitelist_subject.cf
> 
> I think plugins that are basically rules should move to the rules project.
> It is probable that at some point the rules project will be creating
> plugins, and quite possibly want to modify existing plugins.
> 
> Would suggest something like /core/plugins for the base for plugins.
> 
> Might break that down into
>     /core/plugins
>     /core/plugins/net or /core/netplugins
> 
> The vast majority of plugins at the moment appear to be net plugins.  I
> suspect over time a more even balance of net and non-net plugin tests will
> come into existance.

I think we don't even need to do that; once we get the "search directories
recursively" code worked out for configuration and rules, plugins will be
loadable from *any* directory in the rules project:

    ROOT/rules/group/20_name_of_file.cf
    ROOT/rules/group/20_name_of_file.pm

You just use "loadplugin ./20_name_of_file.pm Name::Of::Class".

So there won't be a need for a special "plugins area".

I agree with Daniel than in the interim, we should keep those files he
listed, along with the code, at least until they can be disentangled and
moved to the rules project.   

BTW what he neglected to point out, was that the code and rules project
need to be able to have different branch structure.

This is what causes all the trouble -- moving the current set of code-tied
rules files en masse into the rules project would cause a lot of
fireworks, as soon as we eventually branched, since the *code* would
change but the *rules* files, which contain rules tied to that code, would
not.  Hence it's better to keep the most code-tied files with the code
(for now).

> Files like v310.pre are a little awkward.  I suspect this really needs to be
> thought out better in some way in general, people seem to be having a little
> problem with finding what is in plugins and determining what/where they are
> enabled or not.

well, that's our fault.  We keep moving stuff out of core into
not-enabled-by-default plugins, forcing people to *find* this file so
they can turn them back on.  ;)

(in fairness, there's generally good reasons, although I'm not sure why
we did this with ok_languages/TextCat?)

v310.pre and init.pre are important to keep beside the code, because
they list the most code-tied stuff of all -- the loadplugins lines as
they're presented to the user.

> However, it seems that the file that contains the (probably mostly disabled)
> plugin declarations needs to be tied in some way to the existing plugins.
> So maybe this file also would go in /core/plugins.

I think it needs to stick with the code -- see above; since it's
based on the .

> >    b. other standard code/evals:
> >
> >       20_dnsbl_tests.cf
> >       20_html_tests.cf (rawbody ones can move to ROOT/rules/core/)
> >       20_net_tests.cf
> >       23_bayes.cf
> >       60_whitelist.cf
> 
> Possibly there could be a /core/net directory to drop all of the net tests
> in that are not the actual plugin code itself.

I think they're all going to be 100% code-tied.   Although DNSBLs are
not... hmm! good point.

> >    c. miscellaneous stuff
> >
> >       init.pre
> >       local.cf
> >       name-triplets.txt
> >       regression_tests.cf
> >       triplets.txt
> >       user_prefs.template
> >       v310.pre
> 
> Agree, should not be part of rules, except probably for *.pre.

See above for the .pre stuff.

> > 3. spamware signature rules = 20_anti_ratware.cf
> 
> I assume you intended this category to go into /rules/core?

actually I think he meant "nuke it" -- it's an empty file ;)

Bob Menschel asked:
]DQ> 5. type-specific rules move to ROOT/rules/core/ for now
]DQ>    20_advance_fee.cf
]DQ>    20_drugs.cf
]DQ>    20_porn.cf 
]DQ> 6. and the rest to ROOT/rules/core/
]Why the separation of (5) and (6)?

I don't think there was any reason for that.  I'm certainly
ignoring the separation ;)

- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFDCoV5MJF5cimLx9ARAhC0AJ44IBb10VEC8isqkyNOocr8XQB8CwCgidD4
sUyVdux/ZaQN5rHl5Aj+SRc=
=vFvG
-----END PGP SIGNATURE-----

Re: initial rules organization ideas

[Loren Wilton <lw...@earthlink.net>]

Justin writes:

> I think we don't even need to do that; once we get the "search directories
> recursively" code worked out for configuration and rules, plugins will be
> loadable from *any* directory in the rules project:
>
>     ROOT/rules/group/20_name_of_file.cf
>     ROOT/rules/group/20_name_of_file.pm
>
> You just use "loadplugin ./20_name_of_file.pm Name::Of::Class".
>
> So there won't be a need for a special "plugins area".

I was originally thinking of a plugins area in the rules project more as an
organizational or classification measure than for any absolute need.  Same
with the idea of a 'net rules' area.

I hadn't thought of the problem of keeping rules project branches in sync
with code branches.  Definitely a problem, and probably a good reason to
leave the plugins in the main code at the moment.

That said: I would hope that in many cases a user plugin for 3.1 would work
on 3.2 and maybe even 4.0 without requiring major (or even minor) changes.

Likewise, I would hope that by 3.3 or so (or whatever time-period equivalent
comes to exist) things will have become stable enough in the plugin
interface area that the reasons for changing code in plugins will be to
change the functionality, and not to keep the interface in sync with a
moving target.

SARE currently has a bunch of rules files with names like _pre_30 and _25x
and the like, which are rules variations that are valid over a certain
sequence of SA releases.  The ideal thing would be if RDJ could determine
the target system version and refuse to allow the 3.1 admin to load the
pre-2.5 rulesets and then complain that they are all broken, as happens
frequently now.

I can see that the rules project should probably have some similar
structure: an output stream for the current release version, and N output
streams for prior versions.  These would after some period of time die off,
as people decide that supporting version x.y has become a silly waste of
time.  New and improved rules would presumably get more or less
automatically salted into the various release buckets, so that everyone
benefits to at least some extent from new rules, even if they aren't on the
latest version.

Under those happy conditions I think the rules, including plugins, should be
part of the rules project, and even should be developed as part of rules.

However, at the moment where there is a lot of movement of internals to
plugins, and changing of interfaces, it would be a bother to keep plugins
separated from code.