RFC: move "full" rule functionality into a default-off plugin

[Theo Van Dinter <fe...@apache.org>]

Since we have no full rules, and in fact we actively discourage people
from writing/using full rules, I'd like to see the functionality get
moved into a plugin which is off by default.

Thoughts?

-- 
Randomly Generated Tagline:
What about WRITING it first and rationalizing it afterwords?  :-)
              -- Larry Wall in <81...@jpl-devvax.JPL.NASA.GOV>

Re: RFC: move "full" rule functionality into a default-off plugin

[John Myers <jg...@proofpoint.com>]

Theo Van Dinter wrote:
> I have no
> evidence for this, but if full rules are helping cause problems such as
> high memory use and large scan times (and it's certainly possible because
> we all know how inefficient full rules are and have been -- that's why we
> stopped using them in the first place), then shouldn't we try to help by
> disabling the problematic rules by default?
No, I believe disabling the problematic rule type is too indirect an 
approach.  We should try to help by measuring scan times and disabling 
those tests whose contribution to the scan times exceeds their value.

Re: RFC: move "full" rule functionality into a default-off plugin

[Theo Van Dinter <fe...@apache.org>]

On Tue, Mar 07, 2006 at 08:49:56PM +0000, Justin Mason wrote:
> *We* may not have full rules in the core ruleset, but I'm pretty sure SARE
> and other third parties have a few. SpamAssassin isn't just a bundled set
> of rules -- it's a platform, too ;)   Changing the platform -- in
> a backwards incompatible way -- is a bad thing to do imo.

I would generally agree, except for this type of situation.  I have no
evidence for this, but if full rules are helping cause problems such as
high memory use and large scan times (and it's certainly possible because
we all know how inefficient full rules are and have been -- that's why we
stopped using them in the first place), then shouldn't we try to help by
disabling the problematic rules by default?  If the user understands the
risks and decides to use full rules anyway, they could easily enable them.

On the flip side, if people are downloading rules from third parties, they'll
already be making a conscious decision to use those full rules, so maybe this
doesn't actually help.

<shrug>  If people generally don't think it's a good idea, that's fine with
me.  It was just a thought. :)

-- 
Randomly Generated Tagline:
 nappster.com: Download any celebrity from A.A. Milne to Z.Z. Top

Re: RFC: move "full" rule functionality into a default-off plugin

[Theo Van Dinter <fe...@apache.org>]

On Tue, Mar 07, 2006 at 10:19:07AM -0800, John Myers wrote:
> I'm not so big on turning off the functionality.  We shouldn't erect 
> barriers against our being able to later publish full rules through 
> sa-update.

My POV is that we will never publish full rules again (so far, everything
we would have previously (circa v2.[234]) used a full rule for is better
handled by code that targets what we want,) and we actively discourage
others from writing and using full rules.  Therefore I think we should
definitely erect barriers against being able to publish full rules!  :)

I don't want to get rid of them altogether, but if someone wants to use
them I think they need to actively think about what the issues are and
then do something to allow use if they decide full rules are worth it.
It's similar to allow_user_rules -- they generally work fine, but
there's a large potential for pain if used the wrong way, or if there's
an accident, or ...

-- 
Randomly Generated Tagline:
"Have we had the chance to play with a blow torch Nikki?" - Mr. Wizard

Re: RFC: move "full" rule functionality into a default-off plugin

[John Myers <jg...@proofpoint.com>]

Theo Van Dinter wrote:
> Since we have no full rules, and in fact we actively discourage people
> from writing/using full rules, I'd like to see the functionality get
> moved into a plugin which is off by default.
>   
Moving the functionality into a plugin is part of bug 4778.  The status 
of that is that Michael Parker are trying to schedule a meeting where we 
can hopefully work out a combined proposal for that and bug 4776.

I'm not so big on turning off the functionality.  We shouldn't erect 
barriers against our being able to later publish full rules through 
sa-update.

Re: move "full" rule functionality into a default-off plugin

[Theo Van Dinter <fe...@apache.org>]

On Wed, Mar 08, 2006 at 01:09:10AM -0800, Loren Wilton wrote:
> Let me suggest that this is a *REALLY* *BAD* idea.
> No, make that "an *EXCEPTIONALLY* *AWFUL* idea".

Really?  If it weren't a rule type -- a section of code that is not
being used, and who use has been actively discouraged for years ...
it'd be deleted without having to think about it.

> what a one-line rule should be able to do.  Probably a third of the SARE
> rules are 'full' rules as a result of rawbody being deliberately useless.

You found us out!  We write deliberately useless code!

> If you remove full rules you will have effectively eliminated any chance of
> SA catching ANY of the modern forms of spam, all of which are currently only
> detectable with full rules, or with Dallas's rule forms that won't get
> approved.

Really?  I have no full rules and few rawbody rules (the default) and
catch literally all of the spam I receive.

> If 'full' rules are disabled, SA is effectively disabled with the exception
> of Bayes and URIBL.  There would probably be a much lower-overhead solution,
> say SpamBayes, if SA's rules capability is effectively removed.  Which seems
> to be the effective intent of this proposal.

Please.  If you don't like a proposal, you can say so simply without having to
spew diatribe all over the place.  Claiming the world is going to end if
something changed is not necessarily the best way to get people to agree with
you.

-- 
Randomly Generated Tagline:
"I do not like Santiago.  I've always thought that a leader should have a
 strong chin.  He has no chin, and his Vice-President has several.  This to me
 is not a good combination."
         - Ivonova, Babylon 5 (Midnight on the Firing Line)

Re: move "full" rule functionality into a default-off plugin

[Loren Wilton <lw...@earthlink.net>]

Let me suggest that this is a *REALLY* *BAD* idea.
No, make that "an *EXCEPTIONALLY* *AWFUL* idea".

Rawbody rules are useless  for 80% of the things they should detect because
they only handle one line at a time, and spammers 5 years ago leanred they
can break HTML over two lines and disable all useful HTML checks that aren't
evals.

Dallas's stuff that would get around this is hung up in review because
Michael either dislikes the idea of useful rules, or insists they have to be
plugins, or noticed that it fixes the rawbody rule problem and makes them
useful, so is against it.

The ONLY, I repeat **ONLY** alternative to the useless rawbody rule if the
'full' rule, or else going in and writing 100 lines of unreadable Perl to do
what a one-line rule should be able to do.  Probably a third of the SARE
rules are 'full' rules as a result of rawbody being deliberately useless.
If you remove full rules you will have effectively eliminated any chance of
SA catching ANY of the modern forms of spam, all of which are currently only
detectable with full rules, or with Dallas's rule forms that won't get
approved.

If 'full' rules are disabled, SA is effectively disabled with the exception
of Bayes and URIBL.  There would probably be a much lower-overhead solution,
say SpamBayes, if SA's rules capability is effectively removed.  Which seems
to be the effective intent of this proposal.

        Loren Wilton