You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Andrew Talbot <an...@gmail.com> on 2013/04/24 18:32:05 UTC
More longer rules or fewer shorter ones?
Hey, all -
I have my customized deployment split up into a bunch of separate CF files
(by category) and I have those further split up into rules based on score.
So, I have a bunch of stuff like:
header RULE_1 Subject =~ /\b(this|that|theother|blah|blah)/i
score RULE_1 1
describe RULE_1 Rule 1
header RULE_2 Subject =~ /\b(foo|bar|etc)/i
score RULE_2 2
describe RULE_2 Rule 2
They are WAY longer than that (and some of them include further nesting of
the pipe), but that's the general idea.
My question is: is it better performance-wise to have the rules set up like
this, or to have each separate thing have its own separate rule?
RE: More longer rules or fewer shorter ones?
Posted by John Hardin <jh...@impsec.org>.
On Wed, 24 Apr 2013, Andrew Talbot wrote:
> Hi again, John -
>
> It's a good idea to add the realtime rules to the beginning of the filter. I
> didn't realize that would have such an impact.
It will have *some* impact. How much will depend on how many alternates
and how complex they are. It might also be a good idea to put the
more-complex (which is not necessarily longer) alternates at the end of
the list, in the hopes something simpler will hit first.
> And the (?=x) tip is a good one too; thank you for that.
That can be extended to two-letter combos as well if a single-letter
grouping doesn't reduce the rule sizes enough.
> As far as Bayes, don't get me started! :) I work for an Email Service
> Provider and about 2 million messages go through our servers every day, so
> we have Bayes turned off because it would be too computationally expensive.
I could see not using Bayes because a widely varied userbase would make it
hard to train effectively, but computationally expensive? Versus long,
fairly complex REs being dynamically updated in "real time"? I suspect
that's not a valid concern; but if anyone has actual stats for how much
load Bayes adds to message processing it would be interesting to see.
How often are you updating the REs and restarting spamd? RE
recompile/reparse and SA restart on a several-times-a-day schedule may be
as much as or more load than Bayes would add.
Also: don't think solely about SA - there are lighter-weight tools if your
primary focus is poison-pill text in the message subject (which all your
examples so far have been). For example, regular expression based Subject
checks in your MTA to cause SMTP-time rejects may be more efficient
overall.
> -----Original Message-----
> From: John Hardin [mailto:jhardin@impsec.org]
> Sent: Wednesday, April 24, 2013 1:53 PM
> To: users@spamassassin.apache.org
> Subject: RE: More longer rules or fewer shorter ones?
>
> On Wed, 24 Apr 2013, Andrew Talbot wrote:
>
>> John,
>>
>> Thanks for your prompt response!
>>
>> A lot of the rules are big jumbles of rules we are generating in real
>> time and adding to as things come in. Like I said in my original
>> question, we have them separated into separate cf files by category,
>> and within those cf files they are separated by score. So we have just
>> absolutely gargantuan rules for (for instance) sex words that we assign a
> 5 to automatically.
>> There's also lists of specific words and phrases that we see in
>> real-time spam (like the *$#ing garden hose spam).
>>
>> We are just tacking new rules on to the end to make them easier to
>> read. Our rules properly work with (this|that|theother) if it hits any
>> one of the words.
>>
>> Should we maybe have separate rules for all the phrases, since they're
>> longer strings? There's rules in there that are like RULE Subject =~
>> /you.have.(new|waiting|blah|blah).*(ecard|message|calendar.invite|blah
>> |blah)
>> )|(garden|new|stretchy|bendy|whatever).*(hose|vaccum|other.thing) . . .
>>
>> Etc. It goes on. .. My syntax is terrible and obviously those aren't
>> the actual rules but the point is that it's a bunch of "Or" for some
>> really long strings. Should I separate them out and have those long
>> (this|that|theother) rules be only for single words?
>
> Simple alternations on phrases are equivalent to simple alternations on
> single words with respect to the performance concerns. Performance is more
> governed by the number of alternations and the presence of repetition and
> .* than their raw length. You might want to limit the total number of
> alternations per rule.
>
> Another performance optimization would be to ensure all of the alternations
> in a given rule start with the same letter, and put (?=x) before the list of
> alternatatives e.g. /\b(?=x)(x1|x2|x3|x4)/ so that the engine can skip more
> easily.
>
> If they are simple alternations, it also depends on how you want to score
> them.
>
> For "poison pill" words or phrases, sure, a long alternation with a high
> score will be pretty efficient. I'd suggest tacking new hits onto the
> *front* of the list of alternatives, though, as it's reasonable to assume a
> spam run will use the same phrasing for a while, then change.
>
>> Alternately, should I separate out the rules with embedded pipes in
>> them (like in the example above)?
>
> Yeah, avoiding nested alternatives where possible will help.
>
> Is Bayes not catching things like this?
>
>> -----Original Message-----
>> From: John Hardin [mailto:jhardin@impsec.org]
>> Sent: Wednesday, April 24, 2013 12:58 PM
>> To: users@spamassassin.apache.org
>> Subject: Re: More longer rules or fewer shorter ones?
>>
>> On Wed, 24 Apr 2013, Andrew Talbot wrote:
>>
>>> Hey, all -
>>>
>>> I have my customized deployment split up into a bunch of separate CF
>>> files (by category) and I have those further split up into rules
>>> based on
>> score.
>>>
>>> So, I have a bunch of stuff like:
>>>
>>> header RULE_1 Subject =~ /\b(this|that|theother|blah|blah)/i
>>> score RULE_1 1
>>> describe RULE_1 Rule 1
>>>
>>> header RULE_2 Subject =~ /\b(foo|bar|etc)/i score RULE_2 2 describe
>>> RULE_2 Rule 2
>>>
>>> They are WAY longer than that (and some of them include further
>>> nesting of the pipe), but that's the general idea.
>>>
>>> My question is: is it better performance-wise to have the rules set
>>> up like this, or to have each separate thing have its own separate rule?
>>
>> For performance, with simple lists of variant values having no
>> repetition across the list e.g. (x|y|z){n,m}, if the most-likely
>> variants are listed first a "big" rule will (generally-speaking)
>> process less than a set of individual rules for each variant. The big
>> rule will stop trying as soon as a match for one variant is found,
>> whereas all of the individual rules must be tried regardless of what
>> other rules may have hit. RULE_1 won't try matching "that", "theother",
> "blah", etc. if "this" matches.
>>
>> Ignoring performance, the alternatives are *not* syntactically equivalent.
>> Absent "tflags multiple", RULE_1 would hit only once on a subject
>> containing both "this" and "that" and "theother", but if you split it
>> up into separate rules *each* would hit. This likely would affect scoring.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
It is not the business of government to make men virtuous or
religious, or to preserve the fool from the consequences of his own
folly. -- Henry George
-----------------------------------------------------------------------
328 days since the first successful private support mission to ISS (SpaceX)
Re: More longer rules or fewer shorter ones?
Posted by Axb <ax...@gmail.com>.
On 04/24/2013 08:16 PM, Andrew Talbot wrote:
> As far as Bayes, don't get me started! :) I work for an Email Service
> Provider and about 2 million messages go through our servers every day, so
> we have Bayes turned off because it would be too computationally expensive.
> I wish we could turn it on - it'd certainly make my job easier - but The
> Boss says no. Go figure. Autolearn, same story.
SA 3.4 (trunk) has a Redis backend which is amazingly fast, adds next to
zero overhead compared to some SQL or file based backend.
Since January I've been using it on 14 spamd machines querying a dual
server Redis cluster, with enabled autolearn.
One hardly notices any performance difference with or without Bayes.
You'd love it. If your boss needs any convincing/specs, contact me offlist.
RE: More longer rules or fewer shorter ones?
Posted by Andrew Talbot <an...@gmail.com>.
Hi again, John -
It's a good idea to add the realtime rules to the beginning of the filter. I
didn't realize that would have such an impact. And the (?=x) tip is a good
one too; thank you for that.
As far as Bayes, don't get me started! :) I work for an Email Service
Provider and about 2 million messages go through our servers every day, so
we have Bayes turned off because it would be too computationally expensive.
I wish we could turn it on - it'd certainly make my job easier - but The
Boss says no. Go figure. Autolearn, same story.
Having such a large organization makes it a difficult balance to avoid false
positives, too. We have one client who deals with credit reports and
refinancing and stuff and pretty much every message that goes to their
mailboxes looks like spam. We just have them set up to avoid all our
financial rules.
Luckily we don't have too many doctors' offices so we needn't really concern
ourselves with legitimate Viagra email! :)
I've scoured the net looking for rulesets from others that already have a
lot of this stuff in there but I haven't found any rulesets since 2006. A
lot of what I've seen is irrelevant - do you know a good place to get custom
rulesets? I feel like there's someone else out there who already figured out
how to write a rule that captures all those "learn a new language" spam
messages so I don't need to just score "Language" as +4 ! : )
-----Original Message-----
From: John Hardin [mailto:jhardin@impsec.org]
Sent: Wednesday, April 24, 2013 1:53 PM
To: users@spamassassin.apache.org
Subject: RE: More longer rules or fewer shorter ones?
On Wed, 24 Apr 2013, Andrew Talbot wrote:
> John,
>
> Thanks for your prompt response!
>
> A lot of the rules are big jumbles of rules we are generating in real
> time and adding to as things come in. Like I said in my original
> question, we have them separated into separate cf files by category,
> and within those cf files they are separated by score. So we have just
> absolutely gargantuan rules for (for instance) sex words that we assign a
5 to automatically.
> There's also lists of specific words and phrases that we see in
> real-time spam (like the *$#ing garden hose spam).
>
> We are just tacking new rules on to the end to make them easier to
> read. Our rules properly work with (this|that|theother) if it hits any
> one of the words.
>
> Should we maybe have separate rules for all the phrases, since they're
> longer strings? There's rules in there that are like RULE Subject =~
> /you.have.(new|waiting|blah|blah).*(ecard|message|calendar.invite|blah
> |blah)
> )|(garden|new|stretchy|bendy|whatever).*(hose|vaccum|other.thing) . . . .
.
> .
>
> Etc. It goes on. .. My syntax is terrible and obviously those aren't
> the actual rules but the point is that it's a bunch of "Or" for some
> really long strings. Should I separate them out and have those long
> (this|that|theother) rules be only for single words?
Simple alternations on phrases are equivalent to simple alternations on
single words with respect to the performance concerns. Performance is more
governed by the number of alternations and the presence of repetition and
.* than their raw length. You might want to limit the total number of
alternations per rule.
Another performance optimization would be to ensure all of the alternations
in a given rule start with the same letter, and put (?=x) before the list of
alternatatives e.g. /\b(?=x)(x1|x2|x3|x4)/ so that the engine can skip more
easily.
If they are simple alternations, it also depends on how you want to score
them.
For "poison pill" words or phrases, sure, a long alternation with a high
score will be pretty efficient. I'd suggest tacking new hits onto the
*front* of the list of alternatives, though, as it's reasonable to assume a
spam run will use the same phrasing for a while, then change.
> Alternately, should I separate out the rules with embedded pipes in
> them (like in the example above)?
Yeah, avoiding nested alternatives where possible will help.
Is Bayes not catching things like this?
> -----Original Message-----
> From: John Hardin [mailto:jhardin@impsec.org]
> Sent: Wednesday, April 24, 2013 12:58 PM
> To: users@spamassassin.apache.org
> Subject: Re: More longer rules or fewer shorter ones?
>
> On Wed, 24 Apr 2013, Andrew Talbot wrote:
>
>> Hey, all -
>>
>> I have my customized deployment split up into a bunch of separate CF
>> files (by category) and I have those further split up into rules
>> based on
> score.
>>
>> So, I have a bunch of stuff like:
>>
>> header RULE_1 Subject =~ /\b(this|that|theother|blah|blah)/i
>> score RULE_1 1
>> describe RULE_1 Rule 1
>>
>> header RULE_2 Subject =~ /\b(foo|bar|etc)/i score RULE_2 2 describe
>> RULE_2 Rule 2
>>
>> They are WAY longer than that (and some of them include further
>> nesting of the pipe), but that's the general idea.
>>
>> My question is: is it better performance-wise to have the rules set
>> up like this, or to have each separate thing have its own separate rule?
>
> For performance, with simple lists of variant values having no
> repetition across the list e.g. (x|y|z){n,m}, if the most-likely
> variants are listed first a "big" rule will (generally-speaking)
> process less than a set of individual rules for each variant. The big
> rule will stop trying as soon as a match for one variant is found,
> whereas all of the individual rules must be tried regardless of what
> other rules may have hit. RULE_1 won't try matching "that", "theother",
"blah", etc. if "this" matches.
>
> Ignoring performance, the alternatives are *not* syntactically equivalent.
> Absent "tflags multiple", RULE_1 would hit only once on a subject
> containing both "this" and "that" and "theother", but if you split it
> up into separate rules *each* would hit. This likely would affect scoring.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Maxim IX: Never turn your back on an enemy.
-----------------------------------------------------------------------
328 days since the first successful private support mission to ISS
(SpaceX)
RE: More longer rules or fewer shorter ones?
Posted by John Hardin <jh...@impsec.org>.
On Wed, 24 Apr 2013, Andrew Talbot wrote:
> John,
>
> Thanks for your prompt response!
>
> A lot of the rules are big jumbles of rules we are generating in real time
> and adding to as things come in. Like I said in my original question, we
> have them separated into separate cf files by category, and within those cf
> files they are separated by score. So we have just absolutely gargantuan
> rules for (for instance) sex words that we assign a 5 to automatically.
> There's also lists of specific words and phrases that we see in real-time
> spam (like the *$#ing garden hose spam).
>
> We are just tacking new rules on to the end to make them easier to read. Our
> rules properly work with (this|that|theother) if it hits any one of the
> words.
>
> Should we maybe have separate rules for all the phrases, since they're
> longer strings? There's rules in there that are like RULE Subject =~
> /you.have.(new|waiting|blah|blah).*(ecard|message|calendar.invite|blah|blah)
> )|(garden|new|stretchy|bendy|whatever).*(hose|vaccum|other.thing) . . . . .
> .
>
> Etc. It goes on. .. My syntax is terrible and obviously those aren't the
> actual rules but the point is that it's a bunch of "Or" for some really long
> strings. Should I separate them out and have those long (this|that|theother)
> rules be only for single words?
Simple alternations on phrases are equivalent to simple alternations on
single words with respect to the performance concerns. Performance is more
governed by the number of alternations and the presence of repetition and
.* than their raw length. You might want to limit the total number of
alternations per rule.
Another performance optimization would be to ensure all of the
alternations in a given rule start with the same letter, and put (?=x)
before the list of alternatatives e.g. /\b(?=x)(x1|x2|x3|x4)/ so that the
engine can skip more easily.
If they are simple alternations, it also depends on how you want to score
them.
For "poison pill" words or phrases, sure, a long alternation with a high
score will be pretty efficient. I'd suggest tacking new hits onto the
*front* of the list of alternatives, though, as it's reasonable to assume
a spam run will use the same phrasing for a while, then change.
> Alternately, should I separate out the rules with embedded pipes in them
> (like in the example above)?
Yeah, avoiding nested alternatives where possible will help.
Is Bayes not catching things like this?
> -----Original Message-----
> From: John Hardin [mailto:jhardin@impsec.org]
> Sent: Wednesday, April 24, 2013 12:58 PM
> To: users@spamassassin.apache.org
> Subject: Re: More longer rules or fewer shorter ones?
>
> On Wed, 24 Apr 2013, Andrew Talbot wrote:
>
>> Hey, all -
>>
>> I have my customized deployment split up into a bunch of separate CF
>> files (by category) and I have those further split up into rules based on
> score.
>>
>> So, I have a bunch of stuff like:
>>
>> header RULE_1 Subject =~ /\b(this|that|theother|blah|blah)/i
>> score RULE_1 1
>> describe RULE_1 Rule 1
>>
>> header RULE_2 Subject =~ /\b(foo|bar|etc)/i score RULE_2 2 describe
>> RULE_2 Rule 2
>>
>> They are WAY longer than that (and some of them include further
>> nesting of the pipe), but that's the general idea.
>>
>> My question is: is it better performance-wise to have the rules set up
>> like this, or to have each separate thing have its own separate rule?
>
> For performance, with simple lists of variant values having no repetition
> across the list e.g. (x|y|z){n,m}, if the most-likely variants are listed
> first a "big" rule will (generally-speaking) process less than a set of
> individual rules for each variant. The big rule will stop trying as soon as
> a match for one variant is found, whereas all of the individual rules must
> be tried regardless of what other rules may have hit. RULE_1 won't try
> matching "that", "theother", "blah", etc. if "this" matches.
>
> Ignoring performance, the alternatives are *not* syntactically equivalent.
> Absent "tflags multiple", RULE_1 would hit only once on a subject containing
> both "this" and "that" and "theother", but if you split it up into separate
> rules *each* would hit. This likely would affect scoring.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Maxim IX: Never turn your back on an enemy.
-----------------------------------------------------------------------
328 days since the first successful private support mission to ISS (SpaceX)
RE: More longer rules or fewer shorter ones?
Posted by Andrew Talbot <an...@gmail.com>.
John,
Thanks for your prompt response!
A lot of the rules are big jumbles of rules we are generating in real time
and adding to as things come in. Like I said in my original question, we
have them separated into separate cf files by category, and within those cf
files they are separated by score. So we have just absolutely gargantuan
rules for (for instance) sex words that we assign a 5 to automatically.
There's also lists of specific words and phrases that we see in real-time
spam (like the *$#ing garden hose spam).
We are just tacking new rules on to the end to make them easier to read. Our
rules properly work with (this|that|theother) if it hits any one of the
words.
Should we maybe have separate rules for all the phrases, since they're
longer strings? There's rules in there that are like RULE Subject =~
/you.have.(new|waiting|blah|blah).*(ecard|message|calendar.invite|blah|blah)
)|(garden|new|stretchy|bendy|whatever).*(hose|vaccum|other.thing) . . . . .
.
Etc. It goes on. .. My syntax is terrible and obviously those aren't the
actual rules but the point is that it's a bunch of "Or" for some really long
strings. Should I separate them out and have those long (this|that|theother)
rules be only for single words?
Alternately, should I separate out the rules with embedded pipes in them
(like in the example above)?
-----Original Message-----
From: John Hardin [mailto:jhardin@impsec.org]
Sent: Wednesday, April 24, 2013 12:58 PM
To: users@spamassassin.apache.org
Subject: Re: More longer rules or fewer shorter ones?
On Wed, 24 Apr 2013, Andrew Talbot wrote:
> Hey, all -
>
> I have my customized deployment split up into a bunch of separate CF
> files (by category) and I have those further split up into rules based on
score.
>
> So, I have a bunch of stuff like:
>
> header RULE_1 Subject =~ /\b(this|that|theother|blah|blah)/i
> score RULE_1 1
> describe RULE_1 Rule 1
>
> header RULE_2 Subject =~ /\b(foo|bar|etc)/i score RULE_2 2 describe
> RULE_2 Rule 2
>
> They are WAY longer than that (and some of them include further
> nesting of the pipe), but that's the general idea.
>
> My question is: is it better performance-wise to have the rules set up
> like this, or to have each separate thing have its own separate rule?
For performance, with simple lists of variant values having no repetition
across the list e.g. (x|y|z){n,m}, if the most-likely variants are listed
first a "big" rule will (generally-speaking) process less than a set of
individual rules for each variant. The big rule will stop trying as soon as
a match for one variant is found, whereas all of the individual rules must
be tried regardless of what other rules may have hit. RULE_1 won't try
matching "that", "theother", "blah", etc. if "this" matches.
Ignoring performance, the alternatives are *not* syntactically equivalent.
Absent "tflags multiple", RULE_1 would hit only once on a subject containing
both "this" and "that" and "theother", but if you split it up into separate
rules *each* would hit. This likely would affect scoring.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Vista "security improvements" consist of attempting to shift blame
onto the user when things go wrong.
-----------------------------------------------------------------------
328 days since the first successful private support mission to ISS
(SpaceX)
Re: More longer rules or fewer shorter ones?
Posted by John Hardin <jh...@impsec.org>.
On Wed, 24 Apr 2013, Andrew Talbot wrote:
> Hey, all -
>
> I have my customized deployment split up into a bunch of separate CF files
> (by category) and I have those further split up into rules based on score.
>
> So, I have a bunch of stuff like:
>
> header RULE_1 Subject =~ /\b(this|that|theother|blah|blah)/i
> score RULE_1 1
> describe RULE_1 Rule 1
>
> header RULE_2 Subject =~ /\b(foo|bar|etc)/i
> score RULE_2 2
> describe RULE_2 Rule 2
>
> They are WAY longer than that (and some of them include further nesting of
> the pipe), but that's the general idea.
>
> My question is: is it better performance-wise to have the rules set up like
> this, or to have each separate thing have its own separate rule?
For performance, with simple lists of variant values having no repetition
across the list e.g. (x|y|z){n,m}, if the most-likely variants are listed
first a "big" rule will (generally-speaking) process less than a set of
individual rules for each variant. The big rule will stop trying as soon
as a match for one variant is found, whereas all of the individual rules
must be tried regardless of what other rules may have hit. RULE_1 won't
try matching "that", "theother", "blah", etc. if "this" matches.
Ignoring performance, the alternatives are *not* syntactically equivalent.
Absent "tflags multiple", RULE_1 would hit only once on a subject
containing both "this" and "that" and "theother", but if you split it up
into separate rules *each* would hit. This likely would affect scoring.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Vista "security improvements" consist of attempting to shift blame
onto the user when things go wrong.
-----------------------------------------------------------------------
328 days since the first successful private support mission to ISS (SpaceX)
Re: More longer rules or fewer shorter ones?
Posted by John Hardin <jh...@impsec.org>.
On Thu, 25 Apr 2013, Andrew Talbot wrote:
> I never thought about scoring each rule as a 0.001 or something really low
> then tying them all together with meta-rules.
Best practice is to define them as __SUBRULES, which don't score at all,
and then tie those together in metas.
The only reason to make a scoring rule with an intentional score of 0.001
is if you want it to appear in the list of rules hit that's added to the
message headers. __SUBRULES don't appear in that list.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Trusting in anti-gun laws to keep you from being shot is like
refusing to wear your seatbelt because you trust traffic laws to
keep you from being in a car accident. -- Erin Palette
-----------------------------------------------------------------------
329 days since the first successful private support mission to ISS (SpaceX)
Re: More longer rules or fewer shorter ones?
Posted by Martin Gregorie <ma...@gregorie.org>.
On Fri, 2013-04-26 at 17:44 -0700, John Hardin wrote:
> On Sat, 27 Apr 2013, Martin Gregorie wrote:
>
> > Question to JH: I can see that portmanteau rules on high volume sites
> > would benefit from the (?=x) optimisation, but so would a lot of rules
> > that use regexes containing alternations. So, is there any possibility
> > of slotting it into the SA rule compiler rather than me implementing it
> > as part of my portmanteau rule generator when doing the latter would
> > limit its use to a subset of rules that might benefit.
>
> The internal compiler within SA itself, or the external tool that compiles
> rules?
>
> The former is just essentially generation of perl code for the rule
> evaluation and reusing that. It doesn't do any analysis or rule-combining
> that would allow leverage of this.
>
> I have no familiarity with the latter, so I couldn't comment.
>
> It seems to me that only-the-fly generation of (?=x)(xa|xb|xc) or x(a|b|c)
> optimizations of alterations is a fairly non-trivial task given something
> other than alternations built from a sorted set of substrings or sub-REs -
> and in fact, now that I think about it, that sort of optimization is
> properly the job of an optimizer for the RE parser built into perl... the
> programmer writing the REs shouldn't have to worry about doing something
> (fairly) obvious like that. Think of it the same way you think about the
> query plan optimizer in a database engine.
>
'swat I thought. Thanks for the confirmation.
As you say, implementing a general solution that first decides whether
its a worthwhile optimisation for a given alternates list and then, if
worthwhile, applies it is non-trivial.
Martin
Martin
Re: More longer rules or fewer shorter ones?
Posted by John Hardin <jh...@impsec.org>.
On Sat, 27 Apr 2013, Martin Gregorie wrote:
> Question to JH: I can see that portmanteau rules on high volume sites
> would benefit from the (?=x) optimisation, but so would a lot of rules
> that use regexes containing alternations. So, is there any possibility
> of slotting it into the SA rule compiler rather than me implementing it
> as part of my portmanteau rule generator when doing the latter would
> limit its use to a subset of rules that might benefit.
The internal compiler within SA itself, or the external tool that compiles
rules?
The former is just essentially generation of perl code for the rule
evaluation and reusing that. It doesn't do any analysis or rule-combining
that would allow leverage of this.
I have no familiarity with the latter, so I couldn't comment.
It seems to me that only-the-fly generation of (?=x)(xa|xb|xc) or x(a|b|c)
optimizations of alterations is a fairly non-trivial task given something
other than alternations built from a sorted set of substrings or sub-REs -
and in fact, now that I think about it, that sort of optimization is
properly the job of an optimizer for the RE parser built into perl... the
programmer writing the REs shouldn't have to worry about doing something
(fairly) obvious like that. Think of it the same way you think about the
query plan optimizer in a database engine.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
...we talk about creating "millions of shovel-ready jobs" for a
society that doesn't really encourage people to pick up a shovel.
-- Mike Rowe, testifying before Congress
-----------------------------------------------------------------------
330 days since the first successful private support mission to ISS (SpaceX)
Re: More longer rules or fewer shorter ones?
Posted by Martin Gregorie <ma...@gregorie.org>.
On Fri, 2013-04-26 at 16:04 -0400, Andrew Talbot wrote:
> Martin -
>
> Interesting. How many mailboxes does your deployment cover?
>
Effectively one.
I said I was a small setup: I run a personal system, organised as a
house server, which accepts all incoming mail and only runs SA against
the incoming stream, with some other boxes on the LAN. These send
outbound mail via the house server.
The central MTA (Postfix on the house server) sends all messages it
receives to a single mailbox, which Dovecot makes available to other
hosts on the LAN, and also BCCs them to a mail archive. This, a Postgres
DB, also acts as as an automatic whitelister: SA runs a query against it
that whitelists any address I've sent mail to.
I realise my setup is almost entirely unlike yours, but there seem to be
some common requirements. However, as always, the devil is in the detail
and my implementation may well fail at your volumes.
Question to JH: I can see that portmanteau rules on high volume sites
would benefit from the (?=x) optimisation, but so would a lot of rules
that use regexes containing alternations. So, is there any possibility
of slotting it into the SA rule compiler rather than me implementing it
as part of my portmanteau rule generator when doing the latter would
limit its use to a subset of rules that might benefit.
Martin
RE: More longer rules or fewer shorter ones?
Posted by Andrew Talbot <an...@gmail.com>.
Martin -
Interesting. How many mailboxes does your deployment cover?
-----Original Message-----
From: Martin Gregorie [mailto:martin@gregorie.org]
Sent: Thursday, April 25, 2013 8:08 PM
To: users@spamassassin.apache.org
Subject: Re: More longer rules or fewer shorter ones?
On Thu, 2013-04-25 at 18:45 -0400, Andrew Talbot wrote:
> I like your point about the portmanteau rules (and I award you two
> Points for using one of my favorite words in a new - yet appropriate -
> manner!).
>
:-)
> I never thought about scoring each rule as a 0.001 or something really
> low then tying them all together with meta-rules. It's been a while
> since I separated everything out but I believe I have around 1000
> different checks (most of them portmanteau'd) so it seems like those
> meta rules would just get ... Messy. But it's a good idea, and I think
> I can especially make use of it in my Specific Word list.
>
The metas aren't too bad, though I must admit to building some of them as metas of metas to keep all lines down to 72 chars or so. Most of these submetas are simply lists of other rules that have been ANDed or ORed together.
You may find that the Portmanteau Generator reduces your rule count because it too can generate metas, which I use to deal with situations where a term can appear in more than one case, e.g. a generated rule can have this form:
describe GENRULE Example rule
header __GR1 Reply-to =~ /(\@spam1\.com|\@spammer\.co\.uk|....)
header __GR2 From =~ /(\@spam1\.com|\@spammer\.co\.uk|....)
uri __GR3 From =~ /(\@spam1\.com|\@spammer\.co\.uk|....)
meta GENRULE (__P1 || __P2 || __P3)
score GENRULE 1.5
which has two advantages. First, that GENRULE is a single name that covers the same spammy term regardless of where it was used and secondly, since each generated rule has its own source file, this makes the three related lists easier to edit, since there's a good chance that a spammy term might be used in more than one of the related lists.
> Keeping the rules under 1-2mb is a good rule of thumb to follow.
> Luckily we're nowhere near that point yet.
>
Nor am I. As I said, my biggest generated rule is a bit over 9 KB.
> Can I ask how many rules you have, and how many of those are meta
> rules?
>
I have 31 portmanteau rules, of which 9 contain metas. Only 12 of these have a score exceeding 1.0 and these are not usually used as part of higher level metarules
My local.cf is where any very specific rules live, along with the higher level metarules that use the low scoring portmanteau rules. This contains 129 rules which between them contain 96 'meta' statements. 36 of these have scores of under 1.0, so are probably used as components of metarules. The total number of rules was obtained by using grep+wc to count lines containing '^score'.
my local.cf and portmanteau.cf files are both 29 KB in size.
Martin
Re: More longer rules or fewer shorter ones?
Posted by Martin Gregorie <ma...@gregorie.org>.
On Thu, 2013-04-25 at 18:45 -0400, Andrew Talbot wrote:
> I like your point about the portmanteau rules (and I award you two
> Points for using one of my favorite words in a new - yet appropriate -
> manner!).
>
:-)
> I never thought about scoring each rule as a 0.001 or something really
> low then tying them all together with meta-rules. It's been a while
> since I separated everything out but I believe I have around 1000
> different checks (most of them portmanteau'd) so it seems like those
> meta rules would just get ... Messy. But it's a good idea, and I think
> I can especially make use of it in my Specific Word list.
>
The metas aren't too bad, though I must admit to building some of them
as metas of metas to keep all lines down to 72 chars or so. Most of
these submetas are simply lists of other rules that have been ANDed or
ORed together.
You may find that the Portmanteau Generator reduces your rule count
because it too can generate metas, which I use to deal with situations
where a term can appear in more than one case, e.g. a generated rule can
have this form:
describe GENRULE Example rule
header __GR1 Reply-to =~ /(\@spam1\.com|\@spammer\.co\.uk|....)
header __GR2 From =~ /(\@spam1\.com|\@spammer\.co\.uk|....)
uri __GR3 From =~ /(\@spam1\.com|\@spammer\.co\.uk|....)
meta GENRULE (__P1 || __P2 || __P3)
score GENRULE 1.5
which has two advantages. First, that GENRULE is a single name that
covers the same spammy term regardless of where it was used and
secondly, since each generated rule has its own source file, this makes
the three related lists easier to edit, since there's a good chance that
a spammy term might be used in more than one of the related lists.
> Keeping the rules under 1-2mb is a good rule of thumb to follow.
> Luckily we're nowhere near that point yet.
>
Nor am I. As I said, my biggest generated rule is a bit over 9 KB.
> Can I ask how many rules you have, and how many of those are meta
> rules?
>
I have 31 portmanteau rules, of which 9 contain metas. Only 12 of these
have a score exceeding 1.0 and these are not usually used as part of
higher level metarules
My local.cf is where any very specific rules live, along with the higher
level metarules that use the low scoring portmanteau rules. This
contains 129 rules which between them contain 96 'meta' statements. 36
of these have scores of under 1.0, so are probably used as components of
metarules. The total number of rules was obtained by using grep+wc to
count lines containing '^score'.
my local.cf and portmanteau.cf files are both 29 KB in size.
Martin
Re: More longer rules or fewer shorter ones?
Posted by Andrew Talbot <an...@gmail.com>.
Hi, Martin -
Thank you for your response.
I like your point about the portmanteau rules (and I award you two Points
for using one of my favorite words in a new - yet appropriate - manner!).
I never thought about scoring each rule as a 0.001 or something really low
then tying them all together with meta-rules. It's been a while since I
separated everything out but I believe I have around 1000 different checks
(most of them portmanteau'd) so it seems like those meta rules would just
get ... Messy. But it's a good idea, and I think I can especially make use
of it in my Specific Word list.
It's interesting that you don't use Bayes for the opposite reason that we
don't - we don't do it because of high volume, you don't do it because of
low volume. Go figure.
Keeping the rules under 1-2mb is a good rule of thumb to follow. Luckily
we're nowhere near that point yet.
Can I ask how many rules you have, and how many of those are meta rules?
-----Original Message-----
From: Martin Gregorie [mailto:martin@gregorie.org]
Sent: Wednesday, April 24, 2013 3:03 PM
To: users@spamassassin.apache.org
Subject: Re: More longer rules or fewer shorter ones?
On Wed, 2013-04-24 at 12:32 -0400, Andrew Talbot wrote:
> I have my customized deployment split up into a bunch of separate CF
> files (by category) and I have those further split up into rules based
> on score.
>
I also use very long rules, mainly due to spamiferous mailing lists,
because all the headers are pretty much the same (apart from sender names),
so about all you're left with for spam recognition is the body content.
I found a problem with very long rules, where for me 'very long' means
"rules longer than the width of my editor's screen". I refer to these as
'portmanteau rules' (private slang). As I hate editing anything that's
longer than my editor's text line and find it particularly annoying to deal
with such a line containing a regex consisting of a lot of alternates, I
wrote a portmanteau rule generator to make their maintenance a bit easier.
It is a gawk script that assembles an arbitrarily long rule from a file
containing rule fragments (regexes,
etc) that are each placed on a separate line. Since sounds as though you
may have a similar problem, you may also find it useful. You can find it
and its documentation here:
http://www.libelle-systems.com/free/portmanteau/portmanteau.tgz
I find it particularly helpful to make the portmanteau rules fairly low
scoring and to combine them into higher scoring meta-rules, e.g. if I'm
trapping sales spiel I'll have a portmanteau rule listing selling phrases,
one containing monetary terms and another containing product terms and
names, all scores at 0.001. I'll also have a meta-rule that ANDs these
three rules together and scores around 5. This approach is much better at
distinguishing spam from ham than a series of higher scoring non-meta rules
and has the additional benefit of recognising sales-related text from
previously unseen combinations of elements in the three rules.
BTW, I don't use Bayes because my mail volume is small and I have
difficulty collecting decent training corpuses and find my current setup
easier to manage.
They are WAY longer than that (and some of them include further nesting
of the pipe), but that's the general idea.
> My question is: is it better performance-wise to have the rules set up
> like this, or to have each separate thing have its own separate rule?
>
What JH said. When I was thinking of setting up this approach I asked about
performance and limits on the size of the generated rules and was told that
I shouldn't worry about rule size until they exceeded a megabyte or two.
Currently my longest rule is just over 9KB, with the averages being just
under 1KB and 51 alternates per rule.
Martin
Re: More longer rules or fewer shorter ones?
Posted by Martin Gregorie <ma...@gregorie.org>.
On Wed, 2013-04-24 at 12:32 -0400, Andrew Talbot wrote:
> I have my customized deployment split up into a bunch of separate CF
> files (by category) and I have those further split up into rules based
> on score.
>
I also use very long rules, mainly due to spamiferous mailing lists,
because all the headers are pretty much the same (apart from sender
names), so about all you're left with for spam recognition is the body
content.
I found a problem with very long rules, where for me 'very long' means
"rules longer than the width of my editor's screen". I refer to these as
'portmanteau rules' (private slang). As I hate editing anything that's
longer than my editor's text line and find it particularly annoying to
deal with such a line containing a regex consisting of a lot of
alternates, I wrote a portmanteau rule generator to make their
maintenance a bit easier. It is a gawk script that assembles an
arbitrarily long rule from a file containing rule fragments (regexes,
etc) that are each placed on a separate line. Since sounds as though you
may have a similar problem, you may also find it useful. You can find it
and its documentation here:
http://www.libelle-systems.com/free/portmanteau/portmanteau.tgz
I find it particularly helpful to make the portmanteau rules fairly low
scoring and to combine them into higher scoring meta-rules, e.g. if I'm
trapping sales spiel I'll have a portmanteau rule listing selling
phrases, one containing monetary terms and another containing product
terms and names, all scores at 0.001. I'll also have a meta-rule that
ANDs these three rules together and scores around 5. This approach is
much better at distinguishing spam from ham than a series of higher
scoring non-meta rules and has the additional benefit of recognising
sales-related text from previously unseen combinations of elements in
the three rules.
BTW, I don't use Bayes because my mail volume is small and I have
difficulty collecting decent training corpuses and find my current setup
easier to manage.
They are WAY longer than that (and some of them include further
nesting of the pipe), but that's the general idea.
> My question is: is it better performance-wise to have the rules set up
> like this, or to have each separate thing have its own separate rule?
>
What JH said. When I was thinking of setting up this approach I asked
about performance and limits on the size of the generated rules and was
told that I shouldn't worry about rule size until they exceeded a
megabyte or two. Currently my longest rule is just over 9KB, with the
averages being just under 1KB and 51 alternates per rule.
Martin