You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Chris Pollock <cp...@embarqmail.com> on 2018/12/09 17:04:53 UTC
Subtest __E_LIKE_LETTER and __LOWER_E listed many times in message
header
This is probably very trivial and doesn't affect anything except maybe
the size of the headers but I have to ask. When looking at the headers
of some ham I noticed - https://pastebin.com/H7euxqVX the two rules I
mention above are in 72_active.cf. Is there a reason for the number of
times it's listed? Couldn't each subtest be listed just once instead of
multiple times?
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
10:58:20 up 1 day, 15:22, 1 user, load average: 0.67, 0.49, 0.31
Description: Ubuntu 18.04.1 LTS, kernel 4.15.0-42-generic
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Bill Cole <bi...@apache.org>.
On 10 Dec 2018, at 1:46, Henrik Krohns wrote:
> On Sun, Dec 09, 2018 at 01:06:01PM -0500, Bill Cole wrote:
>>
>> To make this determination, the rules require the 'multiple' flag
>> without
>> a cap on thne number of matches which a 'maxhits' parameter would
>> set.
>
> Please don't do unlimited maxhits, it's terrible if message accidently
> or
> intentionally contains thousands of e's. The eval code runs all sorts
> of
> crap for every hit, not to mention the mass of debug lines it
> potentially
> creates.
I recognize this as an issue, and I'm trying to think up alternative
approaches. The ruleqa performance of this rule is puzzling.
> If I read right, isn't it enough to set __LOWER_E maxhits=21 and
> __E_LIKE_LETTER maxhits=211 for the clause to evaluate as true?
That would break the *correct* logic, which I just noticed was mangled
by a typo in the revision I made yesterday to evade the 'possible divide
by zero' mis-parse.
The goal is to identify messages where the ratio of all e-like
characters (__E_LIKE_LETTER ) to simple Latin 'e' characters (__LOWER_E)
is between 1.4 and 10. My reasoning for a range of ratios is that
messages of any significant size will use one script predominantly, but
perhaps not exclusively.
Consider a message with 200 U+0065 characters and 220 U+0435 characters:
__LOWER_E = 200, __E_LIKE_LETTER = 420. The ratio is 2.1, so this is a
message which would match the intended logic. However, with your
proposed maxhits limits: __LOWER_E = 21, __E_LIKE_LETTER = 211 so the
ratio is 10.05, no match.
Also consider a message with 200 U+0065 characters and 9 U+0435
characters: __LOWER_E = 200, __E_LIKE_LETTER = 209. The ratio is 1.045,
so this is a message which would NOT match the intended logic. However,
with your proposed maxhits limits: __LOWER_E = 21, __E_LIKE_LETTER =
209 so the ratio is 9.95, a match.
Finding a fine-tuned pair of maxhits values is hard, particularly since
I don't have a good corpus of the target spam or of ham that
*apparently* (according to ruleqa stats) is being matched by the current
rule in some corpora. I've set maxhits at 250 and 400 for now on the
principle that the spam I'm really targeting has less than half of
those.
>
> body __LOWER_E /e/i
> tflags __LOWER_E multiple
> replace_rules __E_LIKE_LETTER
> body __E_LIKE_LETTER /<E>/
> tflags __E_LIKE_LETTER multiple
> meta MIXED_ES ( __LOWER_E > 20 ) && (
> __E_LIKE_LETTER > ( (__LOWER_E * 14 ) / 10) ) && ( __E_LIKE_LETTER > (
> 10 * __LOWER_E ) )
This is now fixed:
meta MIXED_ES ( __LOWER_E > 20 ) && ( __E_LIKE_LETTER > ( (__LOWER_E *
14 ) / 10) ) && ( __E_LIKE_LETTER < ( 10 * __LOWER_E ) )
--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Available For Hire: https://linkedin.com/in/billcole
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Bill Cole <bi...@apache.org>.
On 10 Dec 2018, at 1:56, Henrik K wrote:
> On Mon, Dec 10, 2018 at 08:46:32AM +0200, Henrik Krohns wrote:
>> On Sun, Dec 09, 2018 at 01:06:01PM -0500, Bill Cole wrote:
>>>
>>> To make this determination, the rules require the 'multiple' flag
>>> without
>>> a cap on thne number of matches which a 'maxhits' parameter would
>>> set.
>>
>> Please don't do unlimited maxhits, it's terrible if message
>> accidently or
>> intentionally contains thousands of e's. The eval code runs all
>> sorts of
>> crap for every hit, not to mention the mass of debug lines it
>> potentially
>> creates.
>>
>> If I read right, isn't it enough to set __LOWER_E maxhits=21 and
>> __E_LIKE_LETTER maxhits=211 for the clause to evaluate as true?
>>
>> body __LOWER_E /e/i
>> tflags __LOWER_E multiple
>> replace_rules __E_LIKE_LETTER
>> body __E_LIKE_LETTER /<E>/
>> tflags __E_LIKE_LETTER multiple
>> meta MIXED_ES ( __LOWER_E > 20 ) && (
>> __E_LIKE_LETTER > ( (__LOWER_E * 14 ) / 10) ) && ( __E_LIKE_LETTER >
>> ( 10 * __LOWER_E ) )
>> describe MIXED_ES Too many es are not es
>
> Also consider limiting __HAS_IMG_SRC, __HAS_HREF,
> __HAS_IMG_SRC_ONECASE,
> __HAS_HREF_ONECASE
Done.
> I would use non-greedy .*? in all those also
>
> /^[^>].*<img src=/i
Done.
Thanks for the input!
--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Available For Hire: https://linkedin.com/in/billcole
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Henrik K <he...@hege.li>.
On Mon, Dec 10, 2018 at 08:46:32AM +0200, Henrik Krohns wrote:
> On Sun, Dec 09, 2018 at 01:06:01PM -0500, Bill Cole wrote:
> >
> > To make this determination, the rules require the 'multiple' flag without
> > a cap on thne number of matches which a 'maxhits' parameter would set.
>
> Please don't do unlimited maxhits, it's terrible if message accidently or
> intentionally contains thousands of e's. The eval code runs all sorts of
> crap for every hit, not to mention the mass of debug lines it potentially
> creates.
>
> If I read right, isn't it enough to set __LOWER_E maxhits=21 and
> __E_LIKE_LETTER maxhits=211 for the clause to evaluate as true?
>
> body __LOWER_E /e/i
> tflags __LOWER_E multiple
> replace_rules __E_LIKE_LETTER
> body __E_LIKE_LETTER /<E>/
> tflags __E_LIKE_LETTER multiple
> meta MIXED_ES ( __LOWER_E > 20 ) && ( __E_LIKE_LETTER > ( (__LOWER_E * 14 ) / 10) ) && ( __E_LIKE_LETTER > ( 10 * __LOWER_E ) )
> describe MIXED_ES Too many es are not es
Also consider limiting __HAS_IMG_SRC, __HAS_HREF, __HAS_IMG_SRC_ONECASE,
__HAS_HREF_ONECASE
I would use non-greedy .*? in all those also
/^[^>].*<img src=/i
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Henrik Krohns <he...@hege.li>.
On Sun, Dec 09, 2018 at 01:06:01PM -0500, Bill Cole wrote:
>
> To make this determination, the rules require the 'multiple' flag without
> a cap on thne number of matches which a 'maxhits' parameter would set.
Please don't do unlimited maxhits, it's terrible if message accidently or
intentionally contains thousands of e's. The eval code runs all sorts of
crap for every hit, not to mention the mass of debug lines it potentially
creates.
If I read right, isn't it enough to set __LOWER_E maxhits=21 and
__E_LIKE_LETTER maxhits=211 for the clause to evaluate as true?
body __LOWER_E /e/i
tflags __LOWER_E multiple
replace_rules __E_LIKE_LETTER
body __E_LIKE_LETTER /<E>/
tflags __E_LIKE_LETTER multiple
meta MIXED_ES ( __LOWER_E > 20 ) && ( __E_LIKE_LETTER > ( (__LOWER_E * 14 ) / 10) ) && ( __E_LIKE_LETTER > ( 10 * __LOWER_E ) )
describe MIXED_ES Too many es are not es
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by John Hardin <jh...@impsec.org>.
On Tue, 11 Dec 2018, Chris Pollock wrote:
>> It's got the potential to be VERY noisy (as you've discovered) while
>> not really providing much useful info. Not a big deal on a small
>> system.
>
> I could just go through and comment out this line in my local.cf
> add_header all Subtest Ran _SUBTESTS(,)_
> but I periodically like to see what's going on.
It would be reasonable to file a bug that _SUBTESTS reports duplicates. I
would characterize that as a misbehavior.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
...a great many people are not fit for Liberty, it scares the crap
out of them and they'd much rather be ruled. As Loki said in the
Avengers movie, kneeling is their natural state. -- Mark D @ TSM
-----------------------------------------------------------------------
4 days until Bill of Rights day
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Chris Pollock <cp...@embarqmail.com>.
On Mon, 2018-12-10 at 13:09 -0500, Bill Cole wrote:
> On 9 Dec 2018, at 18:23, Chris Pollock wrote:
>
> > On Sun, 2018-12-09 at 13:06 -0500, Bill Cole wrote:
> > > On 9 Dec 2018, at 12:04, Chris Pollock wrote:
> > >
> > > > This is probably very trivial and doesn't affect anything
> > > > except
> > > > maybe
> > > > the size of the headers but I have to ask. When looking at the
> > > > headers
> > > > of some ham I noticed - https://pastebin.com/H7euxqVX the two
> > > > rules
> > > > I
> > > > mention above are in 72_active.cf. Is there a reason for the
> > > > number
> > > > of
> > > > times it's listed? Couldn't each subtest be listed just once
> > > > instead
> > > > of
> > > > multiple times?
> > >
> > > Not with the current documented behavior of the code, given the
> > > way
> > > those sub-rules are designed to work together. The goal is to
> > > identify
> > > messages which use Latin-script 'e' characters but also use many
> > > non-Latin-script characters which look like 'e' but are not. To
> > > make
> > > this determination, the rules require the 'multiple' flag without
> > > a
> > > cap
> > > on thne number of matches which a 'maxhits' parameter would set.
> >
> > Got it, thanks Bill. I've never noticed this before. I also noticed
> > that according to my daily sa-update output this subtest is
> > apparently
> > new or at least it didn't appear in the output until this past Fri.
>
> Correct. See the thread with the subject "No longer just embedded =9D
> characters in blackmail emails" here last week for the background.
>
> > >
> > > It is not recommended to routinely add the list of matched sub-
> > > rules
> > > to
> > > scanned messages.
> > >
> >
> > Any specific reason why? This is just on my home system.
>
> It's got the potential to be VERY noisy (as you've discovered) while
> not really providing much useful info. Not a big deal on a small
> system.
>
I could just go through and comment out this line in my local.cf
add_header all Subtest Ran _SUBTESTS(,)_
but I periodically like to see what's going on.
>
> Anyway, as of today I've capped those 2 subrules at levels which
> leave ample space to still match the target spam. Should show up in
> tomorrow's update.
Thanks Bill, I'll see how it looks after today's update.
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
08:51:45 up 3 days, 13:15, 1 user, load average: 0.27, 0.59, 0.64
Description: Ubuntu 18.04.1 LTS, kernel 4.15.0-42-generic
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Chris Pollock <cp...@embarqmail.com>.
On Thu, 2018-12-13 at 17:08 -0500, Bill Cole wrote:
> On 13 Dec 2018, at 16:24, Chris Pollock wrote:
>
> > On Thu, 2018-12-13 at 15:14 -0600, Chris Pollock wrote:
> > > On Tue, 2018-12-11 at 19:00 -0500, Bill Cole wrote:
> > > > On 11 Dec 2018, at 16:37, Chris Pollock wrote:
> > > >
> > > > > On Mon, 2018-12-10 at 13:09 -0500, Bill Cole wrote:
> > > >
> > > > [...]
> > > > > > Anyway, as of today I've capped those 2 subrules at levels
> > > > > > which
> > > > > > leave ample space to still match the target spam. Should
> > > > > > show
> > > > > > up
> > > > > > in
> > > > > > tomorrow's update.
> > > >
> > > > I was wrong. The addition of a 'maxhits' parameter to the two
> > > > subrules apparently didn't get committed in time for the
> > > > nightly
> > > > rule
> > > > promotion run. It was in r1848602 and the current ruleset is
> > > > still
> > > > at
> > > > r1848555. Assuming all goes well tonight, the change will
> > > > appear
> > > > tomorrow.
> > > >
> > >
> > > Shouldn't this have stopped by now -
> > > https://pastebin.com/7260daT3
> > > Today's update was '1848731'.
> > >
> >
> > Hit send too fast. Doing a compare between 72_active.cf dated the
> > 11th
> > and the one dated today I do see:
> >
> > Dated 11 Dec
> > if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> > ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
> > body __E_LIKE_LETTER /<E>/
> > tflags __E_LIKE_LETTER multiple
> >
> > if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> > ifplugin
> > Mail::SpamAssassin::Plugin::ReplaceTags
> > body __LOWER_E
> > /e/i
> > tflags __LOWER_E multiple
> >
> > Dated today 13 Dec
> > if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> > ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
> > body __E_LIKE_LETTER /<E>/
> > tflags __E_LIKE_LETTER multiple maxhits=400
> >
> > if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> > ifplugin
> > Mail::SpamAssassin::Plugin::ReplaceTags
> > body __LOWER_E
> > /e/
> > tflags __LOWER_E multiple maxhits=250
> >
> > IIUC then __E_LIKE_LETTER can hit a max of 400 times in one message
> > and
> > __LOWER_E a max of 250 times in one message.
>
> For now, yes. Those numbers were the result of a mis-think on my part
> and will be 320 and 230 once the current rev works its way through.
>
> > Therefore I may still have
> > a large listing of subtest ran.
>
> Yes.
> I don't expect that behavior to change. SA has always tallied rules
> and sub-rules with multiple matches and the 'multiple' tflag this way
> and I see no compelling reason to change that. It almost certainly
> will not change for 3.4.3, which should be the last 3.4.x release.
> If there's a bug opened and someone is willing to work on code for
> whatever changes need to be made to collapse duplicate hit names in
> the lists of rule matches into a single citation with a count of
> hits, I expect that change would be accepted for v4, even though it
> may impact existing users' tooling.
>
Thanks very much Bill, appreciate the explanation.
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
16:47:20 up 5 days, 21:11, 1 user, load average: 0.91, 0.77, 0.67
Description: Ubuntu 18.04.1 LTS, kernel 4.15.0-42-generic
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 13 Dec 2018, at 16:24, Chris Pollock wrote:
> On Thu, 2018-12-13 at 15:14 -0600, Chris Pollock wrote:
>> On Tue, 2018-12-11 at 19:00 -0500, Bill Cole wrote:
>>> On 11 Dec 2018, at 16:37, Chris Pollock wrote:
>>>
>>>> On Mon, 2018-12-10 at 13:09 -0500, Bill Cole wrote:
>>>
>>> [...]
>>>>> Anyway, as of today I've capped those 2 subrules at levels
>>>>> which
>>>>> leave ample space to still match the target spam. Should show
>>>>> up
>>>>> in
>>>>> tomorrow's update.
>>>
>>> I was wrong. The addition of a 'maxhits' parameter to the two
>>> subrules apparently didn't get committed in time for the nightly
>>> rule
>>> promotion run. It was in r1848602 and the current ruleset is still
>>> at
>>> r1848555. Assuming all goes well tonight, the change will appear
>>> tomorrow.
>>>
>>
>> Shouldn't this have stopped by now - https://pastebin.com/7260daT3
>> Today's update was '1848731'.
>>
> Hit send too fast. Doing a compare between 72_active.cf dated the 11th
> and the one dated today I do see:
>
> Dated 11 Dec
> if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
> body __E_LIKE_LETTER /<E>/
> tflags __E_LIKE_LETTER multiple
>
> if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> ifplugin
> Mail::SpamAssassin::Plugin::ReplaceTags
> body __LOWER_E
> /e/i
> tflags __LOWER_E multiple
>
> Dated today 13 Dec
> if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
> body __E_LIKE_LETTER /<E>/
> tflags __E_LIKE_LETTER multiple maxhits=400
>
> if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> ifplugin
> Mail::SpamAssassin::Plugin::ReplaceTags
> body __LOWER_E
> /e/
> tflags __LOWER_E multiple maxhits=250
>
> IIUC then __E_LIKE_LETTER can hit a max of 400 times in one message and
> __LOWER_E a max of 250 times in one message.
For now, yes. Those numbers were the result of a mis-think on my part and will be 320 and 230 once the current rev works its way through.
> Therefore I may still have
> a large listing of subtest ran.
Yes.
I don't expect that behavior to change. SA has always tallied rules and sub-rules with multiple matches and the 'multiple' tflag this way and I see no compelling reason to change that. It almost certainly will not change for 3.4.3, which should be the last 3.4.x release.
If there's a bug opened and someone is willing to work on code for whatever changes need to be made to collapse duplicate hit names in the lists of rule matches into a single citation with a count of hits, I expect that change would be accepted for v4, even though it may impact existing users' tooling.
--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Available For Hire: https://linkedin.com/in/billcole
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Chris Pollock <cp...@embarqmail.com>.
On Thu, 2018-12-13 at 15:14 -0600, Chris Pollock wrote:
> On Tue, 2018-12-11 at 19:00 -0500, Bill Cole wrote:
> > On 11 Dec 2018, at 16:37, Chris Pollock wrote:
> >
> > > On Mon, 2018-12-10 at 13:09 -0500, Bill Cole wrote:
> >
> > [...]
> > > > Anyway, as of today I've capped those 2 subrules at levels
> > > > which
> > > > leave ample space to still match the target spam. Should show
> > > > up
> > > > in
> > > > tomorrow's update.
> >
> > I was wrong. The addition of a 'maxhits' parameter to the two
> > subrules apparently didn't get committed in time for the nightly
> > rule
> > promotion run. It was in r1848602 and the current ruleset is still
> > at
> > r1848555. Assuming all goes well tonight, the change will appear
> > tomorrow.
> >
>
> Shouldn't this have stopped by now - https://pastebin.com/7260daT3
> Today's update was '1848731'.
>
Hit send too fast. Doing a compare between 72_active.cf dated the 11th
and the one dated today I do see:
Dated 11 Dec
if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
body __E_LIKE_LETTER /<E>/
tflags __E_LIKE_LETTER multiple
if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
ifplugin
Mail::SpamAssassin::Plugin::ReplaceTags
body __LOWER_E
/e/i
tflags __LOWER_E multiple
Dated today 13 Dec
if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
body __E_LIKE_LETTER /<E>/
tflags __E_LIKE_LETTER multiple maxhits=400
if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
ifplugin
Mail::SpamAssassin::Plugin::ReplaceTags
body __LOWER_E
/e/
tflags __LOWER_E multiple maxhits=250
IIUC then __E_LIKE_LETTER can hit a max of 400 times in one message and
__LOWER_E a max of 250 times in one message. Therefore I may still have
a large listing of subtest ran.
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
15:16:48 up 5 days, 19:40, 1 user, load average: 1.25, 0.89, 0.67
Description: Ubuntu 18.04.1 LTS, kernel 4.15.0-42-generic
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Chris Pollock <cp...@embarqmail.com>.
On Tue, 2018-12-11 at 19:00 -0500, Bill Cole wrote:
> On 11 Dec 2018, at 16:37, Chris Pollock wrote:
>
> > On Mon, 2018-12-10 at 13:09 -0500, Bill Cole wrote:
>
> [...]
> > > Anyway, as of today I've capped those 2 subrules at levels which
> > > leave ample space to still match the target spam. Should show up
> > > in
> > > tomorrow's update.
>
> I was wrong. The addition of a 'maxhits' parameter to the two
> subrules apparently didn't get committed in time for the nightly rule
> promotion run. It was in r1848602 and the current ruleset is still at
> r1848555. Assuming all goes well tonight, the change will appear
> tomorrow.
>
Shouldn't this have stopped by now - https://pastebin.com/7260daT3
Today's update was '1848731'.
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
15:07:23 up 5 days, 19:31, 1 user, load average: 0.64, 0.73, 0.51
Description: Ubuntu 18.04.1 LTS, kernel 4.15.0-42-generic
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Chris Pollock <cp...@embarqmail.com>.
On Tue, 2018-12-11 at 19:00 -0500, Bill Cole wrote:
> On 11 Dec 2018, at 16:37, Chris Pollock wrote:
>
> > On Mon, 2018-12-10 at 13:09 -0500, Bill Cole wrote:
>
> [...]
> > > Anyway, as of today I've capped those 2 subrules at levels which
> > > leave ample space to still match the target spam. Should show up
> > > in
> > > tomorrow's update.
>
> I was wrong. The addition of a 'maxhits' parameter to the two
> subrules apparently didn't get committed in time for the nightly rule
> promotion run. It was in r1848602 and the current ruleset is still at
> r1848555. Assuming all goes well tonight, the change will appear
> tomorrow.
>
> Rule updates never work quite as fast as I hope...
Ah, thanks for that Bill. I'll be sure to check the debug output when
sa-update is run tomorrow for the update version.
>
> > I see in today's update that the subrule was changed from this:
> >
> > if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> > ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
> > meta T_MIXED_ES ( __LOWER_E > 20 ) && (
> > __E_LIKE_LETTER > ( (__LOWER_E * 14 ) / 10) ) && ( (
> > __E_LIKE_LETTER /
> > __LOWER_E ) < 10 )
> > describe T_MIXED_ES Too many es are not es
>
> That's the functional meta-rule.
>
>
> > To this:
> >
> > if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> > ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
> > body __E_LIKE_LETTER /<E>/
> > tflags __E_LIKE_LETTER multiple
>
> That's one subrule. Once my change filters through the update
> process, the 'tflags' lines for __E_LIKE_LETTER and __LOWER_E will
> be:
>
> tflags __LOWER_E multiple maxhits=250
> tflags __E_LIKE_LETTER multiple maxhits=400
>
>
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
19:24:10 up 3 days, 23:48, 1 user, load average: 1.21, 0.58, 0.31
Description: Ubuntu 18.04.1 LTS, kernel 4.15.0-42-generic
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 11 Dec 2018, at 16:37, Chris Pollock wrote:
> On Mon, 2018-12-10 at 13:09 -0500, Bill Cole wrote:
[...]
>> Anyway, as of today I've capped those 2 subrules at levels which
>> leave ample space to still match the target spam. Should show up in
>> tomorrow's update.
I was wrong. The addition of a 'maxhits' parameter to the two subrules apparently didn't get committed in time for the nightly rule promotion run. It was in r1848602 and the current ruleset is still at r1848555. Assuming all goes well tonight, the change will appear tomorrow.
Rule updates never work quite as fast as I hope...
> I see in today's update that the subrule was changed from this:
>
> if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
> meta T_MIXED_ES ( __LOWER_E > 20 ) && (
> __E_LIKE_LETTER > ( (__LOWER_E * 14 ) / 10) ) && ( ( __E_LIKE_LETTER /
> __LOWER_E ) < 10 )
> describe T_MIXED_ES Too many es are not es
That's the functional meta-rule.
> To this:
>
> if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
> ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
> body __E_LIKE_LETTER /<E>/
> tflags __E_LIKE_LETTER multiple
That's one subrule. Once my change filters through the update process, the 'tflags' lines for __E_LIKE_LETTER and __LOWER_E will be:
tflags __LOWER_E multiple maxhits=250
tflags __E_LIKE_LETTER multiple maxhits=400
--
Bill Cole
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Chris Pollock <cp...@embarqmail.com>.
On Mon, 2018-12-10 at 13:09 -0500, Bill Cole wrote:
> On 9 Dec 2018, at 18:23, Chris Pollock wrote:
>
> > On Sun, 2018-12-09 at 13:06 -0500, Bill Cole wrote:
> > > On 9 Dec 2018, at 12:04, Chris Pollock wrote:
> > >
> > > > This is probably very trivial and doesn't affect anything
> > > > except
> > > > maybe
> > > > the size of the headers but I have to ask. When looking at the
> > > > headers
> > > > of some ham I noticed - https://pastebin.com/H7euxqVX the two
> > > > rules
> > > > I
> > > > mention above are in 72_active.cf. Is there a reason for the
> > > > number
> > > > of
> > > > times it's listed? Couldn't each subtest be listed just once
> > > > instead
> > > > of
> > > > multiple times?
> > >
> > > Not with the current documented behavior of the code, given the
> > > way
> > > those sub-rules are designed to work together. The goal is to
> > > identify
> > > messages which use Latin-script 'e' characters but also use many
> > > non-Latin-script characters which look like 'e' but are not. To
> > > make
> > > this determination, the rules require the 'multiple' flag without
> > > a
> > > cap
> > > on thne number of matches which a 'maxhits' parameter would set.
> >
> > Got it, thanks Bill. I've never noticed this before. I also noticed
> > that according to my daily sa-update output this subtest is
> > apparently
> > new or at least it didn't appear in the output until this past Fri.
>
> Correct. See the thread with the subject "No longer just embedded =9D
> characters in blackmail emails" here last week for the background.
>
> > >
> > > It is not recommended to routinely add the list of matched sub-
> > > rules
> > > to
> > > scanned messages.
> > >
> >
> > Any specific reason why? This is just on my home system.
>
> It's got the potential to be VERY noisy (as you've discovered) while
> not really providing much useful info. Not a big deal on a small
> system.
>
>
> Anyway, as of today I've capped those 2 subrules at levels which
> leave ample space to still match the target spam. Should show up in
> tomorrow's update.
I see in today's update that the subrule was changed from this:
if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
meta T_MIXED_ES ( __LOWER_E > 20 ) && (
__E_LIKE_LETTER > ( (__LOWER_E * 14 ) / 10) ) && ( ( __E_LIKE_LETTER /
__LOWER_E ) < 10 )
describe T_MIXED_ES Too many es are not es
To this:
if can(Mail::SpamAssassin::Conf::feature_bug6558_free)
ifplugin Mail::SpamAssassin::Plugin::ReplaceTags
body __E_LIKE_LETTER /<E>/
tflags __E_LIKE_LETTER multiple
SA-update was run at 12:03pm here on my box. A message that came in
well after the update still shows nearly the same output as before
https://pastebin.com/aSXVj5ri
I can't see where the update made any difference Bill. However, maybe I
don't understand the rule and it's doing what it's supposed to.
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
15:24:32 up 3 days, 19:48, 1 user, load average: 0.54, 0.55, 0.33
Description: Ubuntu 18.04.1 LTS, kernel 4.15.0-42-generic
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 9 Dec 2018, at 18:23, Chris Pollock wrote:
> On Sun, 2018-12-09 at 13:06 -0500, Bill Cole wrote:
>> On 9 Dec 2018, at 12:04, Chris Pollock wrote:
>>
>>> This is probably very trivial and doesn't affect anything except
>>> maybe
>>> the size of the headers but I have to ask. When looking at the
>>> headers
>>> of some ham I noticed - https://pastebin.com/H7euxqVX the two rules
>>> I
>>> mention above are in 72_active.cf. Is there a reason for the number
>>> of
>>> times it's listed? Couldn't each subtest be listed just once
>>> instead
>>> of
>>> multiple times?
>>
>> Not with the current documented behavior of the code, given the way
>> those sub-rules are designed to work together. The goal is to
>> identify
>> messages which use Latin-script 'e' characters but also use many
>> non-Latin-script characters which look like 'e' but are not. To make
>> this determination, the rules require the 'multiple' flag without a
>> cap
>> on thne number of matches which a 'maxhits' parameter would set.
>
> Got it, thanks Bill. I've never noticed this before. I also noticed
> that according to my daily sa-update output this subtest is apparently
> new or at least it didn't appear in the output until this past Fri.
Correct. See the thread with the subject "No longer just embedded =9D characters in blackmail emails" here last week for the background.
>>
>> It is not recommended to routinely add the list of matched sub-rules
>> to
>> scanned messages.
>>
> Any specific reason why? This is just on my home system.
It's got the potential to be VERY noisy (as you've discovered) while not really providing much useful info. Not a big deal on a small system.
Anyway, as of today I've capped those 2 subrules at levels which leave ample space to still match the target spam. Should show up in tomorrow's update.
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Chris Pollock <cp...@embarqmail.com>.
On Sun, 2018-12-09 at 13:06 -0500, Bill Cole wrote:
> On 9 Dec 2018, at 12:04, Chris Pollock wrote:
>
> > This is probably very trivial and doesn't affect anything except
> > maybe
> > the size of the headers but I have to ask. When looking at the
> > headers
> > of some ham I noticed - https://pastebin.com/H7euxqVX the two rules
> > I
> > mention above are in 72_active.cf. Is there a reason for the number
> > of
> > times it's listed? Couldn't each subtest be listed just once
> > instead
> > of
> > multiple times?
>
> Not with the current documented behavior of the code, given the way
> those sub-rules are designed to work together. The goal is to
> identify
> messages which use Latin-script 'e' characters but also use many
> non-Latin-script characters which look like 'e' but are not. To make
> this determination, the rules require the 'multiple' flag without a
> cap
> on thne number of matches which a 'maxhits' parameter would set.
Got it, thanks Bill. I've never noticed this before. I also noticed
that according to my daily sa-update output this subtest is apparently
new or at least it didn't appear in the output until this past Fri.
>
> It is not recommended to routinely add the list of matched sub-rules
> to
> scanned messages.
>
Any specific reason why? This is just on my home system.
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
16:26:13 up 1 day, 20:50, 1 user, load average: 1.37, 0.78, 0.54
Description: Ubuntu 18.04.1 LTS, kernel 4.15.0-42-generic
Re: Subtest __E_LIKE_LETTER and __LOWER_E listed many times in
message header
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 9 Dec 2018, at 12:04, Chris Pollock wrote:
> This is probably very trivial and doesn't affect anything except maybe
> the size of the headers but I have to ask. When looking at the headers
> of some ham I noticed - https://pastebin.com/H7euxqVX the two rules I
> mention above are in 72_active.cf. Is there a reason for the number of
> times it's listed? Couldn't each subtest be listed just once instead
> of
> multiple times?
Not with the current documented behavior of the code, given the way
those sub-rules are designed to work together. The goal is to identify
messages which use Latin-script 'e' characters but also use many
non-Latin-script characters which look like 'e' but are not. To make
this determination, the rules require the 'multiple' flag without a cap
on thne number of matches which a 'maxhits' parameter would set.
It is not recommended to routinely add the list of matched sub-rules to
scanned messages.
--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Available For Hire: https://linkedin.com/in/billcole