You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Benny Pedersen <me...@junc.eu> on 2020/11/27 00:53:21 UTC
kam channel
Nov 27 00:05:35.183 [32757] info: body_500: 129 base strings extracted
in 0 seconds
Nov 27 00:05:35.558 [32757] info: rules: meta test KAM_COMBOJDR has
dependency 'KAM_SPAMJDR' with a zero score
Nov 27 00:05:35.558 [32757] info: rules: meta test KAM_COMBOJDR has
dependency 'KAM_SPAMKING' with a zero score
Nov 27 00:05:35.560 [32757] info: rules: meta test KAM_NOTIFY2 has
dependency 'KAM_IFRAME' with a zero score
Nov 27 00:05:35.560 [32757] info: rules: meta test KAM_AUTONEW2 has
dependency 'KAM_NEWNOTICE' with a zero score
Nov 27 00:05:35.561 [32757] info: rules: meta test KAM_PET2 has
dependency 'KAM_PET' with a zero score
Nov 27 00:05:35.564 [32757] info: rules: meta test
KAM_REALLY_FAKE_DELIVER has dependency 'KAM_RPTR_PASSED' with a zero
score
Nov 27 00:05:35.564 [32757] info: rules: meta test KAM_SETTING2 has
dependency 'KAM_SETTING' with a zero score
Nov 27 00:05:35.565 [32757] info: rules: meta test KAM_JURY has
dependency 'KAM_RAPTOR_ALTERED' with a zero score
Nov 27 00:05:35.569 [32757] info: rules: meta test KAM_LIVEURI2 has
dependency 'KAM_PRODUCT' with a zero score
Nov 27 00:05:35.570 [32757] info: rules: meta test KAM_NEWNOTICE2 has
dependency 'KAM_NEWNOTICE' with a zero score
Nov 27 00:05:35.571 [32757] info: rules: meta test KAM_BADPDF2 has
dependency 'KAM_RPTR_SUSPECT' with a zero score
Nov 27 00:05:35.571 [32757] info: rules: meta test KAM_DRILL has
dependency 'KAM_MUSTREAD' with a zero score
Nov 27 00:05:35.572 [32757] info: rules: meta test KAM_DRUG has
dependency 'KAM_REPLACE' with a zero score
Nov 27 00:05:35.581 [32757] info: rules: meta test KAM_GARAGE2 has
dependency 'KAM_GARAGE' with a zero score
Nov 27 00:05:35.585 [32757] info: rules: meta test KAM_DISH2 has
dependency 'KAM_DISH' with a zero score
Nov 27 00:05:35.586 [32757] info: rules: meta test JMQ_CONGRAT has
dependency 'KAM_RAPTOR_ALTERED' with a zero score
Nov 27 00:05:35.586 [32757] info: rules: meta test KAM_CARD has
dependency 'KAM_RPTR_SUSPECT' with a zero score
Nov 27 00:05:35.586 [32757] info: rules: meta test KAM_RE_PLUS has
dependency 'KAM_RE' with a zero score
Nov 27 00:05:35.589 [32757] info: rules: meta test KAM_REFINEW2 has
dependency 'KAM_REFINEW' with a zero score
Nov 27 00:05:35.589 [32757] info: rules: meta test KAM_REFINEW2 has
dependency 'KAM_NEWNOTICE' with a zero score
Nov 27 00:05:35.589 [32757] info: rules: meta test KAM_FAKE_DELIVER has
dependency 'KAM_RAPTOR_ALTERED' with a zero score
why so many rules with 0 score ?
should rules not be __KAM... then ?
Re: kam channel
Posted by "Kevin A. McGrail" <km...@apache.org>.
Thanks! We'll look to use this info to refine our rules more.
On 11/29/2020 8:05 PM, RW wrote:
> On Sat, 28 Nov 2020 09:32:38 -0500
> Kevin A. McGrail wrote:
>
>> Some thoughts in line below.
>>
>> On Fri, Nov 27, 2020, 12:13 RW <rw...@googlemail.com> wrote:
>>> Maybe there's a difference in trunk, but otherwise the sub-rules
>>> that do the work still run when they aren't used, so there's little
>>> benefit.
>> I believe if you look you'll find that we actually redefine some of
>> the rules and then score them zero just for that purpose but if you
>> find any that look to be still running please let me know.
> I looked at a half-dozen random examples of zero scored meta rules and
> none of then had suppressed sub-rules. I didn't recurse them very
> thoroughly though, but here's an obvious one:
>
> score FROM_FMBLA_NDBLOCKED 0 # __FROM_FMBLA_NDBLOCKED
>
> __FROM_FMBLA_NDBLOCKED is an askdns rule that used by nothing but
> FROM_FMBLA_NDBLOCKED, and it's not redefined in KAM_deadweight2_sub.cf
>
>
> One thing I noticed is:
>
> meta __RCVD_IN_ZEN 0
> meta __RCVD_IN_DNSWL 0
>
> which will presumably turn-off RCVD_IN_SBL, RCVD_IN_SBL_CSS, and the
> various RCVD_IN_DNSWL_* rules.
>
> There's also:
>
> meta __RCVD_IN_LASHBACK 0
> meta __RCVD_IN_HOSTKARMA 0
> meta __RCVD_IN_RPBL 0
> meta __RCVD_IN_SORBS 0
> meta __RCVD_IN_IADB 0
> meta __RCVD_IN_MSPIKE_B 0
> meta __RCVD_IN_MSPIKE_L 0
>
>
>
>
>
>
>
>
>
--
Kevin A. McGrail
KMcGrail@Apache.org
Member, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171
Re: kam channel
Posted by RW <rw...@googlemail.com>.
On Sat, 28 Nov 2020 09:32:38 -0500
Kevin A. McGrail wrote:
> Some thoughts in line below.
>
> On Fri, Nov 27, 2020, 12:13 RW <rw...@googlemail.com> wrote:
> > Maybe there's a difference in trunk, but otherwise the sub-rules
> > that do the work still run when they aren't used, so there's little
> > benefit.
> I believe if you look you'll find that we actually redefine some of
> the rules and then score them zero just for that purpose but if you
> find any that look to be still running please let me know.
I looked at a half-dozen random examples of zero scored meta rules and
none of then had suppressed sub-rules. I didn't recurse them very
thoroughly though, but here's an obvious one:
score FROM_FMBLA_NDBLOCKED 0 # __FROM_FMBLA_NDBLOCKED
__FROM_FMBLA_NDBLOCKED is an askdns rule that used by nothing but
FROM_FMBLA_NDBLOCKED, and it's not redefined in KAM_deadweight2_sub.cf
One thing I noticed is:
meta __RCVD_IN_ZEN 0
meta __RCVD_IN_DNSWL 0
which will presumably turn-off RCVD_IN_SBL, RCVD_IN_SBL_CSS, and the
various RCVD_IN_DNSWL_* rules.
There's also:
meta __RCVD_IN_LASHBACK 0
meta __RCVD_IN_HOSTKARMA 0
meta __RCVD_IN_RPBL 0
meta __RCVD_IN_SORBS 0
meta __RCVD_IN_IADB 0
meta __RCVD_IN_MSPIKE_B 0
meta __RCVD_IN_MSPIKE_L 0
Re: kam channel
Posted by "Kevin A. McGrail" <km...@apache.org>.
Some thoughts in line below.
On Fri, Nov 27, 2020, 12:13 RW <rw...@googlemail.com> wrote:
> On Thu, 26 Nov 2020 19:56:49 -0500
> Kevin A. McGrail wrote:
>
>
> > We have the ability to automatically determine rules that are not
> > needed and use zero scores to turn them off.
>
> A lot of them are core rules which should really be removed properly.
>
With the way that the rules are generated, scored, qa'd, etc. This concept
has worked for us for years.
>
> > To my knowledge in the history of the project, no one is ever looked
> > at turning rules off and especially not doing so with meta rules.
>
> Maybe there's a difference in trunk, but otherwise the sub-rules
> that do the work still run when they aren't used, so there's little
> benefit.
>
I believe if you look you'll find that we actually redefine some of the
rules and then score them zero just for that purpose but if you find any
that look to be still running please let me know.
According to a comment in "KAM_heavyweight.cf" 15% comes from...
>
I wouldn't put too much reading into that specific comment. Heavyweight was
my first idea for how to improve the set and then the dead weight and then
the dead weight too and then the dead weight for meta.
Then others like Karsten figured out how we could actually disable meta
rules.
Throughout all of it we've maintained a baseline machine and corpora to do
comparisons for efficiency and efficacy that takes into account the entire
system.
We've also been working heavily on improving the entire SA, especially
Giovanni, because the code got significantly slower with some of the recent
CVEs. Which we have also been able to use on a baseline machine too.
Regards, KAM
>
Re: kam channel
Posted by RW <rw...@googlemail.com>.
On Thu, 26 Nov 2020 19:56:49 -0500
Kevin A. McGrail wrote:
> We have the ability to automatically determine rules that are not
> needed and use zero scores to turn them off.
A lot of them are core rules which should really be removed properly.
> To my knowledge in the history of the project, no one is ever looked
> at turning rules off and especially not doing so with meta rules.
Maybe there's a difference in trunk, but otherwise the sub-rules
that do the work still run when they aren't used, so there's little
benefit.
It's right to suppress them for causing FPs of course.
> The ruleset will improve
> your performance by about 20% without lowering the efficacy rate.
According to a comment in "KAM_heavyweight.cf" 15% comes from
redefining away just 8 sub-rules.
__FILL_THIS_FORM_FRAUD_PHISH1
__FILL_THIS_FORM_LOAN1
__FILL_THIS_FORM_LONG2
__FILL_THIS_FORM_SHORT2
__KAM_ALARM3
__KAM_DISH3
__KAM_SKIN3
__KAM_WEIGHT4
Re: kam channel
Posted by "Kevin A. McGrail" <km...@apache.org>.
Hi Benny, thanks for the feedback. If you look in the channel files you'll
notice there are some files like dead weight, dead weight Meta and
heavyweight.
We have the ability to automatically determine rules that are not needed
and use zero scores to turn them off. You can ignore this output.
To my knowledge in the history of the project, no one is ever looked at
turning rules off and especially not doing so with meta rules. If I
remember correctly, you'll find that using The ruleset will improve your
performance by about 20% without lowering the efficacy rate. The zero
scoring rule technology is an important part of that.
On Thu, Nov 26, 2020, 19:53 Benny Pedersen <me...@junc.eu> wrote:
> Nov 27 00:05:35.183 [32757] info: body_500: 129 base strings extracted
> in 0 seconds
> Nov 27 00:05:35.558 [32757] info: rules: meta test KAM_COMBOJDR has
> dependency 'KAM_SPAMJDR' with a zero score
> Nov 27 00:05:35.558 [32757] info: rules: meta test KAM_COMBOJDR has
> dependency 'KAM_SPAMKING' with a zero score
> Nov 27 00:05:35.560 [32757] info: rules: meta test KAM_NOTIFY2 has
> dependency 'KAM_IFRAME' with a zero score
> Nov 27 00:05:35.560 [32757] info: rules: meta test KAM_AUTONEW2 has
> dependency 'KAM_NEWNOTICE' with a zero score
> Nov 27 00:05:35.561 [32757] info: rules: meta test KAM_PET2 has
> dependency 'KAM_PET' with a zero score
> Nov 27 00:05:35.564 [32757] info: rules: meta test
> KAM_REALLY_FAKE_DELIVER has dependency 'KAM_RPTR_PASSED' with a zero
> score
> Nov 27 00:05:35.564 [32757] info: rules: meta test KAM_SETTING2 has
> dependency 'KAM_SETTING' with a zero score
> Nov 27 00:05:35.565 [32757] info: rules: meta test KAM_JURY has
> dependency 'KAM_RAPTOR_ALTERED' with a zero score
> Nov 27 00:05:35.569 [32757] info: rules: meta test KAM_LIVEURI2 has
> dependency 'KAM_PRODUCT' with a zero score
> Nov 27 00:05:35.570 [32757] info: rules: meta test KAM_NEWNOTICE2 has
> dependency 'KAM_NEWNOTICE' with a zero score
> Nov 27 00:05:35.571 [32757] info: rules: meta test KAM_BADPDF2 has
> dependency 'KAM_RPTR_SUSPECT' with a zero score
> Nov 27 00:05:35.571 [32757] info: rules: meta test KAM_DRILL has
> dependency 'KAM_MUSTREAD' with a zero score
> Nov 27 00:05:35.572 [32757] info: rules: meta test KAM_DRUG has
> dependency 'KAM_REPLACE' with a zero score
> Nov 27 00:05:35.581 [32757] info: rules: meta test KAM_GARAGE2 has
> dependency 'KAM_GARAGE' with a zero score
> Nov 27 00:05:35.585 [32757] info: rules: meta test KAM_DISH2 has
> dependency 'KAM_DISH' with a zero score
> Nov 27 00:05:35.586 [32757] info: rules: meta test JMQ_CONGRAT has
> dependency 'KAM_RAPTOR_ALTERED' with a zero score
> Nov 27 00:05:35.586 [32757] info: rules: meta test KAM_CARD has
> dependency 'KAM_RPTR_SUSPECT' with a zero score
> Nov 27 00:05:35.586 [32757] info: rules: meta test KAM_RE_PLUS has
> dependency 'KAM_RE' with a zero score
> Nov 27 00:05:35.589 [32757] info: rules: meta test KAM_REFINEW2 has
> dependency 'KAM_REFINEW' with a zero score
> Nov 27 00:05:35.589 [32757] info: rules: meta test KAM_REFINEW2 has
> dependency 'KAM_NEWNOTICE' with a zero score
> Nov 27 00:05:35.589 [32757] info: rules: meta test KAM_FAKE_DELIVER has
> dependency 'KAM_RAPTOR_ALTERED' with a zero score
>
> why so many rules with 0 score ?
>
> should rules not be __KAM... then ?
>