You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Nuno Fernandes <np...@eurotux.com> on 2014/03/19 10:14:11 UTC
Disable awl when some other rule hit
Hello,
Is it possible to disable awl (or at least score it 0.001) when a special rule
hit like:
if URIBL_DBL_SPAM
score AWL 0
endif
Is there any other way to achieve this goal?
Thanks,
Nuno Fernandes
Re: Disable awl when some other rule hit
Posted by Nuno Fernandes <np...@eurotux.com>.
On Monday 07 April 2014 08:00:38 Kevin A. McGrail wrote:
> Please move this patch to a bug for SA. Add more comments and some
> documentation for the feature and I don't see why we couldn't patch for
> you. However, also realize that I'm looking heavily at things like
> TxRep to replace AWL which is in trunk.
Done. Please find it at
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=7032
Best regards,
Nuno Fernandes
Re: Disable awl when some other rule hit
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 4/7/2014 5:08 AM, Nuno Fernandes wrote:
>> Nevertheless i think the following one liner would do the trick (have to
>> test it though):
>>
>> --- Mail/SpamAssassin/Plugin/AWL.pm.orig 2014-03-24
>> 11:31:18.000000000 +0000
>> +++ Mail/SpamAssassin/Plugin/AWL.pm 2014-03-24 11:31:20.000000000 +0000
>> @@ -437,6 +437,7 @@
>> # or if the test is marked as "noautolearn"
>> next if !$scores->{$test};
>> next if exists $tflags->{$test} && $tflags->{$test} =~
>> /\bnoautolearn\b/; + return 0 if $tflags->{$test} =~ /\bnoawl\b/;
>> $points += $scores->{$test};
>> }
>>
>
> The correct patch should be:
>
> diff -Naur Mail-SpamAssassin-3.4.0.orig/lib/Mail/SpamAssassin/Plugin/AWL.pm
> Mail-SpamAssassin-3.4.0/lib/Mail/SpamAssassin/Plugin/AWL.pm
> --- Mail-SpamAssassin-3.4.0.orig/lib/Mail/SpamAssassin/Plugin/AWL.pm
> 2014-02-07 08:36:27.000000000 +0000
> +++ Mail-SpamAssassin-3.4.0/lib/Mail/SpamAssassin/Plugin/AWL.pm 2014-03-25
> 10:25:27.000000000 +0000
> @@ -437,6 +437,7 @@
> # or if the test is marked as "noautolearn"
> next if !$scores->{$test};
> next if exists $tflags->{$test} && $tflags->{$test} =~ /\bnoautolearn\b/;
> + return 0 if exists $tflags->{$test} && $tflags->{$test} =~ /\bnoawl\b/;
> $points += $scores->{$test};
> }
>
> As sa-learn was giving errors like:
>
> Use of uninitialized value in pattern match (m//) at
> /usr/share/perl5/vendor_perl/Mail/SpamAssassin/Plugin/AWL.pm line 440.
>
> Best regards,
> Nuno Fernandes
Please move this patch to a bug for SA. Add more comments and some
documentation for the feature and I don't see why we couldn't patch for
you. However, also realize that I'm looking heavily at things like
TxRep to replace AWL which is in trunk.
Re: Disable awl when some other rule hit
Posted by Nuno Fernandes <np...@eurotux.com>.
> Nevertheless i think the following one liner would do the trick (have to
> test it though):
>
> --- Mail/SpamAssassin/Plugin/AWL.pm.orig 2014-03-24
> 11:31:18.000000000 +0000
> +++ Mail/SpamAssassin/Plugin/AWL.pm 2014-03-24 11:31:20.000000000 +0000
> @@ -437,6 +437,7 @@
> # or if the test is marked as "noautolearn"
> next if !$scores->{$test};
> next if exists $tflags->{$test} && $tflags->{$test} =~
> /\bnoautolearn\b/; + return 0 if $tflags->{$test} =~ /\bnoawl\b/;
> $points += $scores->{$test};
> }
>
The correct patch should be:
diff -Naur Mail-SpamAssassin-3.4.0.orig/lib/Mail/SpamAssassin/Plugin/AWL.pm
Mail-SpamAssassin-3.4.0/lib/Mail/SpamAssassin/Plugin/AWL.pm
--- Mail-SpamAssassin-3.4.0.orig/lib/Mail/SpamAssassin/Plugin/AWL.pm
2014-02-07 08:36:27.000000000 +0000
+++ Mail-SpamAssassin-3.4.0/lib/Mail/SpamAssassin/Plugin/AWL.pm 2014-03-25
10:25:27.000000000 +0000
@@ -437,6 +437,7 @@
# or if the test is marked as "noautolearn"
next if !$scores->{$test};
next if exists $tflags->{$test} && $tflags->{$test} =~ /\bnoautolearn\b/;
+ return 0 if exists $tflags->{$test} && $tflags->{$test} =~ /\bnoawl\b/;
$points += $scores->{$test};
}
As sa-learn was giving errors like:
Use of uninitialized value in pattern match (m//) at
/usr/share/perl5/vendor_perl/Mail/SpamAssassin/Plugin/AWL.pm line 440.
Best regards,
Nuno Fernandes
Re: Disable awl when some other rule hit
Posted by Ivo Truxa <iv...@truxa.cz>.
I see that you know exactly what you want, but still feel obliged to post a
reply anyway, because this thread is publicly accessible in archives, hence
people who search a solution for a similar problem may replicate your patch
without realizing the consequences. So you can ignore my comments below,
they are not meant for you, they are here rather just for the reference:
Disabling AWL with the proposed patch has the following consequences:
1) AWL does not work and does not store the score when the condition in the
patch is met, hence not adjusting the sender history track in such cases
2) By skipping AWL on certain conditions, you open door to false positives
that AWL was designed to handle. For example if you skip AWL at certain
URIBL hits, it can happen that a good sender forwards a recipient on your
server a message containing a banned URI (I see it happening often on my
server). AWL would compensate it, and let the email from the good sender
pass unless there were other spam markers sufficiently high to override it.
3) Bypassing the AWL does not save any considerable time - the AWL run times
are negligible in comparison to RBL lookups, or even in comparison with the
parsing of rules.
4) It is a non-systematic modification and it is necessary to maintain it,
re-patching the AWL plugin after each update
So before applying this patch, other alternatives should be considered
first:
- adjusting scores for given spam markers sufficiently high, so that AWL
won't override them too easily
- reducing the awl_factor parameter
- adding rules suppressing AWL on certain condition (as shown in the
previous reply)
- using a shortcircuit
- checking RBL's in the MTU before SpamAssassin
- replacing AWL with TxRep
And when, despite the above, patching is chosen anyway, preferably it should
be done in the way suggested in my previous post, which lets AWL recording
the score while not returning any value.
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Disable-awl-when-some-other-rule-hit-tp108598p108687.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Disable awl when some other rule hit
Posted by Nuno Fernandes <np...@eurotux.com>.
On Sunday 30 March 2014 13:52:43 Ivo Truxa wrote:
> Nuno Fernandes-2 wrote
>
> > Yes.. you are correct. The result is not added to the AWL database but i'm
> > ok with that.
>
> Personally I think it makes no sense using AWL when you do not let it work,
Oh.. but it works. Only in some scenarios i would like it to be disabled
(scoring and adding to the DB).
> and do not store all scores. A better place to make the modification is at
> the bottom of the sub check_from_in_auto_whitelist(), in the condition "if
> ($delta != 0)". Store the value of the noawl tag in the loop where your
> current code is, but don't return from there. Let the sub parse through, and
> skip returning the delta score at the bottom.
Yes.. but in this case i'm doing unnecessary cpu instructions only to record
in db the value. For me the flag noawl should be to disable all awl features.
If I created something like the flag "noawlscoring" your idea seems fine.
> However, unless you manage to get this modification to the official code,
> you have to redo the patch after each update, so as I wrote previously, IMHO
> the better way to handle it, is adjusting the tag scores so that you do not
> need to disable AWL. You could also write a rule based on the concerned tag
> values, in combination with the AWL value, so that it does the same trick
> without the need to hack the code. Something in a way similar to this one:
>
> meta AWL_FIX (URIBL_DBL_SPA || SOMETHING_ELSE || ANOTHER_ONE) && AWL < -3
> describe AWL_FIX Suppressing AWL
> priority AWL_FIX 1100
> score AWL_FIX 5.0
I've considered and even implemented rules similar to the above but they all
remain short off completely remove the awl score. I need this to have complete
predictability of the final score on some emails that had the same issue of awl
working against the sysadmin (like you've stated in your TxRep code).
> It is always preferable customizing your installation through settings and
> rules, than hacking the code.
Yes.. but i already maintain our company spamassassin package with some other
patches (a lot bigger that this) and this oneliner is trivial to maintain (at
least for now).
Best regards,
Nuno Fernandes
Re: Disable awl when some other rule hit
Posted by Ivo Truxa <iv...@truxa.cz>.
Ivo Truxa wrote
>
> RW-15 wrote
>> Ivo Truxa wrote:
>>> meta AWL_FIX (URIBL_DBL_SPA || SOMETHING_ELSE || ANOTHER_ONE) && AWL
>>> < -3
>>
>> The value of AWL in the above is either 0 or 1, so the test is
>> unconditionally false. AFAIK there's no way to write a meta rule that
>> tests a rule's score.
> No, the value of AWL is a float value different than 0, and the meta rule
> works fine - I tested it prior posting.
I apologize, I re-tested, and you are right. At the previous test I did not
verified all values, so it seemed to work, since tested with small positive
values.
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Disable-awl-when-some-other-rule-hit-tp108598p108689.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Disable awl when some other rule hit
Posted by Ivo Truxa <iv...@truxa.cz>.
RW-15 wrote
> Ivo Truxa wrote:
>> meta AWL_FIX (URIBL_DBL_SPA || SOMETHING_ELSE || ANOTHER_ONE) && AWL <
>> -3
>
> The value of AWL in the above is either 0 or 1, so the test is
> unconditionally false. AFAIK there's no way to write a meta rule that
> tests a rule's score.
No, the value of AWL is a float value different than 0, and the meta rule
works fine - I tested it prior posting.
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Disable-awl-when-some-other-rule-hit-tp108598p108688.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Disable awl when some other rule hit
Posted by RW <rw...@googlemail.com>.
On Sun, 30 Mar 2014 13:52:43 -0700 (PDT)
Ivo Truxa wrote:
> You could also write a rule based on the concerned tag
> values, in combination with the AWL value, so that it does the same
> trick without the need to hack the code. Something in a way similar
> to this one:
>
> meta AWL_FIX (URIBL_DBL_SPA || SOMETHING_ELSE || ANOTHER_ONE) && AWL < -3
The value of AWL in the above is either 0 or 1, so the test is
unconditionally false. AFAIK there's no way to write a meta rule that
tests a rule's score.
Re: Disable awl when some other rule hit
Posted by Ivo Truxa <iv...@truxa.cz>.
Nuno Fernandes-2 wrote
> Yes.. you are correct. The result is not added to the AWL database but i'm
> ok with that.
Personally I think it makes no sense using AWL when you do not let it work,
and do not store all scores. A better place to make the modification is at
the bottom of the sub check_from_in_auto_whitelist(), in the condition "if
($delta != 0)". Store the value of the noawl tag in the loop where your
current code is, but don't return from there. Let the sub parse through, and
skip returning the delta score at the bottom.
However, unless you manage to get this modification to the official code,
you have to redo the patch after each update, so as I wrote previously, IMHO
the better way to handle it, is adjusting the tag scores so that you do not
need to disable AWL. You could also write a rule based on the concerned tag
values, in combination with the AWL value, so that it does the same trick
without the need to hack the code. Something in a way similar to this one:
meta AWL_FIX (URIBL_DBL_SPA || SOMETHING_ELSE || ANOTHER_ONE) && AWL < -3
describe AWL_FIX Suppressing AWL
priority AWL_FIX 1100
score AWL_FIX 5.0
It is always preferable customizing your installation through settings and
rules, than hacking the code.
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Disable-awl-when-some-other-rule-hit-tp108598p108678.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Disable awl when some other rule hit
Posted by Nuno Fernandes <np...@eurotux.com>.
On Tuesday 25 March 2014 16:18:42 Ivo Truxa wrote:
> Looks OK, but the problem is that the new score won't be added to the AWL
> database. So for example if AWL tells the average score is 1.0, your RBL
> tells it ought to be 10.0, AWL would normally reflect it, the average score
> for the given email/IP combination would increase, and the next time AWL
> could catch the spammer even if the RBL failed to do it (for example when
> using another dynamic IP on the same block, or when the RBL was offline).
Yes.. you are correct. The result is not added to the AWL database but i'm ok
with that.
> If you fear that AWL would override the score, just set the score for your
> RBL sufficiently high, and it won't happen. You could also try the new TxRep
> replacement of AWL - it uses a more sophisticated method, learns quicker,
> and avoids better false results. Unlike AWL, it can be also trained on past
> spam and relearned on new one, and it can also use the auto-learn feature,
> so if your high priority RBL triggers the auto-learn process, the sender's
> reputation will be adjusted accordingly (much stronger than just by adding
> the new score to the total).
Thanks for the tip.. I'll look into it...
Nuno Fernandes
Re: Disable awl when some other rule hit
Posted by Ivo Truxa <iv...@truxa.cz>.
Looks OK, but the problem is that the new score won't be added to the AWL
database. So for example if AWL tells the average score is 1.0, your RBL
tells it ought to be 10.0, AWL would normally reflect it, the average score
for the given email/IP combination would increase, and the next time AWL
could catch the spammer even if the RBL failed to do it (for example when
using another dynamic IP on the same block, or when the RBL was offline).
If you fear that AWL would override the score, just set the score for your
RBL sufficiently high, and it won't happen. You could also try the new TxRep
replacement of AWL - it uses a more sophisticated method, learns quicker,
and avoids better false results. Unlike AWL, it can be also trained on past
spam and relearned on new one, and it can also use the auto-learn feature,
so if your high priority RBL triggers the auto-learn process, the sender's
reputation will be adjusted accordingly (much stronger than just by adding
the new score to the total).
--
View this message in context: http://spamassassin.1065346.n5.nabble.com/Disable-awl-when-some-other-rule-hit-tp108598p108644.html
Sent from the SpamAssassin - Users mailing list archive at Nabble.com.
Re: Disable awl when some other rule hit
Posted by Nuno Fernandes <np...@eurotux.com>.
On Monday 24 March 2014 12:18:05 Tom Hendrikx wrote:
> On 03/24/2014 12:14 PM, Nuno Fernandes wrote:
> > On Thursday 20 March 2014 07:50:50 Matt Kettler wrote:
> >>> Does this do it?
> >>>
> >>> score AWL 0
> >>> meta LOCAL_SCORE_AWL AWL && !URIBL_DBL_SPAM
> >>> score LOCAL_SCORE_AWL -10
> >>>
> >>> where -10 is whatever score AWL usually has (I forget)
> >>
> >> AWL has a variable score, so you can't negate it by a fixed-score rule..
> >> Fundamentally, it is a past-history based score averaging system, hence
> >> the scores for it constantly change.
> >
> > That's why i would like to remove it in certain scenarios. If it the mail
> > hits my local RBL i would like to remove AWL all together.
> >
> > Guess i will have to hack AWL.pm :(
> >
> > Thanks,
> > Nuno Fernandes
>
> If the rbl gets preference above spamassassin scoring, then why don't
> you just implement the rbl as a separate check in your mta, and make the
> mta reject/quarantine the message without consulting spamassassin at
> all? Just an idea;)
>
> Tom
Hello,
You are correct but the rbl in the mta (in my postfix configuration) would
bounce the email to the sender. My spamassassin+amavis quarantines it so that
the user can release it later on.
Nevertheless i think the following one liner would do the trick (have to test
it though):
--- Mail/SpamAssassin/Plugin/AWL.pm.orig 2014-03-24 11:31:18.000000000
+0000
+++ Mail/SpamAssassin/Plugin/AWL.pm 2014-03-24 11:31:20.000000000 +0000
@@ -437,6 +437,7 @@
# or if the test is marked as "noautolearn"
next if !$scores->{$test};
next if exists $tflags->{$test} && $tflags->{$test} =~ /\bnoautolearn\b/;
+ return 0 if $tflags->{$test} =~ /\bnoawl\b/;
$points += $scores->{$test};
}
If some other test (that has score!=0) has the tflag noawl, this plugin
returns immediately.
Any ideas or problems is this one liner?
Thanks,
Nuno Fernandes
--
Re: Disable awl when some other rule hit
Posted by Tom Hendrikx <to...@whyscream.net>.
On 03/24/2014 12:14 PM, Nuno Fernandes wrote:
> On Thursday 20 March 2014 07:50:50 Matt Kettler wrote:
>
>>> Does this do it?
>>>
>>> score AWL 0
>>> meta LOCAL_SCORE_AWL AWL && !URIBL_DBL_SPAM
>>> score LOCAL_SCORE_AWL -10
>>>
>>> where -10 is whatever score AWL usually has (I forget)
>>
>> AWL has a variable score, so you can't negate it by a fixed-score rule..
>> Fundamentally, it is a past-history based score averaging system, hence
>> the scores for it constantly change.
>
> That's why i would like to remove it in certain scenarios. If it the mail hits
> my local RBL i would like to remove AWL all together.
>
> Guess i will have to hack AWL.pm :(
>
> Thanks,
> Nuno Fernandes
>
If the rbl gets preference above spamassassin scoring, then why don't
you just implement the rbl as a separate check in your mta, and make the
mta reject/quarantine the message without consulting spamassassin at
all? Just an idea;)
Tom
Re: Disable awl when some other rule hit
Posted by Nuno Fernandes <np...@eurotux.com>.
On Thursday 20 March 2014 07:50:50 Matt Kettler wrote:
> > Does this do it?
> >
> > score AWL 0
> > meta LOCAL_SCORE_AWL AWL && !URIBL_DBL_SPAM
> > score LOCAL_SCORE_AWL -10
> >
> > where -10 is whatever score AWL usually has (I forget)
>
> AWL has a variable score, so you can't negate it by a fixed-score rule..
> Fundamentally, it is a past-history based score averaging system, hence
> the scores for it constantly change.
That's why i would like to remove it in certain scenarios. If it the mail hits
my local RBL i would like to remove AWL all together.
Guess i will have to hack AWL.pm :(
Thanks,
Nuno Fernandes
Re: Disable awl when some other rule hit
Posted by Matt Kettler <mk...@verizon.net>.
On 3/19/2014 1:44 PM, Joseph Brennan wrote:
>
>
> --On March 19, 2014 9:58:29 -0400 "Kevin A. McGrail"
> <KM...@pccc.com> wrote:
>
>> On 3/19/2014 5:14 AM, Nuno Fernandes wrote:
>>> Hello,
>>>
>>> Is it possible to disable awl (or at least score it 0.001) when a
>>> special rule hit like:
>>>
>>>
>>> if URIBL_DBL_SPAM
>>> score AWL 0
>>> endif
>>>
>>>
>>> Is there any other way to achieve this goal?
>>
>> I can't think of anyway to do it without adding functionality to SA,
>> sorry.
>
> Does this do it?
>
> score AWL 0
> meta LOCAL_SCORE_AWL AWL && !URIBL_DBL_SPAM
> score LOCAL_SCORE_AWL -10
>
> where -10 is whatever score AWL usually has (I forget)
>
>
AWL has a variable score, so you can't negate it by a fixed-score rule..
Fundamentally, it is a past-history based score averaging system, hence
the scores for it constantly change.
Re: Disable awl when some other rule hit
Posted by Joseph Brennan <br...@columbia.edu>.
--On March 19, 2014 9:58:29 -0400 "Kevin A. McGrail" <KM...@pccc.com>
wrote:
> On 3/19/2014 5:14 AM, Nuno Fernandes wrote:
>> Hello,
>>
>> Is it possible to disable awl (or at least score it 0.001) when a
>> special rule hit like:
>>
>>
>> if URIBL_DBL_SPAM
>> score AWL 0
>> endif
>>
>>
>> Is there any other way to achieve this goal?
>
> I can't think of anyway to do it without adding functionality to SA,
> sorry.
Does this do it?
score AWL 0
meta LOCAL_SCORE_AWL AWL && !URIBL_DBL_SPAM
score LOCAL_SCORE_AWL -10
where -10 is whatever score AWL usually has (I forget)
Joseph Brennan
Columbia U I T
Re: Disable awl when some other rule hit
Posted by "Kevin A. McGrail" <KM...@PCCC.com>.
On 3/19/2014 5:14 AM, Nuno Fernandes wrote:
> Hello,
>
> Is it possible to disable awl (or at least score it 0.001) when a special rule
> hit like:
>
>
> if URIBL_DBL_SPAM
> score AWL 0
> endif
>
>
> Is there any other way to achieve this goal?
I can't think of anyway to do it without adding functionality to SA, sorry.