You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Simon Wilson <si...@simonandkate.net> on 2019/10/16 10:19:07 UTC
Trusted network mail spam detection
Hi, I have a Horde system submitting to a
postfix/amavisd-new/spamassassin server for spam detection (different
servers, same subnet). I *do* consciously run SA over internally
submitted emails to catch compromised accounts (it happened once to me
when a family member's email password was compromised and a bunch of
spam got sent out).
I'm having occasional issues with mail sent by some users from their
home ISP connections (i.e. Chrome client on ISP dynamic IP -> Horde
server/postfix etc). Email validly sent through the trusted host Horde
server gets a bonus (ALL_TRUSTED = -2) which SA is triggering fine
when appropriate, but some emails are still triggering thresholds, so
I was wondering what others do for configuring for traffic that is
*mostly* trusted but should still be checked for obvious spam?
This is not a new system, it's well trained with thousands of ham and
spam over several years. This email was genuine ham, and was discarded
(Amavis threshold 6.0 -> discard).
Content analysis details: (6.0 points, 6.2 required)
pts rule name description
---- ---------------------- --------------------------------------------------
-2.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60%
[score: 0.5000]
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
0.0 HTML_MESSAGE BODY: HTML included in message
3.6 HELO_DYNAMIC_IPADDR2 Relay HELO'd using suspicious hostname (IP addr
2)
1.0 RDNS_DYNAMIC Delivered to internal network by host with
dynamic-looking rDNS
2.1 TO_NO_BRKTS_DYNIP To: lacks brackets and dynamic rDNS
0.5 NO_FM_NAME_IP_HOSTN No From name + hostname using IP address
It feels like the dynamic IP rules are killing it here - what do
others do for valid dynamic IP emails inbound from a web client email
through trusted hosts? Just give ALL_TRUSTED more of a boost? Or
anything more scientific??
Simon.
--
Simon Wilson
M: 0400 12 11 16
Re: Trusted network mail spam detection
Posted by Grant Taylor <gt...@tnetconsulting.net>.
On 10/16/19 6:57 AM, Simon Wilson wrote:
> So how do I configure it such that if it's an authenticated submission
> (587) mail through my mail host at (int)192.68.1.230/(ext)119.18.34.29
> further upstream RECEIVED headers are NOT scanned by SA for dynamic IP?
> Am I still totally misunderstanding trust path in SA?
Can you configure your upstream MSA to not include the client IP address
in it's Received: header?
I have my MSA insert a header like the following:
Received: from Contact-TNet-Consulting-Abuse-for-assistance …
The rest of the header has germane information that TNet Abuse can use
to identify the sending host.
This means that there is nothing for downstream servers to detect as a
dynamic IP and get cross with.
--
Grant. . . .
unix || die
Re: Trusted network mail spam detection
Posted by Simon Wilson <si...@simonandkate.net>.
Quoting Simon Wilson <si...@simonandkate.net>:
> Quoting Tom Hendrikx <to...@whyscream.net>:
>
>> On 16-10-19 12:19, Simon Wilson wrote:
>>> Hi, I have a Horde system submitting to a
>>> postfix/amavisd-new/spamassassin server for spam detection
>>> (different servers, same subnet). I *do* consciously run SA over
>>> internally submitted emails to catch compromised accounts (it
>>> happened once to me when a family member's email password was
>>> compromised and a bunch of spam got sent out).
>>>
>>> I'm having occasional issues with mail sent by some users from
>>> their home ISP connections (i.e. Chrome client on ISP dynamic IP
>>> -> Horde server/postfix etc). Email validly sent through the
>>> trusted host Horde server gets a bonus (ALL_TRUSTED = -2) which SA
>>> is triggering fine when appropriate, but some emails are still
>>> triggering thresholds, so I was wondering what others do for
>>> configuring for traffic that is *mostly* trusted but should still
>>> be checked for obvious spam?
>>>
>>> This is not a new system, it's well trained with thousands of ham
>>> and spam over several years. This email was genuine ham, and was
>>> discarded (Amavis threshold 6.0 -> discard).
>>>
>>> Content analysis details: (6.0 points, 6.2 required)
>>>
>>> pts rule name description
>>> ---- ----------------------
>>> --------------------------------------------------
>>> -2.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
>>> 0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60%
>>> [score: 0.5000]
>>> 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
>>> 0.0 HTML_MESSAGE BODY: HTML included in message
>>> 3.6 HELO_DYNAMIC_IPADDR2 Relay HELO'd using suspicious hostname (IP addr
>>> 2)
>>> 1.0 RDNS_DYNAMIC Delivered to internal network by host with
>>> dynamic-looking rDNS
>>> 2.1 TO_NO_BRKTS_DYNIP To: lacks brackets and dynamic rDNS
>>> 0.5 NO_FM_NAME_IP_HOSTN No From name + hostname using IP address
>>>
>>> It feels like the dynamic IP rules are killing it here - what do
>>> others do for valid dynamic IP emails inbound from a web client
>>> email through trusted hosts? Just give ALL_TRUSTED more of a
>>> boost? Or anything more scientific??
>>
>> The default rule scores obviously don't apply for your use case
>> here: dynamic RDNS is to be expected for the relayed emails you are
>> scanning.
>>
>> Also it is not an indicator that the sender is abusing a (hacked)
>> end-user host. So you should adjust the scores of the rules that
>> are not applicable for your use case:
>>
>> score RDNS_DYNAMIC 0.001
>> score HELO_DYNAMIC_IPADDR2 0.001
>>
>> Something to note: RDNS_DYNAMIC tries to exclude authenticated
>> email. Are you accepting email from senders without authentication?
>> Or maybe your trusted_networks/internal_networks are misconfigured,
>> so the authentication is not properly detected?
>>
>> Kind regards,
>> Tom
>
> Hi Tom,
>
> Thanks for the reply.
>
> Re:
>> RDNS_DYNAMIC tries to exclude authenticated email. Are you
>> accepting email from senders without authentication? Or maybe your
>> trusted_networks/internal_networks are misconfigured, so the
>> authentication is not properly detected?
>
> Example - user goes to my webmail site, logs in from his dynamic IP,
> sends an email from Chrome.
>
> The email from the big bad world is sent through https Horde at my
> external IP, and the Horde webserver (internally 'emp06' @
> 192.168.1.230) submits it to postfix on the mail server on port 587,
> authenticated only.
>
> Oct 16 16:56:36 emp07 postfix/submission/smtpd[28474]: 885383050AA4:
> client=emp06.simonandkate.lan[192.168.1.230], sasl_method=PLAIN,
> sasl_username=username
>
> Postfix pushes into amavisdnew content_filter on 127.0.0.1:10026,
> and amavisdnew correctly identifies the MYNETS policybank:
>
> Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) lookup_ip_acl
> (client_ipaddr_policy) arr.obj: key="192.168.1.230" matches
> "192.168.0.0/16", result=1
> Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) loaded policy bank
> "MYNETS" over "ORIGINATING"
>
> Amavisdnew then does a trace back up the chain to the source, and
> identifies the web client as a public IP whilst correctly ignoring
> the local IPs:
>
> Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) lookup_ip_acl
> (public_nets) arr.obj: key="127.0.0.1" matches "!127.0.0.0/8",
> result=0
> Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) lookup_ip_acl
> (public_nets) arr.obj: key="192.168.1.230" matches
> "!192.168.0.0/16", result=0
> Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) lookup_ip_acl
> (public_nets) arr.obj: key="127.0.0.1" matches "!127.0.0.0/8",
> result=0
> Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) lookup_ip_acl
> (public_nets) arr.obj: key="180.150.6.200" matches "::ffff:0:0/96",
> result=1
> Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) trace:
> LMTP://[127.0.0.1]:49288 < ESMTP://[192.168.1.230]:40266 <
> ESMTPSA://127.0.0.1 < HTTPS://180.150.6.200
>
> Amavis appears to have picked up that it is auth'ed (ESMPTA)
>
> SA is then called. SA config includes (/etc/mail/spamassassin/local.cf):
>
> trusted_networks 192.168.1. 119.18.34.29
> internal_networks !192.168.1.230 192.168.1. 119.18.34.29
> score ALL_TRUSTED -2.0
>
> By which I wanted it to see my local subnet 192.168.1.0/24 as
> internal, but see the Horde webserver (.230) as the first point
> under my control, which was how I read:
> https://cwiki.apache.org/confluence/display/spamassassin/TrustPath
> and
> https://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html
>
> But... looking at it now and your comment I think I have that
> incorrectly set and should drop the !192.168.1.230 ? Some advice
> there would be appreciated...
>
> Simon.
>
After setting trust path to be internal subnet and MX only:
trusted_networks 192.168.1. 119.18.34.29
internal_networks 192.168.1. 119.18.34.29
score ALL_TRUSTED -2.4
I got the user to resend the email. It came through OK as the revised
ALL_TRUSTED dropped it below 6, but SA is still scanning the mail from
them as if unauth'ed (full RECEIVED headers:)
X-Spam-Status: No, score=5.614 tagged_above=-999 required=6.2
tests=[ALL_TRUSTED=-2.4, BAYES_50=0.8, HELO_DYNAMIC_IPADDR2=3.607,
HTML_MESSAGE=0.001, NO_FM_NAME_IP_HOSTN=0.548, RDNS_DYNAMIC=0.982,
SPF_HELO_NONE=0.001, TO_NO_BRKTS_DYNIP=2.075]
autolearn=no autolearn_force=no
Received: from mail.simonandkate.net ([127.0.0.1])
by localhost (mail-amavis.simonandkate.net [127.0.0.1]) (amavisd-new,
port 10026)
with LMTP id eWyPoCwiqTIu for <si...@simonandkate.net>;
Wed, 16 Oct 2019 22:37:32 +1000 (AEST)
Received: from [127.0.0.1] (localhost [127.0.0.1])
by mail.simonandkate.net (Postfix) with ESMTPSA id 11E573050AA4
for <si...@simonandkate.net>; Wed, 16 Oct 2019 22:37:32 +1000 (AEST)
Received: from 180-150-6-200.b49606.syd.nbn.aussiebb.net
(180-150-6-200.b49606.syd.nbn.aussiebb.net [180.150.6.200]) by
mail.howiesue.net (Horde Framework) with HTTPS; Wed, 16 Oct 2019 22:37:23
+1000
So how do I configure it such that if it's an authenticated submission
(587) mail through my mail host at (int)192.68.1.230/(ext)119.18.34.29
further upstream RECEIVED headers are NOT scanned by SA for dynamic
IP? Am I still totally misunderstanding trust path in SA?
Simon
--
Simon Wilson
M: 0400 12 11 16
Re: Trusted network mail spam detection
Posted by Simon Wilson <si...@simonandkate.net>.
Quoting Bill Cole <sa...@billmail.scconsult.com>:
>>>>> RDNS_DYNAMIC tries to exclude authenticated email. Are you
>>>>> accepting email from senders without authentication? Or maybe
>>>>> your trusted_networks/internal_networks are misconfigured, so
>>>>> the authentication is not properly detected?
>>>>
>>>> Example - user goes to my webmail site, logs in from his dynamic
>>>> IP, sends an email from Chrome.
>>>>
>>>> The email from the big bad world is sent through https Horde at
>>>> my external IP, and the Horde webserver (internally 'emp06' @
>>>> 192.168.1.230) submits it to postfix on the mail server on port
>>>> 587, authenticated only.
>>> [...]
>>>> SA is then called. SA config includes (/etc/mail/spamassassin/local.cf):
>>>>
>>>> trusted_networks 192.168.1. 119.18.34.29
>>>> internal_networks !192.168.1.230 192.168.1. 119.18.34.29
>>>> score ALL_TRUSTED -2.0
>>>>
>>>> By which I wanted it to see my local subnet 192.168.1.0/24 as
>>>> internal, but see the Horde webserver (.230) as the first point
>>>> under my control, which was how I read:
>>>> https://cwiki.apache.org/confluence/display/spamassassin/TrustPath and
>>>> https://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html
>>>>
>>>> But... looking at it now and your comment I think I have that
>>>> incorrectly set and should drop the !192.168.1.230 ? Some advice
>>>> there would be appreciated...
>>>
>>> Drop the '!192.168.1.230' from internal_networks. Add
>>> '192.168.1.230' to msa_networks.
>>>
>>> trusted_networks is for hosts you trust to not forge (or
>>> obfuscate) Received headers or originate spam.
>>> internal_networks is for hosts you trust to not forge (or
>>> obfuscate) Received headers or originate spam, whose detectable
>>> authentication of senders you trust, but which may relay mail from
>>> unauthenticated senders.
>>> msa_networks is for hosts you trust to ONLY offer you mail from
>>> authenticated users.
>>>
>>> A different viable option (if internal_networks & msa_networks are
>>> too strong for you) is to construct compensatory meta-rules to
>>> conditionally reverse (maybe only partially) the high scores of
>>> RDNS_DYNAMIC and HELO_DYNAMIC_IPADDR2, e.g.:
>>>
>>> meta TRUSTED_USERS ( RDNS_DYNAMIC || HELO_DYNAMIC_IPADDR2 )
>>> && ALL_TRUSTED
>>> score TRUSTED_USERS -2.0
>>> describe TRUSTED_USERS Dynamic address submitters to trusted hosts.
>>>
>>>
>>
>> Thanks Bill, very informative.
>>
>> I've altered local.cf, and sent myself an externally originated
>> test message (I'll save the meta rule for later when I understand
>> what is happening).
>>
>> trusted_networks 192.168.1. 119.18.34.29
>> internal_networks 192.168.1. 119.18.34.29
>> msa_networks 192.168.1.230
>>
>> So trusted_ and internal_ both reflect my internal IP range
>> (including the internal address of the Horde webserver) and my MX /
>> external address (which is also external IP for the Horde webserver
>> - is that an issue?).
>>
>> Test result to myself:
>>
>> X-Spam-Status: No, score=2.134 tagged_above=-999 required=6.2
>> tests=[ALL_TRUSTED=-2.4, BAYES_50=0.8, DKIM_ADSP_ALL=0.8,
>> HELO_DYNAMIC_IPADDR=1.951, RDNS_DYNAMIC=0.982, SPF_HELO_NONE=0.001]
>> autolearn=no autolearn_force=no
>> Received: from mail.simonandkate.net ([127.0.0.1])
>> by localhost (mail-amavis.simonandkate.net [127.0.0.1])
>> (amavisd-new, port 10026)
>> with LMTP id Kj_HtGN_pHXs for <si...@simonmwilson.net>;
>> Thu, 17 Oct 2019 07:32:43 +1000 (AEST)
>> Received: from [127.0.0.1] (localhost [127.0.0.1])
>> by mail.simonandkate.net (Postfix) with ESMTPSA id 20C5F3050A92
>> for <si...@simonmwilson.net>; Thu, 17 Oct 2019 07:32:43 +1000 (AEST)
>> Received: from pa49-197-19-57.pa.qld.optusnet.com.au
>> (pa49-197-19-57.pa.qld.optusnet.com.au [49.197.19.57]) by
>> mail.simonandkate.net (Horde Framework) with HTTPS; Thu, 17 Oct 2019
>> 07:32:40 +1000
>
> That's helpful...
>
> I had not understood that your webmail and MTA were on the same
> machine, so that the Received headers never refer to 192.168.1.230
> at all, just to 127.0.0.1. You may find success by adding the
> loopback to msa_networks.
>
> OR maybe not. But I think so. Probably. SA is obviously trusting
> that header (because 127.0.0.1 is always in trusted_hosts unless you
> clear it) and parsing it to find the dynamic HELO & rDNS.
>
>
>> I do note that the 4th score for e.g. HELO_DYNAMIC_IPADDR is
>> triggering (1.951) not the 3rd which triggered on the other email I
>> was using to test (I've asked them to resend as well), which from
>> doc tells me that this means Bayes and network tests are triggering.
>>
>> Adding the MSA has not removed the tests running - but is the
>> different score parameter a result of the change?
>
> The previous message triggered HELO_DYNAMIC_IPADDR2 not
> HELO_DYNAMIC_IPADDR, according to the report that you quoted in your
> first message. That is what accounts for the different scores.
>
>
> --
> Bill Cole
Hi Bill, thanks for the help and support :)
Externally I have one public IP address, which is both MX for my
domain and also host for the Horde webserver. So externally - yes, MTA
and webhost are "the same". Internally, they are separate: .230 for
the web host, .235 for the MTA.
Internally the MTA's hostname resolves to a local IP, not the MX:
# ping mail.simonandkate.net
PING emp06.simonandkate.lan (192.168.1.230)
BUT!! Your comments got me thinking, and you know how a lightbulb goes
on sometimes? I had a header rewrite that was put in place a long time
ago, that was taking OUT the 192. internal addressing info. Old times,
pretty pointless security through obfuscation... along the lines of
this:
http://realtechtalk.com/Postfix_how_to_secure_outgoing_authenticated_emails_for_privacy_and_hide_the_IP_address_mailer_and_other_things-1573-articles
I took that out, and tried again. Now the headers:
X-Spam-Status: No, score=-0.799 tagged_above=-999 required=6.2
tests=[ALL_TRUSTED=-2.4, BAYES_50=0.8, DKIM_ADSP_ALL=0.8,
HTML_MESSAGE=0.001] autolearn=no autolearn_force=no
Received: from mail.simonandkate.net ([127.0.0.1])
by localhost (mail-amavis.simonandkate.net [127.0.0.1]) (amavisd-new,
port 10026)
with LMTP id bhg52kh3MlO4 for <si...@simonmwilson.net>;
Thu, 17 Oct 2019 17:11:32 +1000 (AEST)
Received: from emp06.simonandkate.lan (emp06.simonandkate.lan [192.168.1.230])
by mail.simonandkate.net (Postfix) with ESMTPSA id AFE373050A92
for <si...@simonmwilson.net>; Thu, 17 Oct 2019 17:11:32 +1000 (AEST)
The "first" public IP 'Received: from' entry has been removed entirely
(so is that triggered by SA now correctly detecting trust?), the MSA
shows correctly as the first source, and the RDNS sigs etc. are not
being triggered. Bingo... I think that is now working like it is
supposed to. Thank you very much... (y)
Is there *any* point in rewriting / anonymising headers? As it is now
configured I just have it removing User Agent, but that seems pretty
pointless. What is being left after SA and amavisd finish processing
the authenticated outbound email seems pretty harmless...
I now have the following in local.cf and it seems to be working OK,
but I'll keep an eye on it.
trusted_networks 192.168.1. 119.18.34.29
internal_networks 192.168.1. 119.18.34.29
msa_networks 192.168.1.230
score ALL_TRUSTED -1.4
Thanks again.
--
Simon Wilson
M: 0400 12 11 16
Re: Trusted network mail spam detection
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 16 Oct 2019, at 18:11, Simon Wilson wrote:
> Quoting Bill Cole <sa...@billmail.scconsult.com>:
>
>> On 16 Oct 2019, at 7:55, Simon Wilson wrote:
>>
>>> Quoting Tom Hendrikx <to...@whyscream.net>:
>> [...]
>>>> RDNS_DYNAMIC tries to exclude authenticated email. Are you
>>>> accepting email from senders without authentication? Or maybe your
>>>> trusted_networks/internal_networks are misconfigured, so the
>>>> authentication is not properly detected?
>>>
>>> Example - user goes to my webmail site, logs in from his dynamic IP,
>>> sends an email from Chrome.
>>>
>>> The email from the big bad world is sent through https Horde at my
>>> external IP, and the Horde webserver (internally 'emp06' @
>>> 192.168.1.230) submits it to postfix on the mail server on port 587,
>>> authenticated only.
>> [...]
>>> SA is then called. SA config includes
>>> (/etc/mail/spamassassin/local.cf):
>>>
>>> trusted_networks 192.168.1. 119.18.34.29
>>> internal_networks !192.168.1.230 192.168.1. 119.18.34.29
>>> score ALL_TRUSTED -2.0
>>>
>>> By which I wanted it to see my local subnet 192.168.1.0/24 as
>>> internal, but see the Horde webserver (.230) as the first point
>>> under my control, which was how I read:
>>> https://cwiki.apache.org/confluence/display/spamassassin/TrustPath
>>> and
>>> https://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html
>>>
>>> But... looking at it now and your comment I think I have that
>>> incorrectly set and should drop the !192.168.1.230 ? Some advice
>>> there would be appreciated...
>>
>> Drop the '!192.168.1.230' from internal_networks. Add '192.168.1.230'
>> to msa_networks.
>>
>> trusted_networks is for hosts you trust to not forge (or obfuscate)
>> Received headers or originate spam.
>> internal_networks is for hosts you trust to not forge (or obfuscate)
>> Received headers or originate spam, whose detectable authentication
>> of senders you trust, but which may relay mail from unauthenticated
>> senders.
>> msa_networks is for hosts you trust to ONLY offer you mail from
>> authenticated users.
>>
>> A different viable option (if internal_networks & msa_networks are
>> too strong for you) is to construct compensatory meta-rules to
>> conditionally reverse (maybe only partially) the high scores of
>> RDNS_DYNAMIC and HELO_DYNAMIC_IPADDR2, e.g.:
>>
>> meta TRUSTED_USERS ( RDNS_DYNAMIC || HELO_DYNAMIC_IPADDR2 ) &&
>> ALL_TRUSTED
>> score TRUSTED_USERS -2.0
>> describe TRUSTED_USERS Dynamic address submitters to trusted hosts.
>>
>>
>
> Thanks Bill, very informative.
>
> I've altered local.cf, and sent myself an externally originated test
> message (I'll save the meta rule for later when I understand what is
> happening).
>
> trusted_networks 192.168.1. 119.18.34.29
> internal_networks 192.168.1. 119.18.34.29
> msa_networks 192.168.1.230
>
> So trusted_ and internal_ both reflect my internal IP range (including
> the internal address of the Horde webserver) and my MX / external
> address (which is also external IP for the Horde webserver - is that
> an issue?).
>
> Test result to myself:
>
> X-Spam-Status: No, score=2.134 tagged_above=-999 required=6.2
> tests=[ALL_TRUSTED=-2.4, BAYES_50=0.8, DKIM_ADSP_ALL=0.8,
> HELO_DYNAMIC_IPADDR=1.951, RDNS_DYNAMIC=0.982, SPF_HELO_NONE=0.001]
> autolearn=no autolearn_force=no
> Received: from mail.simonandkate.net ([127.0.0.1])
> by localhost (mail-amavis.simonandkate.net [127.0.0.1]) (amavisd-new,
> port 10026)
> with LMTP id Kj_HtGN_pHXs for <si...@simonmwilson.net>;
> Thu, 17 Oct 2019 07:32:43 +1000 (AEST)
> Received: from [127.0.0.1] (localhost [127.0.0.1])
> by mail.simonandkate.net (Postfix) with ESMTPSA id 20C5F3050A92
> for <si...@simonmwilson.net>; Thu, 17 Oct 2019 07:32:43 +1000 (AEST)
> Received: from pa49-197-19-57.pa.qld.optusnet.com.au
> (pa49-197-19-57.pa.qld.optusnet.com.au [49.197.19.57]) by
> mail.simonandkate.net (Horde Framework) with HTTPS; Thu, 17 Oct 2019
> 07:32:40 +1000
That's helpful...
I had not understood that your webmail and MTA were on the same machine,
so that the Received headers never refer to 192.168.1.230 at all, just
to 127.0.0.1. You may find success by adding the loopback to
msa_networks.
OR maybe not. But I think so. Probably. SA is obviously trusting that
header (because 127.0.0.1 is always in trusted_hosts unless you clear
it) and parsing it to find the dynamic HELO & rDNS.
> I do note that the 4th score for e.g. HELO_DYNAMIC_IPADDR is
> triggering (1.951) not the 3rd which triggered on the other email I
> was using to test (I've asked them to resend as well), which from doc
> tells me that this means Bayes and network tests are triggering.
>
> Adding the MSA has not removed the tests running - but is the
> different score parameter a result of the change?
The previous message triggered HELO_DYNAMIC_IPADDR2 not
HELO_DYNAMIC_IPADDR, according to the report that you quoted in your
first message. That is what accounts for the different scores.
--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Not Currently Available For Hire
Re: Trusted network mail spam detection
Posted by Simon Wilson <si...@simonandkate.net>.
Quoting Bill Cole <sa...@billmail.scconsult.com>:
> On 16 Oct 2019, at 7:55, Simon Wilson wrote:
>
>> Quoting Tom Hendrikx <to...@whyscream.net>:
> [...]
>>> RDNS_DYNAMIC tries to exclude authenticated email. Are you
>>> accepting email from senders without authentication? Or maybe your
>>> trusted_networks/internal_networks are misconfigured, so the
>>> authentication is not properly detected?
>>
>> Example - user goes to my webmail site, logs in from his dynamic
>> IP, sends an email from Chrome.
>>
>> The email from the big bad world is sent through https Horde at my
>> external IP, and the Horde webserver (internally 'emp06' @
>> 192.168.1.230) submits it to postfix on the mail server on port
>> 587, authenticated only.
> [...]
>> SA is then called. SA config includes (/etc/mail/spamassassin/local.cf):
>>
>> trusted_networks 192.168.1. 119.18.34.29
>> internal_networks !192.168.1.230 192.168.1. 119.18.34.29
>> score ALL_TRUSTED -2.0
>>
>> By which I wanted it to see my local subnet 192.168.1.0/24 as
>> internal, but see the Horde webserver (.230) as the first point
>> under my control, which was how I read:
>> https://cwiki.apache.org/confluence/display/spamassassin/TrustPath
>> and
>> https://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html
>>
>> But... looking at it now and your comment I think I have that
>> incorrectly set and should drop the !192.168.1.230 ? Some advice
>> there would be appreciated...
>
> Drop the '!192.168.1.230' from internal_networks. Add
> '192.168.1.230' to msa_networks.
>
> trusted_networks is for hosts you trust to not forge (or obfuscate)
> Received headers or originate spam.
> internal_networks is for hosts you trust to not forge (or obfuscate)
> Received headers or originate spam, whose detectable authentication
> of senders you trust, but which may relay mail from unauthenticated
> senders.
> msa_networks is for hosts you trust to ONLY offer you mail from
> authenticated users.
>
> A different viable option (if internal_networks & msa_networks are
> too strong for you) is to construct compensatory meta-rules to
> conditionally reverse (maybe only partially) the high scores of
> RDNS_DYNAMIC and HELO_DYNAMIC_IPADDR2, e.g.:
>
> meta TRUSTED_USERS ( RDNS_DYNAMIC || HELO_DYNAMIC_IPADDR2 ) &&
> ALL_TRUSTED
> score TRUSTED_USERS -2.0
> describe TRUSTED_USERS Dynamic address submitters to trusted hosts.
>
>
Thanks Bill, very informative.
I've altered local.cf, and sent myself an externally originated test
message (I'll save the meta rule for later when I understand what is
happening).
trusted_networks 192.168.1. 119.18.34.29
internal_networks 192.168.1. 119.18.34.29
msa_networks 192.168.1.230
So trusted_ and internal_ both reflect my internal IP range (including
the internal address of the Horde webserver) and my MX / external
address (which is also external IP for the Horde webserver - is that
an issue?).
Test result to myself:
X-Spam-Status: No, score=2.134 tagged_above=-999 required=6.2
tests=[ALL_TRUSTED=-2.4, BAYES_50=0.8, DKIM_ADSP_ALL=0.8,
HELO_DYNAMIC_IPADDR=1.951, RDNS_DYNAMIC=0.982, SPF_HELO_NONE=0.001]
autolearn=no autolearn_force=no
Received: from mail.simonandkate.net ([127.0.0.1])
by localhost (mail-amavis.simonandkate.net [127.0.0.1]) (amavisd-new,
port 10026)
with LMTP id Kj_HtGN_pHXs for <si...@simonmwilson.net>;
Thu, 17 Oct 2019 07:32:43 +1000 (AEST)
Received: from [127.0.0.1] (localhost [127.0.0.1])
by mail.simonandkate.net (Postfix) with ESMTPSA id 20C5F3050A92
for <si...@simonmwilson.net>; Thu, 17 Oct 2019 07:32:43 +1000 (AEST)
Received: from pa49-197-19-57.pa.qld.optusnet.com.au
(pa49-197-19-57.pa.qld.optusnet.com.au [49.197.19.57]) by
mail.simonandkate.net (Horde Framework) with HTTPS; Thu, 17 Oct 2019
07:32:40 +1000
I do note that the 4th score for e.g. HELO_DYNAMIC_IPADDR is
triggering (1.951) not the 3rd which triggered on the other email I
was using to test (I've asked them to resend as well), which from doc
tells me that this means Bayes and network tests are triggering.
Adding the MSA has not removed the tests running - but is the
different score parameter a result of the change?
Simon
--
Simon Wilson
M: 0400 12 11 16
Re: Trusted network mail spam detection
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 16 Oct 2019, at 7:55, Simon Wilson wrote:
> Quoting Tom Hendrikx <to...@whyscream.net>:
[...]
>> RDNS_DYNAMIC tries to exclude authenticated email. Are you accepting
>> email from senders without authentication? Or maybe your
>> trusted_networks/internal_networks are misconfigured, so the
>> authentication is not properly detected?
>
> Example - user goes to my webmail site, logs in from his dynamic IP,
> sends an email from Chrome.
>
> The email from the big bad world is sent through https Horde at my
> external IP, and the Horde webserver (internally 'emp06' @
> 192.168.1.230) submits it to postfix on the mail server on port 587,
> authenticated only.
[...]
> SA is then called. SA config includes
> (/etc/mail/spamassassin/local.cf):
>
> trusted_networks 192.168.1. 119.18.34.29
> internal_networks !192.168.1.230 192.168.1. 119.18.34.29
> score ALL_TRUSTED -2.0
>
> By which I wanted it to see my local subnet 192.168.1.0/24 as
> internal, but see the Horde webserver (.230) as the first point under
> my control, which was how I read:
> https://cwiki.apache.org/confluence/display/spamassassin/TrustPath and
> https://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html
>
> But... looking at it now and your comment I think I have that
> incorrectly set and should drop the !192.168.1.230 ? Some advice there
> would be appreciated...
Drop the '!192.168.1.230' from internal_networks. Add '192.168.1.230' to
msa_networks.
trusted_networks is for hosts you trust to not forge (or obfuscate)
Received headers or originate spam.
internal_networks is for hosts you trust to not forge (or obfuscate)
Received headers or originate spam, whose detectable authentication of
senders you trust, but which may relay mail from unauthenticated
senders.
msa_networks is for hosts you trust to ONLY offer you mail from
authenticated users.
A different viable option (if internal_networks & msa_networks are too
strong for you) is to construct compensatory meta-rules to conditionally
reverse (maybe only partially) the high scores of RDNS_DYNAMIC and
HELO_DYNAMIC_IPADDR2, e.g.:
meta TRUSTED_USERS ( RDNS_DYNAMIC || HELO_DYNAMIC_IPADDR2 ) &&
ALL_TRUSTED
score TRUSTED_USERS -2.0
describe TRUSTED_USERS Dynamic address submitters to trusted hosts.
--
Bill Cole
bill@scconsult.com or billcole@apache.org
(AKA @grumpybozo and many *@billmail.scconsult.com addresses)
Re: Trusted network mail spam detection
Posted by Simon Wilson <si...@simonandkate.net>.
Quoting Tom Hendrikx <to...@whyscream.net>:
> On 16-10-19 12:19, Simon Wilson wrote:
>> Hi, I have a Horde system submitting to a
>> postfix/amavisd-new/spamassassin server for spam detection
>> (different servers, same subnet). I *do* consciously run SA over
>> internally submitted emails to catch compromised accounts (it
>> happened once to me when a family member's email password was
>> compromised and a bunch of spam got sent out).
>>
>> I'm having occasional issues with mail sent by some users from
>> their home ISP connections (i.e. Chrome client on ISP dynamic IP ->
>> Horde server/postfix etc). Email validly sent through the trusted
>> host Horde server gets a bonus (ALL_TRUSTED = -2) which SA is
>> triggering fine when appropriate, but some emails are still
>> triggering thresholds, so I was wondering what others do for
>> configuring for traffic that is *mostly* trusted but should still
>> be checked for obvious spam?
>>
>> This is not a new system, it's well trained with thousands of ham
>> and spam over several years. This email was genuine ham, and was
>> discarded (Amavis threshold 6.0 -> discard).
>>
>> Content analysis details: (6.0 points, 6.2 required)
>>
>> pts rule name description
>> ---- ----------------------
>> --------------------------------------------------
>> -2.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
>> 0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60%
>> [score: 0.5000]
>> 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
>> 0.0 HTML_MESSAGE BODY: HTML included in message
>> 3.6 HELO_DYNAMIC_IPADDR2 Relay HELO'd using suspicious hostname (IP addr
>> 2)
>> 1.0 RDNS_DYNAMIC Delivered to internal network by host with
>> dynamic-looking rDNS
>> 2.1 TO_NO_BRKTS_DYNIP To: lacks brackets and dynamic rDNS
>> 0.5 NO_FM_NAME_IP_HOSTN No From name + hostname using IP address
>>
>> It feels like the dynamic IP rules are killing it here - what do
>> others do for valid dynamic IP emails inbound from a web client
>> email through trusted hosts? Just give ALL_TRUSTED more of a boost?
>> Or anything more scientific??
>
> The default rule scores obviously don't apply for your use case
> here: dynamic RDNS is to be expected for the relayed emails you are
> scanning.
>
> Also it is not an indicator that the sender is abusing a (hacked)
> end-user host. So you should adjust the scores of the rules that are
> not applicable for your use case:
>
> score RDNS_DYNAMIC 0.001
> score HELO_DYNAMIC_IPADDR2 0.001
>
> Something to note: RDNS_DYNAMIC tries to exclude authenticated
> email. Are you accepting email from senders without authentication?
> Or maybe your trusted_networks/internal_networks are misconfigured,
> so the authentication is not properly detected?
>
> Kind regards,
> Tom
Hi Tom,
Thanks for the reply.
Re:
> RDNS_DYNAMIC tries to exclude authenticated email. Are you accepting
> email from senders without authentication? Or maybe your
> trusted_networks/internal_networks are misconfigured, so the
> authentication is not properly detected?
Example - user goes to my webmail site, logs in from his dynamic IP,
sends an email from Chrome.
The email from the big bad world is sent through https Horde at my
external IP, and the Horde webserver (internally 'emp06' @
192.168.1.230) submits it to postfix on the mail server on port 587,
authenticated only.
Oct 16 16:56:36 emp07 postfix/submission/smtpd[28474]: 885383050AA4:
client=emp06.simonandkate.lan[192.168.1.230], sasl_method=PLAIN,
sasl_username=username
Postfix pushes into amavisdnew content_filter on 127.0.0.1:10026, and
amavisdnew correctly identifies the MYNETS policybank:
Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) lookup_ip_acl
(client_ipaddr_policy) arr.obj: key="192.168.1.230" matches
"192.168.0.0/16", result=1
Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) loaded policy bank
"MYNETS" over "ORIGINATING"
Amavisdnew then does a trace back up the chain to the source, and
identifies the web client as a public IP whilst correctly ignoring the
local IPs:
Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) lookup_ip_acl
(public_nets) arr.obj: key="127.0.0.1" matches "!127.0.0.0/8", result=0
Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) lookup_ip_acl
(public_nets) arr.obj: key="192.168.1.230" matches "!192.168.0.0/16",
result=0
Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) lookup_ip_acl
(public_nets) arr.obj: key="127.0.0.1" matches "!127.0.0.0/8", result=0
Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) lookup_ip_acl
(public_nets) arr.obj: key="180.150.6.200" matches "::ffff:0:0/96",
result=1
Oct 16 16:56:36 emp07 amavis[26639]: (26639-05) trace:
LMTP://[127.0.0.1]:49288 < ESMTP://[192.168.1.230]:40266 <
ESMTPSA://127.0.0.1 < HTTPS://180.150.6.200
Amavis appears to have picked up that it is auth'ed (ESMPTA)
SA is then called. SA config includes (/etc/mail/spamassassin/local.cf):
trusted_networks 192.168.1. 119.18.34.29
internal_networks !192.168.1.230 192.168.1. 119.18.34.29
score ALL_TRUSTED -2.0
By which I wanted it to see my local subnet 192.168.1.0/24 as
internal, but see the Horde webserver (.230) as the first point under
my control, which was how I read:
https://cwiki.apache.org/confluence/display/spamassassin/TrustPath and
https://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Conf.html
But... looking at it now and your comment I think I have that
incorrectly set and should drop the !192.168.1.230 ? Some advice there
would be appreciated...
Simon.
--
Simon Wilson
M: 0400 12 11 16
Re: Trusted network mail spam detection
Posted by Tom Hendrikx <to...@whyscream.net>.
On 16-10-19 12:19, Simon Wilson wrote:
> Hi, I have a Horde system submitting to a
> postfix/amavisd-new/spamassassin server for spam detection (different
> servers, same subnet). I *do* consciously run SA over internally
> submitted emails to catch compromised accounts (it happened once to me
> when a family member's email password was compromised and a bunch of
> spam got sent out).
>
> I'm having occasional issues with mail sent by some users from their
> home ISP connections (i.e. Chrome client on ISP dynamic IP -> Horde
> server/postfix etc). Email validly sent through the trusted host Horde
> server gets a bonus (ALL_TRUSTED = -2) which SA is triggering fine when
> appropriate, but some emails are still triggering thresholds, so I was
> wondering what others do for configuring for traffic that is *mostly*
> trusted but should still be checked for obvious spam?
>
> This is not a new system, it's well trained with thousands of ham and
> spam over several years. This email was genuine ham, and was discarded
> (Amavis threshold 6.0 -> discard).
>
> Content analysis details: (6.0 points, 6.2 required)
>
> pts rule name description
> ---- ----------------------
> --------------------------------------------------
> -2.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
> 0.8 BAYES_50 BODY: Bayes spam probability is 40 to 60%
> [score: 0.5000]
> 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
> 0.0 HTML_MESSAGE BODY: HTML included in message
> 3.6 HELO_DYNAMIC_IPADDR2 Relay HELO'd using suspicious hostname (IP
> addr
> 2)
> 1.0 RDNS_DYNAMIC Delivered to internal network by host with
> dynamic-looking rDNS
> 2.1 TO_NO_BRKTS_DYNIP To: lacks brackets and dynamic rDNS
> 0.5 NO_FM_NAME_IP_HOSTN No From name + hostname using IP address
>
> It feels like the dynamic IP rules are killing it here - what do others
> do for valid dynamic IP emails inbound from a web client email through
> trusted hosts? Just give ALL_TRUSTED more of a boost? Or anything more
> scientific??
The default rule scores obviously don't apply for your use case here:
dynamic RDNS is to be expected for the relayed emails you are scanning.
Also it is not an indicator that the sender is abusing a (hacked)
end-user host. So you should adjust the scores of the rules that are not
applicable for your use case:
score RDNS_DYNAMIC 0.001
score HELO_DYNAMIC_IPADDR2 0.001
Something to note: RDNS_DYNAMIC tries to exclude authenticated email.
Are you accepting email from senders without authentication? Or maybe
your trusted_networks/internal_networks are misconfigured, so the
authentication is not properly detected?
Kind regards,
Tom