You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@spamassassin.apache.org by Matus UHLAR - fantomas <uh...@fantomas.sk> on 2018/08/30 13:14:39 UTC
__HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Hello,
the __HDR_ORDER_FTSDMCXXXX rule catches mail sent from windows live mail
(and outlook express, which, while obsolete, seems to be still used often)
That further causes hitting HDR_ORDER_FTSDMCXX_DIRECT and
HDR_ORDER_FTSDMCXX_NORDNS in cases where client uses the mail client on
local network, without SMTP authentication, and without DNS (which may be
quite common in some organizations).
as a workaround, I recommend to add && !ALL_TRUSTED to
HDR_ORDER_FTSDMCXX_DIRECT and HDR_ORDER_FTSDMCXX_NORDNS rules.
an example:
X-Spam-Status: Yes, score=9.154 required=5.6 tests=[ALL_TRUSTED=-1,
DOS_OE_TO_MX=3.086, FSL_HELO_NON_FQDN_1=0.001,
HDR_ORDER_FTSDMCXX_DIRECT=1.999, HDR_ORDER_FTSDMCXX_NORDNS=3.5,
HTML_MESSAGE=0.001, MIMEOLE_DIRECT_TO_MX=0.293, RDNS_NONE=1.274]
autolearn=no autolearn_force=no
X-Mailer: Microsoft Outlook Express 6.00.2900.5931
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157
I have filled out bug 7607, it got rejected immediately:
https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7607
while I agree that fixing RDNS will help, internal networks DNS is not
always easy, especially when maintained by different people and when
internal DNS is in LAn, not in DMZ.
note that this problem has been reported on spamassassin-users a month ago:
http://spamassassin.1065346.n5.nabble.com/Problem-with-new-rules-td152105.html
So, to avoid discussions on bugzilla, I prefer asking here:
Is it really a problem to add && !ALL_TRUSTED to HDR_ORDER_FTSDMCXX_DIRECT
and HDR_ORDER_FTSDMCXX_NORDNS ?
(maybe even HDR_ORDER_FTSDMCXX_001C and HDR_ORDER_FTSDMCXX_BAT, if their
score will be more than zero)
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
99 percent of lawyers give the rest a bad name.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by John Hardin <jh...@impsec.org>.
On Thu, 30 Aug 2018, Matus UHLAR - fantomas wrote:
> Is it really a problem to add && !ALL_TRUSTED to HDR_ORDER_FTSDMCXX_DIRECT
> and HDR_ORDER_FTSDMCXX_NORDNS ?
As I said on the bug: I'll review masscheck and add that if it appears
reasonable to do so.
> (maybe even HDR_ORDER_FTSDMCXX_001C and HDR_ORDER_FTSDMCXX_BAT, if their
> score will be more than zero)
>
> --
> Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
> Warning: I wish NOT to receive e-mail advertising to this address.
> Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
> 99 percent of lawyers give the rest a bad name.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Joan Peterson is like that: you expect at least a pseudological
argument, but instead you get the weird ramblings of a woman with
the critical thinking abilities of an 18th century peasant. -- Ken
-----------------------------------------------------------------------
518 days since the first commercial re-flight of an orbital booster (SpaceX)
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>On Thu, 30 Aug 2018 12:16:33 -0400
>Bill Cole wrote:
>> I think the fix for all is for everyone to get their
>> internal_networks and trusted_networks configurations correct.
On 30.08.18 20:35, RW wrote:
>Whatever is happening in this particular case (whatever that is), any
>rule that works on last-external should distinguish between trusted and
>untrusted.
>
>Tests that use __DOS_SINGLE_EXT_RELAY should be looking for a single
>untrusted and external relay.
>'&& ! ALL_TRUSTED' is one way of doing this for __DOS_SINGLE_EXT_RELAY,
>but unfortunately ALL_TRUSTED is a bit fragile because there's a check
>for unparsable relays in the perl.
__DOS_SINGLE_EXT_RELAY would work in my case (client sending direclty to
mailserver).
But when considering multiple trusted server (client, trusted and internal
MTA, my MTA), it would hit again.
I will have to think of this again...
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Nothing is fool-proof to a talented fool.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by RW <rw...@googlemail.com>.
On Thu, 30 Aug 2018 12:16:33 -0400
Bill Cole wrote:
> I think the fix for all is for everyone to get their
> internal_networks and trusted_networks configurations correct.
Whatever is happening in this particular case (whatever that is), any
rule that works on last-external should distinguish between trusted and
untrusted.
Tests that use __DOS_SINGLE_EXT_RELAY should be looking for a single
untrusted and external relay.
'&& ! ALL_TRUSTED' is one way of doing this for __DOS_SINGLE_EXT_RELAY,
but unfortunately ALL_TRUSTED is a bit fragile because there's a check
for unparsable relays in the perl.
Probably the best general test for whether the last-external is
untrusted is to count the number of '[' characters in
X-Spam-Relays-Untrusted and X-Spam-Relays-External and compare the
counts.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 31 Aug 2018, at 4:05, Matus UHLAR - fantomas wrote:
>>> On 08/30/2018 10:16 AM, Bill Cole wrote:
>>>> It's hard to understand this circumstance based on the generic
>>>> description.
>>>>
>>>> It appears that you have a configuration where a relay is in
>>>> trusted_networks (i.e. you believe what it asserts in Received
>>>> headers)
>>>> but it is NOT in internal_networks so it is in the synthetic
>>>> X-Spam-Relays-External pseudo-header, it is the only element in
>>>> X-Spam-Relays-External so the message
>>>> matches__DOS_SINGLE_EXT_RELAY, and
>>>> it has no rDNS so the message matches __RDNS_NONE.
>>>>
>>>> So: why is that nameless machine that you cannot make a named
>>>> machine NOT in internal_networks?
>
> multiple client PCs in the local network.
>
> and as client PCs, I don't want to put them into internal_networks.
> (And if I remember correctly, I should not).
This is a great example of why it is always helpful to have actual (or
carefully constructed) samples of mail and of how that mail is analyzed
by SA in order to solve a classification problem. I still don't have a
solid understanding of how this mail is flowing and what sort of trust
you have in the behavior of the specific machines involved in generating
and/or transporting the mislabeled email, so I can't say for sure how
you should classify those client PCs.
As I said in my earlier message today, I think you have a circumstance
that can't be forced into how SA classifies hosts.
>
>> On 30 Aug 2018, at 12:40, Grant Taylor wrote:
>>> I don't know if this is the OP's case or not, but the following
>>> example
>>> comes to mind.
>>>
>>> SA (running on your receiving MTA) receives a message from an MTA
>>> (which
>>> is itself an MSA) of an external Business-to-Business partner (thus
>>> a
>>> trusted MTA that is not internal to the recipient's organization)
>>> which
>>> itself received the message from a client on an RFC 1918 network
>>> without
>>> reverse DNS.
>
> On 30.08.18 15:08, Bill Cole wrote:
>> If that MSA is requiring authentication (as it should) and recording
>> that
>> in the Received header (as it should) then as I understand it, the
>> handoff
>> of the message will not be considered for __RDNS_NONE.
>
> Authentication not implemented yet, and telling the network admins
> they must
> to implement it now that I have installed spamassassin, is not
> acceptable.
>
> Tuning DNS is of course possible but it requires some time.
Yes. My response to Grant was solely in regards to his hypothetical.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>> On 08/30/2018 10:16 AM, Bill Cole wrote:
>>> It's hard to understand this circumstance based on the generic description.
>>>
>>> It appears that you have a configuration where a relay is in
>>> trusted_networks (i.e. you believe what it asserts in Received headers)
>>> but it is NOT in internal_networks so it is in the synthetic
>>> X-Spam-Relays-External pseudo-header, it is the only element in
>>> X-Spam-Relays-External so the message matches__DOS_SINGLE_EXT_RELAY, and
>>> it has no rDNS so the message matches __RDNS_NONE.
>>>
>>> So: why is that nameless machine that you cannot make a named machine NOT in internal_networks?
multiple client PCs in the local network.
and as client PCs, I don't want to put them into internal_networks.
(And if I remember correctly, I should not).
>On 30 Aug 2018, at 12:40, Grant Taylor wrote:
>> I don't know if this is the OP's case or not, but the following example
>> comes to mind.
>>
>> SA (running on your receiving MTA) receives a message from an MTA (which
>> is itself an MSA) of an external Business-to-Business partner (thus a
>> trusted MTA that is not internal to the recipient's organization) which
>> itself received the message from a client on an RFC 1918 network without
>> reverse DNS.
On 30.08.18 15:08, Bill Cole wrote:
>If that MSA is requiring authentication (as it should) and recording that
> in the Received header (as it should) then as I understand it, the handoff
> of the message will not be considered for __RDNS_NONE.
Authentication not implemented yet, and telling the network admins they must
to implement it now that I have installed spamassassin, is not acceptable.
Tuning DNS is of course possible but it requires some time.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I wonder how much deeper the ocean would be without sponges.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Pedro David Marco <pe...@yahoo.com>.
I have not checked whether SA orders the Received headers in correct date/time sequence but if not, remember that many Microsoft products change whimsically the order of Receiveds headers (Kevin sharks are starving)...
----PedroD
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 30 Aug 2018, at 18:02, Grant Taylor wrote:
> On 08/30/2018 03:50 PM, Bill Cole wrote:
>> That will depend on how that particular MTA constructs its Received headers in relation to the parsing in Mail::SpamAssassin::Message::Metadata::Received, which is non-trivial to describe in human language.
>
> Fair enough.
>
> Would it be possible for this scenario to present with the symptoms that the OP described?
I don't think so, given the description, but maybe.
Mail::SpamAssassin::Message::Metadata::Received implements a baroque ad hoc parsing mechanism that has been adapted organically for most of 2 decades and which "knows" many special cases where a particular Received header pattern indicates a trusted hand-off.
My understanding is that the client lacking rDNS in this case is talking directly to the SA host, which is a simpler case.
> Thank you for humoring me as I try to learn.
No problem. I make no claim to knowing absolutely everything about how the *_networks and Received header parsing behaves, or even to know better than anyone else in particular, but I've fought with it a bunch...
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Grant Taylor <gt...@tnetconsulting.net>.
On 08/30/2018 03:50 PM, Bill Cole wrote:
> That will depend on how that particular MTA constructs
> its Received headers in relation to the parsing in
> Mail::SpamAssassin::Message::Metadata::Received, which is non-trivial
> to describe in human language.
Fair enough.
Would it be possible for this scenario to present with the symptoms that
the OP described?
Thank you for humoring me as I try to learn.
--
Grant. . . .
unix || die
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 30 Aug 2018, at 15:56, Grant Taylor wrote:
> On 08/30/2018 01:08 PM, Bill Cole wrote:
>> If that MSA is requiring authentication (as it should) and recording that in the Received header (as it should) then as I understand it, the handoff of the message will not be considered for __RDNS_NONE.
>
> Okay.
>
> What happens if the MSA isn't using authentication and instead is configured to blindly allow relaying from the local / internal / private LAN. As is / was traditional for a long time for ISPs to allow relaying from their (client) IP address space. (Granted, this is against best practices.)
>
> How would this type of scenario effect your statement above?
That will depend on how that particular MTA constructs its Received headers in relation to the parsing in Mail::SpamAssassin::Message::Metadata::Received, which is non-trivial to describe in human language.
>> OK, but in that case the MTA would use an IP that should be in trusted_networks and have rDNS.
>
> Agreed.
>
>> The partner machine's IP should be in trusted_networks AND should have rDNS as an explicit technical requirement of the cooperation, which is entirely reasonable.
>
> Okay.
>
>
>
> --
> Grant. . . .
> unix || die
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Grant Taylor <gt...@tnetconsulting.net>.
On 08/30/2018 01:08 PM, Bill Cole wrote:
> If that MSA is requiring authentication (as it should) and recording
> that in the Received header (as it should) then as I understand it,
> the handoff of the message will not be considered for __RDNS_NONE.
Okay.
What happens if the MSA isn't using authentication and instead is
configured to blindly allow relaying from the local / internal / private
LAN. As is / was traditional for a long time for ISPs to allow relaying
from their (client) IP address space. (Granted, this is against best
practices.)
How would this type of scenario effect your statement above?
> OK, but in that case the MTA would use an IP that should be in
> trusted_networks and have rDNS.
Agreed.
> The partner machine's IP should be in trusted_networks AND should have
> rDNS as an explicit technical requirement of the cooperation, which is
> entirely reasonable.
Okay.
--
Grant. . . .
unix || die
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 30 Aug 2018, at 12:40, Grant Taylor wrote:
> On 08/30/2018 10:16 AM, Bill Cole wrote:
>> It's hard to understand this circumstance based on the generic description.
>>
>> It appears that you have a configuration where a relay is in trusted_networks (i.e. you believe what it asserts in Received headers) but it is NOT in internal_networks so it is in the synthetic X-Spam-Relays-External pseudo-header, it is the only element in X-Spam-Relays-External so the message matches__DOS_SINGLE_EXT_RELAY, and it has no rDNS so the message matches __RDNS_NONE.
>>
>> So: why is that nameless machine that you cannot make a named machine NOT in internal_networks?
>
> I don't know if this is the OP's case or not, but the following example comes to mind.
>
> SA (running on your receiving MTA) receives a message from an MTA (which is itself an MSA) of an external Business-to-Business partner (thus a trusted MTA that is not internal to the recipient's organization) which itself received the message from a client on an RFC 1918 network without reverse DNS.
If that MSA is requiring authentication (as it should) and recording that in the Received header (as it should) then as I understand it, the handoff of the message will not be considered for __RDNS_NONE.
>> Of course not, but if a machine is trusted to tell the truth in Received headers and has no rDNS because it is talking to a close affiliate on a RFC1918 IP, in what sense is it not internal?
>
> Trusting a B2B partner's external MTA.
OK, but in that case the MTA would use an IP that should be in trusted_networks and have rDNS.
>> Or is it in internal_networks but there's something wrong in how SA is parsing Received headers to build X-Spam-Relays-External?
>>
>>
>> I think the fix for all is for everyone to get their internal_networks and trusted_networks configurations correct.
>
> What should trusted_networks and internal_networks be set to in the B2B scenario I'm describing?
The partner machine's IP should be in trusted_networks AND should have rDNS as an explicit technical requirement of the cooperation, which is entirely reasonable.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Grant Taylor <gt...@tnetconsulting.net>.
On 08/30/2018 10:16 AM, Bill Cole wrote:
> It's hard to understand this circumstance based on the generic description.
>
> It appears that you have a configuration where a relay is in
> trusted_networks (i.e. you believe what it asserts in Received headers)
> but it is NOT in internal_networks so it is in the synthetic
> X-Spam-Relays-External pseudo-header, it is the only element in
> X-Spam-Relays-External so the message matches__DOS_SINGLE_EXT_RELAY, and
> it has no rDNS so the message matches __RDNS_NONE.
>
> So: why is that nameless machine that you cannot make a named machine
> NOT in internal_networks?
I don't know if this is the OP's case or not, but the following example
comes to mind.
SA (running on your receiving MTA) receives a message from an MTA (which
is itself an MSA) of an external Business-to-Business partner (thus a
trusted MTA that is not internal to the recipient's organization) which
itself received the message from a client on an RFC 1918 network without
reverse DNS.
> Of course not, but if a machine is trusted to tell the truth in Received
> headers and has no rDNS because it is talking to a close affiliate on a
> RFC1918 IP, in what sense is it not internal?
Trusting a B2B partner's external MTA.
> Or is it in internal_networks but there's something wrong in how SA is
> parsing Received headers to build X-Spam-Relays-External?
>
>
> I think the fix for all is for everyone to get their internal_networks
> and trusted_networks configurations correct.
What should trusted_networks and internal_networks be set to in the B2B
scenario I'm describing?
--
Grant. . . .
unix || die
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 30 Aug 2018, at 10:01, Matus UHLAR - fantomas wrote:
> On 30.08.18 09:49, Kevin A. McGrail wrote:
>> I feel that you are fighting a bigger battle than one rule in SA.
>
> two rules actually ;-) (with two more possible).
>
>> Without RDNS, you are running afoul of the postmaster rules of
>> virtually
>> every major email player. You will have massive deliverability
>> issues..
>
> Those IP addresses are in internal network with private IP ranges.
> When
> connecting to world, their IPs are NAtted to public.
>
> even if I fixed the DNS (and I can't since the network is not in my
> control), HDR_ORDER_FTSDMCXX_DIRECT would still apply.
It's hard to understand this circumstance based on the generic
description.
It appears that you have a configuration where a relay is in
trusted_networks (i.e. you believe what it asserts in Received headers)
but it is NOT in internal_networks so it is in the synthetic
X-Spam-Relays-External pseudo-header, it is the only element in
X-Spam-Relays-External so the message matches__DOS_SINGLE_EXT_RELAY, and
it has no rDNS so the message matches __RDNS_NONE.
So: why is that nameless machine that you cannot make a named machine
NOT in internal_networks?
> I believe faking DNS is not what you advise to me, although it would
> "fix"
> the problem temporarily (but could create another problem should the
> DNS be
> created later).
Of course not, but if a machine is trusted to tell the truth in Received
headers and has no rDNS because it is talking to a close affiliate on a
RFC1918 IP, in what sense is it not internal?
Or is it in internal_networks but there's something wrong in how SA is
parsing Received headers to build X-Spam-Relays-External?
> That is why I believe that adding ALL_TRUSTED would solve the problem
> without unnecessary issues for others.
>
> Yes, I can do that locally - but by redefining rule I could miss it
> getting
> fixes or improved later.
>
> And since different people have already reportted this problem in the
> past,
> I would like to make the fix possible for all, if viable.
I think the fix for all is for everyone to get their internal_networks
and trusted_networks configurations correct.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
On 30.08.18 09:49, Kevin A. McGrail wrote:
>I feel that you are fighting a bigger battle than one rule in SA.
two rules actually ;-) (with two more possible).
>Without RDNS, you are running afoul of the postmaster rules of virtually
>every major email player. You will have massive deliverability issues..
Those IP addresses are in internal network with private IP ranges. When
connecting to world, their IPs are NAtted to public.
even if I fixed the DNS (and I can't since the network is not in my
control), HDR_ORDER_FTSDMCXX_DIRECT would still apply.
I believe faking DNS is not what you advise to me, although it would "fix"
the problem temporarily (but could create another problem should the DNS be
created later).
That is why I believe that adding ALL_TRUSTED would solve the problem
without unnecessary issues for others.
Yes, I can do that locally - but by redefining rule I could miss it getting
fixes or improved later.
And since different people have already reportted this problem in the past,
I would like to make the fix possible for all, if viable.
>> On 30.08.18 09:24, Kevin A. McGrail wrote:
>>> Here is my response on the ticket:
>>>
>>> Outlook express ended production in June 2006. I'm not sure how much
>>> weight we can give to an email sent with it.
>On Thu, Aug 30, 2018 at 9:46 AM, Matus UHLAR - fantomas <uh...@fantomas.sk>
>wrote:
>> note that the issue is exactly the same with Windows Live Mail, which,
>> while unsupported, was available until Jan 2017 (and still seems to be
>> used in some organizations).
>>
>> The issue is at HDR_ORDER_FTSDMCXX_NORDNS with __RDNS_NONE.
>>> RDNS is an expected technology to setup a working mail server on the
>>> internet.
>> as written below, it's not so easy in organizations where mail server is
>> maintained by diferent people than internal network.
>> (and mailserver is in DMZ, while internal DNS servers in internal networ).
>>> Fix that and you have nearly 5 point swing on your email as well as
>>> likely more negative scoring rules will fire.
>> of course, there is more to fix and of course some of those fixes are
>> better
>> than others.
>>
>> However, I try to follow order:
>>
>> 1. what I can fix on mailserver
>> 2. what other admins can fix in the network
>> 3. what users can fix on their workstations.
>>
>> This is why I came with the ALL_TRUSTED workaround.
>>> Your focus on ALL_TRUSTED implies to me this is 100% internal mail. Is
>>> that correct?
>>>
>>
>> internal and/or outgoing.
>>
>> Do you (or anyone other) find problems when using ALL_TRUSTED?
>> On Thu, Aug 30, 2018 at 9:14 AM, Matus UHLAR - fantomas
>> <uh...@fantomas.sk> wrote:
>>>> the __HDR_ORDER_FTSDMCXXXX rule catches mail sent from windows live mail
>>>> (and outlook express, which, while obsolete, seems to be still used
>>>> often)
>>>>
>>>> That further causes hitting HDR_ORDER_FTSDMCXX_DIRECT and
>>>> HDR_ORDER_FTSDMCXX_NORDNS in cases where client uses the mail client on
>>>> local network, without SMTP authentication, and without DNS (which may be
>>>> quite common in some organizations).
>>>>
>>>> as a workaround, I recommend to add && !ALL_TRUSTED to
>>>> HDR_ORDER_FTSDMCXX_DIRECT and HDR_ORDER_FTSDMCXX_NORDNS rules.
>>>>
>>>> an example:
>>>>
>>>> X-Spam-Status: Yes, score=9.154 required=5.6 tests=[ALL_TRUSTED=-1,
>>>> DOS_OE_TO_MX=3.086, FSL_HELO_NON_FQDN_1=0.001,
>>>> HDR_ORDER_FTSDMCXX_DIRECT=1.999, HDR_ORDER_FTSDMCXX_NORDNS=3.5,
>>>> HTML_MESSAGE=0.001, MIMEOLE_DIRECT_TO_MX=0.293, RDNS_NONE=1.274]
>>>> autolearn=no autolearn_force=no
>>>> X-Mailer: Microsoft Outlook Express 6.00.2900.5931
>>>> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157
>>>>
>>>> I have filled out bug 7607, it got rejected immediately:
>>>>
>>>> https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7607
>>>>
>>>>
>>>> while I agree that fixing RDNS will help, internal networks DNS is not
>>>> always easy, especially when maintained by different people and when
>>>> internal DNS is in LAn, not in DMZ.
>>>>
>>>>
>>>> note that this problem has been reported on spamassassin-users a month
>>>> ago:
>>>>
>>>> http://spamassassin.1065346.n5.nabble.com/Problem-with-new-
>>>> rules-td152105.html
>>>>
>>>>
>>>> So, to avoid discussions on bugzilla, I prefer asking here:
>>>>
>>>> Is it really a problem to add && !ALL_TRUSTED to
>>>> HDR_ORDER_FTSDMCXX_DIRECT
>>>> and HDR_ORDER_FTSDMCXX_NORDNS ?
>>>>
>>>> (maybe even HDR_ORDER_FTSDMCXX_001C and HDR_ORDER_FTSDMCXX_BAT, if their
>>>> score will be more than zero)
>>>>
>>>
>> --
>> Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
>> Warning: I wish NOT to receive e-mail advertising to this address.
>> Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
>> I feel like I'm diagonally parked in a parallel universe.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
You have the right to remain silent. Anything you say will be misquoted,
then used against you.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook express)
Posted by "Kevin A. McGrail" <km...@apache.org>.
Matus,
I feel that you are fighting a bigger battle than one rule in SA.
Without RDNS, you are running afoul of the postmaster rules of virtually
every major email player. You will have massive deliverability issues..
You are asking for a bandaid when you've just lost a leg to shark biite and
more are swimming around.
You *MUST* fix RDNS to effectively email the world. It is a waste of time
to look at email deliverability when that basic step is missed.
--
Kevin A. McGrail
VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171
On Thu, Aug 30, 2018 at 9:46 AM, Matus UHLAR - fantomas <uh...@fantomas.sk>
wrote:
> On 30.08.18 09:24, Kevin A. McGrail wrote:
>
>> Thanks Matus. This is a good place to debate, thank you.
>>
>> Here is my response on the ticket:
>>
>> Outlook express ended production in June 2006. I'm not sure how much
>> weight we can give to an email sent with it.
>>
>
> note that the issue is exactly the same with Windows Live Mail, which,
> while
> unsupported, was available until Jan 2017 (and still seems to be used in
> some organizations).
>
> The issue is at HDR_ORDER_FTSDMCXX_NORDNS with __RDNS_NONE.
>>
>> RDNS is an expected technology to setup a working mail server on the
>> internet.
>>
>
> as written below, it's not so easy in organizations where mail server is
> maintained by diferent people than internal network.
> (and mailserver is in DMZ, while internal DNS servers in internal networ).
>
> Fix that and you have nearly 5 point swing on your email as well as
>> likely more negative scoring rules will fire.
>>
>
> of course, there is more to fix and of course some of those fixes are
> better
> than others.
>
> However, I try to follow order:
>
> 1. what I can fix on mailserver
> 2. what other admins can fix in the network
> 3. what users can fix on their workstations.
>
> This is why I came with the ALL_TRUSTED workaround.
>
> Your focus on ALL_TRUSTED implies to me this is 100% internal mail. Is
>> that correct?
>>
>
> internal and/or outgoing.
>
> Do you (or anyone other) find problems when using ALL_TRUSTED?
>
> On Thu, Aug 30, 2018 at 9:14 AM, Matus UHLAR - fantomas <uhlar@fantomas.sk
>> >
>> wrote:
>>
>>> the __HDR_ORDER_FTSDMCXXXX rule catches mail sent from windows live mail
>>> (and outlook express, which, while obsolete, seems to be still used
>>> often)
>>>
>>> That further causes hitting HDR_ORDER_FTSDMCXX_DIRECT and
>>> HDR_ORDER_FTSDMCXX_NORDNS in cases where client uses the mail client on
>>> local network, without SMTP authentication, and without DNS (which may be
>>> quite common in some organizations).
>>>
>>> as a workaround, I recommend to add && !ALL_TRUSTED to
>>> HDR_ORDER_FTSDMCXX_DIRECT and HDR_ORDER_FTSDMCXX_NORDNS rules.
>>>
>>> an example:
>>>
>>> X-Spam-Status: Yes, score=9.154 required=5.6 tests=[ALL_TRUSTED=-1,
>>> DOS_OE_TO_MX=3.086, FSL_HELO_NON_FQDN_1=0.001,
>>> HDR_ORDER_FTSDMCXX_DIRECT=1.999, HDR_ORDER_FTSDMCXX_NORDNS=3.5,
>>> HTML_MESSAGE=0.001, MIMEOLE_DIRECT_TO_MX=0.293, RDNS_NONE=1.274]
>>> autolearn=no autolearn_force=no
>>> X-Mailer: Microsoft Outlook Express 6.00.2900.5931
>>> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157
>>>
>>> I have filled out bug 7607, it got rejected immediately:
>>>
>>> https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7607
>>>
>>>
>>> while I agree that fixing RDNS will help, internal networks DNS is not
>>> always easy, especially when maintained by different people and when
>>> internal DNS is in LAn, not in DMZ.
>>>
>>>
>>> note that this problem has been reported on spamassassin-users a month
>>> ago:
>>>
>>> http://spamassassin.1065346.n5.nabble.com/Problem-with-new-
>>> rules-td152105.html
>>>
>>>
>>> So, to avoid discussions on bugzilla, I prefer asking here:
>>>
>>> Is it really a problem to add && !ALL_TRUSTED to
>>> HDR_ORDER_FTSDMCXX_DIRECT
>>> and HDR_ORDER_FTSDMCXX_NORDNS ?
>>>
>>> (maybe even HDR_ORDER_FTSDMCXX_001C and HDR_ORDER_FTSDMCXX_BAT, if their
>>> score will be more than zero)
>>>
>>
> --
> Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
> Warning: I wish NOT to receive e-mail advertising to this address.
> Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
> I feel like I'm diagonally parked in a parallel universe.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
On 30.08.18 09:24, Kevin A. McGrail wrote:
>Thanks Matus. This is a good place to debate, thank you.
>
>Here is my response on the ticket:
>
>Outlook express ended production in June 2006. I'm not sure how much
>weight we can give to an email sent with it.
note that the issue is exactly the same with Windows Live Mail, which, while
unsupported, was available until Jan 2017 (and still seems to be used in
some organizations).
>The issue is at HDR_ORDER_FTSDMCXX_NORDNS with __RDNS_NONE.
>
>RDNS is an expected technology to setup a working mail server on the internet.
as written below, it's not so easy in organizations where mail server is
maintained by diferent people than internal network.
(and mailserver is in DMZ, while internal DNS servers in internal networ).
>Fix that and you have nearly 5 point swing on your email as well as
>likely more negative scoring rules will fire.
of course, there is more to fix and of course some of those fixes are better
than others.
However, I try to follow order:
1. what I can fix on mailserver
2. what other admins can fix in the network
3. what users can fix on their workstations.
This is why I came with the ALL_TRUSTED workaround.
>Your focus on ALL_TRUSTED implies to me this is 100% internal mail. Is
>that correct?
internal and/or outgoing.
Do you (or anyone other) find problems when using ALL_TRUSTED?
>On Thu, Aug 30, 2018 at 9:14 AM, Matus UHLAR - fantomas <uh...@fantomas.sk>
>wrote:
>> the __HDR_ORDER_FTSDMCXXXX rule catches mail sent from windows live mail
>> (and outlook express, which, while obsolete, seems to be still used often)
>>
>> That further causes hitting HDR_ORDER_FTSDMCXX_DIRECT and
>> HDR_ORDER_FTSDMCXX_NORDNS in cases where client uses the mail client on
>> local network, without SMTP authentication, and without DNS (which may be
>> quite common in some organizations).
>>
>> as a workaround, I recommend to add && !ALL_TRUSTED to
>> HDR_ORDER_FTSDMCXX_DIRECT and HDR_ORDER_FTSDMCXX_NORDNS rules.
>>
>> an example:
>>
>> X-Spam-Status: Yes, score=9.154 required=5.6 tests=[ALL_TRUSTED=-1,
>> DOS_OE_TO_MX=3.086, FSL_HELO_NON_FQDN_1=0.001,
>> HDR_ORDER_FTSDMCXX_DIRECT=1.999, HDR_ORDER_FTSDMCXX_NORDNS=3.5,
>> HTML_MESSAGE=0.001, MIMEOLE_DIRECT_TO_MX=0.293, RDNS_NONE=1.274]
>> autolearn=no autolearn_force=no
>> X-Mailer: Microsoft Outlook Express 6.00.2900.5931
>> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157
>>
>> I have filled out bug 7607, it got rejected immediately:
>>
>> https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7607
>>
>>
>> while I agree that fixing RDNS will help, internal networks DNS is not
>> always easy, especially when maintained by different people and when
>> internal DNS is in LAn, not in DMZ.
>>
>>
>> note that this problem has been reported on spamassassin-users a month ago:
>>
>> http://spamassassin.1065346.n5.nabble.com/Problem-with-new-
>> rules-td152105.html
>>
>>
>> So, to avoid discussions on bugzilla, I prefer asking here:
>>
>> Is it really a problem to add && !ALL_TRUSTED to HDR_ORDER_FTSDMCXX_DIRECT
>> and HDR_ORDER_FTSDMCXX_NORDNS ?
>>
>> (maybe even HDR_ORDER_FTSDMCXX_001C and HDR_ORDER_FTSDMCXX_BAT, if their
>> score will be more than zero)
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I feel like I'm diagonally parked in a parallel universe.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook express)
Posted by "Kevin A. McGrail" <km...@apache.org>.
Thanks Matus. This is a good place to debate, thank you.
Here is my response on the ticket:
Outlook express ended production in June 2006. I'm not sure how much
weight we can give to an email sent with it.
The issue is at HDR_ORDER_FTSDMCXX_NORDNS with __RDNS_NONE.
RDNS is an expected technology to setup a working mail server on the internet.
Fix that and you have nearly 5 point swing on your email as well as
likely more negative scoring rules will fire.
Your focus on ALL_TRUSTED implies to me this is 100% internal mail. Is
that correct?
Regards,
KAM
--
Kevin A. McGrail
VP Fundraising, Apache Software Foundation
Chair Emeritus Apache SpamAssassin Project
https://www.linkedin.com/in/kmcgrail - 703.798.0171
On Thu, Aug 30, 2018 at 9:14 AM, Matus UHLAR - fantomas <uh...@fantomas.sk>
wrote:
> Hello,
>
> the __HDR_ORDER_FTSDMCXXXX rule catches mail sent from windows live mail
> (and outlook express, which, while obsolete, seems to be still used often)
>
> That further causes hitting HDR_ORDER_FTSDMCXX_DIRECT and
> HDR_ORDER_FTSDMCXX_NORDNS in cases where client uses the mail client on
> local network, without SMTP authentication, and without DNS (which may be
> quite common in some organizations).
>
> as a workaround, I recommend to add && !ALL_TRUSTED to
> HDR_ORDER_FTSDMCXX_DIRECT and HDR_ORDER_FTSDMCXX_NORDNS rules.
>
> an example:
>
> X-Spam-Status: Yes, score=9.154 required=5.6 tests=[ALL_TRUSTED=-1,
> DOS_OE_TO_MX=3.086, FSL_HELO_NON_FQDN_1=0.001,
> HDR_ORDER_FTSDMCXX_DIRECT=1.999, HDR_ORDER_FTSDMCXX_NORDNS=3.5,
> HTML_MESSAGE=0.001, MIMEOLE_DIRECT_TO_MX=0.293, RDNS_NONE=1.274]
> autolearn=no autolearn_force=no
> X-Mailer: Microsoft Outlook Express 6.00.2900.5931
> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157
>
> I have filled out bug 7607, it got rejected immediately:
>
> https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7607
>
>
> while I agree that fixing RDNS will help, internal networks DNS is not
> always easy, especially when maintained by different people and when
> internal DNS is in LAn, not in DMZ.
>
>
> note that this problem has been reported on spamassassin-users a month ago:
>
> http://spamassassin.1065346.n5.nabble.com/Problem-with-new-
> rules-td152105.html
>
>
> So, to avoid discussions on bugzilla, I prefer asking here:
>
> Is it really a problem to add && !ALL_TRUSTED to HDR_ORDER_FTSDMCXX_DIRECT
> and HDR_ORDER_FTSDMCXX_NORDNS ?
>
> (maybe even HDR_ORDER_FTSDMCXX_001C and HDR_ORDER_FTSDMCXX_BAT, if their
> score will be more than zero)
> --
> Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
> Warning: I wish NOT to receive e-mail advertising to this address.
> Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
> 99 percent of lawyers give the rest a bad name.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by John Hardin <jh...@impsec.org>.
On Fri, 7 Sep 2018, Matus UHLAR - fantomas wrote:
> For now, I believe that using (ALL_TRUSTED && __DOS_SINGLE_EXT_RELAY)
> is just what I need to prevent all rules from firing:
I think you mean !ALL_TRUSTED, right?
Will digest and comment on the rest in a bit.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Running away is the coward's way out of a war;
appeasement is the coward's way into a war. -- Thorax
-----------------------------------------------------------------------
10 days until the 231st anniversary of the signing of the U.S. Constitution
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>>>On Fri, 31 Aug 2018, John Hardin wrote:
>>>>None of the masscheck corpora that hit __HDR_ORDER_FTSDMCXXXX also
>>>>hit ALL_TRUSTED (or at least the portion is so small it falls off
>>>>the bottom of the report) so I don't feel too worried about adding
>>>>either !ALL_TRUSTED or __ANY_EXTERNAL (or potentially both) as
>>>>exclusions.
>>>>
>>>>I'm adding __ANY_EXTERNAL now...
>>>>
>>>>Comments solicited.
>>On Fri, 31 Aug 2018 16:16:43 -0700 (PDT) John Hardin wrote:
>>>Here's one: should __ANY_EXTERNAL be added to any other rules that
>>>primarily look for abused MSFT-isms?
>>>
>>>For example, MIMEOLE_DIRECT_TO_MX, DOS_OE_TO_MX, DOS_OUTLOOK_TO_MX,
>>>XPRIO_SHORT_SUBJ, ...?
>On Sat, 1 Sep 2018, RW wrote:
>>All but the last one is a direct-to-mx rule, which requires one
>>external relay, so adding __ANY_EXTERNAL to those is pointless.
On 01.09.18 18:57, John Hardin wrote:
>Ugh, you're right. I didn't reread the rule details before posting
>that suggestion - sorry, I've been a little distracted by plumbing
>issues this week. :)
>
>__ANY_EXTERNAL on HDR_ORDER_FTSDMCXX_DIRECT is also pointless because
>it uses __DOS_SINGLE_EXT_RELAY, which is "exactly one external IP
>present." Same for HDR_ORDER_FTSDMCXX_NORDNS with __RDNS_NONE. Taking
>__ANY_EXTERNAL back off of those. Same excuse. :)
>!ALL_TRUSTED will be masscheck-neutral and will help in the situation
>you describe, so I'll add it; the only failure mode I can see there is
>if you add an external ESP to your trusted networks and they discard
>internal and submission details so that they look like a MUA, and then
>one of their clients sends spam that would otherwise hit the rule. Is
>an ESP doing that considered "forging headers" sufficiently to *not*
>earn trust? Or does simply *discarding* headers not cross that line?
I believe that discarding Received: headers DOES cross the line.
As long as the point of trusted_hosts is to skip checking for them in
blacklists BUT checking hosts behind instead, clearing Received: headers
causes sabotaging the trust here.
removing/not adding those headers in fact means forging them
>>I don't think __ANY_EXTERNAL is a good idea, it should be sufficient
>>that the headers are all trusted
>
>Trusted and Internal are different things. I think it's a bad idea to
>conflate them or treat them as equivalent and interchangeable.
>
>I think __ANY_EXTERNAL is still weakly needed. There's a rule for
>exactly one external IP (__DOS_SINGLE_EXT_RELAY) and there's a rule
>for multiple external IPs (__DOS_RELAYED_EXT) but there's nothing for
>"are there *any* external relays?" __DOS_SINGLE_EXT_RELAY ||
>__DOS_RELAYED_EXT would be equivalent but I feel it should be more
>direct than that for clarity, unless we have performance concerns with
>another RE vs. a meta, which is unlikely.
I believe that for the rules in account, __DOS_SINGLE_EXT_RELAY is just fine
- in my case (I am the OP) we are handling mail that came directly from
trusted but external hosts.
>>__ANY_EXTERNAL requires that people read this thread and make a
>>questionable change to their networks to take advantage.
I don't think so. The documentation seems to be quite clear in what should
and what should not be put into trusted_networks and internal_networks.
>Actually listing in internal_networks IPs considered "internal to the
>organization" is questionable?
>
>If there's some issue with listing public dialup (presumably dynamic)
>IPs used by members of the organization in internal_networks,
not internal_networks, only in trusted_networks. Just as documented in
https://wiki.apache.org/spamassassin/DynablockIssues
> then
>maybe we need another way to specify "these IPs are considered
>internal for submission purposes even though they don't authenticate".
This could help much, but it also means much work with SA changes.
For now, I believe that using (ALL_TRUSTED && __DOS_SINGLE_EXT_RELAY)
is just what I need to prevent all rules from firing:
HDR_ORDER_FTSDMCXX_001C
HDR_ORDER_FTSDMCXX_BAT
HDR_ORDER_FTSDMCXX_DIRECT
HDR_ORDER_FTSDMCXX_NORDNS
... as long as some mentioned above:
DOS_HIGH_BAT_TO_MX
DOS_OE_TO_MX
DOS_OE_TO_MX_IMAGE
DOS_OUTLOOK_TO_MX
MIMEOLE_DIRECT_TO_MX
Further thinking about trusted_networks and relaying mail, I think that
the difference between remote server and local trusted client, when both are
listed in trusted_networks, is that mail from remote server contains more
than one Received: header, while local trusted client only one.
In this case, trusted server removing or not providing any Received: lines
would be understood as client, avoiding hitting rules above.
OTOH, trusted clients providing fake headers are something that could cause
troubles by hitting whitelist -firsttrusted rules.
This is the only problem I can see coming from trusting local clients - but
since they must be trusted to avoid local blacklist, I see no way to avoid
this than change to SA trust path (drop all Received: after this hosts).
comments welcome.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
- Holmes, what kind of school did you study to be a detective?
- Elementary, Watson. -- Daffy Duck & Porky Pig
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by John Hardin <jh...@impsec.org>.
On Sat, 1 Sep 2018, RW wrote:
> On Fri, 31 Aug 2018 16:16:43 -0700 (PDT)
> John Hardin wrote:
>
>> On Fri, 31 Aug 2018, John Hardin wrote:
>>
>>> None of the masscheck corpora that hit __HDR_ORDER_FTSDMCXXXX also
>>> hit ALL_TRUSTED (or at least the portion is so small it falls off
>>> the bottom of the report) so I don't feel too worried about adding
>>> either !ALL_TRUSTED or __ANY_EXTERNAL (or potentially both) as
>>> exclusions.
>>>
>>> I'm adding __ANY_EXTERNAL now...
>>>
>>> Comments solicited.
>>
>> Here's one: should __ANY_EXTERNAL be added to any other rules that
>> primarily look for abused MSFT-isms?
>>
>> For example, MIMEOLE_DIRECT_TO_MX, DOS_OE_TO_MX, DOS_OUTLOOK_TO_MX,
>> XPRIO_SHORT_SUBJ, ...?
>
> All but the last one is a direct-to-mx rule, which requires one
> external relay, so adding __ANY_EXTERNAL to those is pointless.
Ugh, you're right. I didn't reread the rule details before posting that
suggestion - sorry, I've been a little distracted by plumbing issues this
week. :)
__ANY_EXTERNAL on HDR_ORDER_FTSDMCXX_DIRECT is also pointless because it
uses __DOS_SINGLE_EXT_RELAY, which is "exactly one external IP present."
Same for HDR_ORDER_FTSDMCXX_NORDNS with __RDNS_NONE. Taking __ANY_EXTERNAL
back off of those. Same excuse. :)
!ALL_TRUSTED will be masscheck-neutral and will help in the situation you
describe, so I'll add it; the only failure mode I can see there is if you
add an external ESP to your trusted networks and they discard internal and
submission details so that they look like a MUA, and then one of their
clients sends spam that would otherwise hit the rule. Is an ESP doing that
considered "forging headers" sufficiently to *not* earn trust? Or does
simply *discarding* headers not cross that line?
> I'm curious why you have
>
> header ANY_EXTERNAL_RELAY ALL-EXTERNAL =~ /\S/
>
> which looks for an external header rather than the more straightforward
>
> header ANY_EXTERNAL_RELAY X-Spam-Relays-External =~ /\S/
>
> which looks for an external relay. I think they are functionally
> equivalent.
You're right, they should be equivalent. The former is a little shorter.
The latter is what I actually checked into SVN, for consistency with (most
of) the other "external" rules.
I don't know that one is more "straightforward" than the other.
> I don't think __ANY_EXTERNAL is a good idea, it should be sufficient
> that the headers are all trusted
Trusted and Internal are different things. I think it's a bad idea to
conflate them or treat them as equivalent and interchangeable.
I think __ANY_EXTERNAL is still weakly needed. There's a rule for exactly
one external IP (__DOS_SINGLE_EXT_RELAY) and there's a rule for multiple
external IPs (__DOS_RELAYED_EXT) but there's nothing for "are there *any*
external relays?" __DOS_SINGLE_EXT_RELAY || __DOS_RELAYED_EXT would be
equivalent but I feel it should be more direct than that for clarity,
unless we have performance concerns with another RE vs. a meta, which is
unlikely.
> __ANY_EXTERNAL requires that people read this thread and make a
> questionable change to their networks to take advantage.
Actually listing in internal_networks IPs considered "internal to the
organization" is questionable?
If there's some issue with listing public dialup (presumably dynamic) IPs
used by members of the organization in internal_networks, then maybe we
need another way to specify "these IPs are considered internal for
submission purposes even though they don't authenticate".
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Should you meet with a person bent on a campaign of terror,
intending to murder their fellow men and women, to leave behind a
swath of widows, widowers and orphans, to grieve families and
nations alike, do the reasonable thing. Kill them.
-- Matthew @ StraightForward
-----------------------------------------------------------------------
520 days since the first commercial re-flight of an orbital booster (SpaceX)
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by RW <rw...@googlemail.com>.
On Fri, 31 Aug 2018 16:16:43 -0700 (PDT)
John Hardin wrote:
> On Fri, 31 Aug 2018, John Hardin wrote:
>
> > None of the masscheck corpora that hit __HDR_ORDER_FTSDMCXXXX also
> > hit ALL_TRUSTED (or at least the portion is so small it falls off
> > the bottom of the report) so I don't feel too worried about adding
> > either !ALL_TRUSTED or __ANY_EXTERNAL (or potentially both) as
> > exclusions.
> >
> > I'm adding __ANY_EXTERNAL now...
> >
> > Comments solicited.
>
> Here's one: should __ANY_EXTERNAL be added to any other rules that
> primarily look for abused MSFT-isms?
>
> For example, MIMEOLE_DIRECT_TO_MX, DOS_OE_TO_MX, DOS_OUTLOOK_TO_MX,
> XPRIO_SHORT_SUBJ, ...?
All but the last one is a direct-to-mx rule, which requires one
external relay, so adding __ANY_EXTERNAL to those is pointless.
I'm curious why you have
header ANY_EXTERNAL_RELAY ALL-EXTERNAL =~ /\S/
which looks for an external header rather than the more straightforward
header ANY_EXTERNAL_RELAY X-Spam-Relays-External =~ /\S/
which looks for an external relay. I think they are functionally
equivalent.
I don't think __ANY_EXTERNAL is a good idea, it should be sufficient
that the headers are all trusted, __ANY_EXTERNAL requires that people
read this thread and make a questionable change to their networks to
take advantage.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>On Fri, 31 Aug 2018, John Hardin wrote:
>>None of the masscheck corpora that hit __HDR_ORDER_FTSDMCXXXX also
>>hit ALL_TRUSTED (or at least the portion is so small it falls off
>>the bottom of the report) so I don't feel too worried about adding
>>either !ALL_TRUSTED or __ANY_EXTERNAL (or potentially both) as
>>exclusions.
>>
>>I'm adding __ANY_EXTERNAL now...
>>
>>Comments solicited.
On 31.08.18 16:16, John Hardin wrote:
>Here's one: should __ANY_EXTERNAL be added to any other rules that
>primarily look for abused MSFT-isms?
>
>For example, MIMEOLE_DIRECT_TO_MX, DOS_OE_TO_MX, DOS_OUTLOOK_TO_MX,
>XPRIO_SHORT_SUBJ, ...?
Now that you pulled this out...
Yes, it would also help on some servers I maintain (where HDR_ORDER_FTSDMCXX*
caused troubles).
The question I still have is, if this is not in contrast with proposed usage
of __ANY_EXTERNAL or !ALL_TRUSTED
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
If Barbie is so popular, why do you have to buy her friends?
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by John Hardin <jh...@impsec.org>.
On Fri, 31 Aug 2018, John Hardin wrote:
> None of the masscheck corpora that hit __HDR_ORDER_FTSDMCXXXX also hit
> ALL_TRUSTED (or at least the portion is so small it falls off the bottom of
> the report) so I don't feel too worried about adding either !ALL_TRUSTED or
> __ANY_EXTERNAL (or potentially both) as exclusions.
>
> I'm adding __ANY_EXTERNAL now...
>
> Comments solicited.
Here's one: should __ANY_EXTERNAL be added to any other rules that
primarily look for abused MSFT-isms?
For example, MIMEOLE_DIRECT_TO_MX, DOS_OE_TO_MX, DOS_OUTLOOK_TO_MX,
XPRIO_SHORT_SUBJ, ...?
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Of the twenty-two civilizations that have appeared in history,
nineteen of them collapsed when they reached the moral state the
United States is in now. -- Arnold Toynbee
-----------------------------------------------------------------------
519 days since the first commercial re-flight of an orbital booster (SpaceX)
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by John Hardin <jh...@impsec.org>.
On Fri, 31 Aug 2018, John Hardin wrote:
> On Fri, 31 Aug 2018, Matus UHLAR - fantomas wrote:
>
>>> On Thu, 30 Aug 2018, Matus UHLAR - fantomas wrote:
>>>> That further causes hitting HDR_ORDER_FTSDMCXX_DIRECT and
>>>> HDR_ORDER_FTSDMCXX_NORDNS in cases where client uses the mail client on
>>>> local network, without SMTP authentication, and without DNS (which may be
>>>> quite common in some organizations).
>>
>> On 30.08.18 16:57, John Hardin wrote:
>>> Are you experiencing this yourself, so that you can do some testing?
>>
>> Yes.
>
> Thanks!
>
>>> If you do have a repro env, can you check whether that internal network is
>>> listed as such in the SA config?
>>>
>>> Would you be willing to do this and report whether it hits on those
>>> messages?
>>>
>>> header ANY_EXTERNAL_RELAY ALL-EXTERNAL =~ /\S/
>>> score ANY_EXTERNAL_RELAY 0.001
>>
>> I have tested: ANY_EXTERNAL_RELAY appears when the client's IP is in
>> trusted_networks, it does not when it's in internal_networks.
>
> It shouldn't have anything to do with trusted_networks, it's intended to
> check whether or not all the participating IPs are in internal_networks.
> There's currently no rule for doing that.
__ANY_EXTERNAL hits 99.9% of spam and 97.6% of ham. I'd suggest that
masscheckers might want to see if they can add ham from internal users to
other internal users, especially if it looks spammy were it to be received
from an external source.
>>> Filtering on "has an external relay" might be preferable to filtering on
>>> !ALL_TRUSTED since "trust" doesn't say anything about rDNS or it being a
>>> MUA.
None of the masscheck corpora that hit __HDR_ORDER_FTSDMCXXXX also hit
ALL_TRUSTED (or at least the portion is so small it falls off the bottom
of the report) so I don't feel too worried about adding either
!ALL_TRUSTED or __ANY_EXTERNAL (or potentially both) as exclusions.
I'm adding __ANY_EXTERNAL now...
Comments solicited.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
At $8 billion per year, the TSA is the most expensive
theatrical production in history. -- David Burge @iowahawkblog
-----------------------------------------------------------------------
519 days since the first commercial re-flight of an orbital booster (SpaceX)
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>On Fri, 31 Aug 2018, Matus UHLAR - fantomas wrote:
>>Note that I list internal clients as trusted, not as internal.
>>
>>Maybe this is the problem. Long time ago I learned to configure
>>dynamic IP addresses (dialups) as trusted, but not as internal.
On 31.08.18 12:07, John Hardin wrote:
>Hrm. Not sure which way to go in that case. Dialup IPs (unless
>statically assigned to a specific user account) are not really a
>reliable indicator of internal or trusted... Any of that ISP's clients
>could get that IP and suddenly be able to get preferential treatment
>by your mail system.
>
>>In this case, clients are internal, not dialup, but I still think they
>>should not be listed in internal_networks (as I don't trust them not to
>>spoof anything).
>
>Trusting to not spoof headers is what the trusted_networks list is for.
I agree and this is something I repeatedfly tought of for long time.
But so far we had nothing else to avoid catching non-authenticated
clients than listing them in *_networks (and I still found trusted_networks
more than internal_networks).
>>HDR_ORDER_FTSDMCXX* is the one I'm trying to solve.
>
>Well, that's basically a just check for MSFT MUAs, and spam tools that
>slavishly mimic the headers such clients produce...
unfortunately, they catch MUAs as long as those spam tools. We need
something to avoid real MUAs until we have better spam tool detection.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
My mind is like a steel trap - rusty and illegal in 37 states.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by John Hardin <jh...@impsec.org>.
On Fri, 31 Aug 2018, Matus UHLAR - fantomas wrote:
>> On Thu, 30 Aug 2018, Matus UHLAR - fantomas wrote:
>>> That further causes hitting HDR_ORDER_FTSDMCXX_DIRECT and
>>> HDR_ORDER_FTSDMCXX_NORDNS in cases where client uses the mail client on
>>> local network, without SMTP authentication, and without DNS (which may be
>>> quite common in some organizations).
>
> On 30.08.18 16:57, John Hardin wrote:
>> Are you experiencing this yourself, so that you can do some testing?
>
> Yes.
Thanks!
>> If you do have a repro env, can you check whether that internal network is
>> listed as such in the SA config?
>>
>> Would you be willing to do this and report whether it hits on those
>> messages?
>>
>> header ANY_EXTERNAL_RELAY ALL-EXTERNAL =~ /\S/
>> score ANY_EXTERNAL_RELAY 0.001
>
> I have tested: ANY_EXTERNAL_RELAY appears when the client's IP is in
> trusted_networks, it does not when it's in internal_networks.
It shouldn't have anything to do with trusted_networks, it's intended to
check whether or not all the participating IPs are in internal_networks.
There's currently no rule for doing that.
> Note that I list internal clients as trusted, not as internal.
>
> Maybe this is the problem. Long time ago I learned to configure dynamic
> IP addresses (dialups) as trusted, but not as internal.
Hrm. Not sure which way to go in that case. Dialup IPs (unless statically
assigned to a specific user account) are not really a reliable indicator
of internal or trusted... Any of that ISP's clients could get that IP and
suddenly be able to get preferential treatment by your mail system.
> In this case, clients are internal, not dialup, but I still think they
> should not be listed in internal_networks (as I don't trust them not to
> spoof anything).
Trusting to not spoof headers is what the trusted_networks list is for.
>> Filtering on "has an external relay" might be preferable to filtering on
>> !ALL_TRUSTED since "trust" doesn't say anything about rDNS or it being a
>> MUA.
>
>
>>> note that this problem has been reported on spamassassin-users a month
>>> ago:
>>>
>>> http://spamassassin.1065346.n5.nabble.com/Problem-with-new-rules-td152105.html
>>
>> I'd say the problems aren't. That's because the ESP was relaying mail and
>> not reporting *any* details of the internal handoff, so it looked to the
>> recipient like the MSA was a mail client.
>>
>> rDNS wasn't an issue there.
>
> rDNS is not the issue on my side too (it is an issue though).
> HDR_ORDER_FTSDMCXX* is the one I'm trying to solve.
Well, that's basically a just check for MSFT MUAs, and spam
tools that slavishly mimic the headers such clients produce...
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Riff: Torg, you traded our magic beans for a _cow_?
Torg: It's a _magic_ cow! It's full of steaks!
Riff: Whoa! -- Sluggy 04/28/2002
-----------------------------------------------------------------------
519 days since the first commercial re-flight of an orbital booster (SpaceX)
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>On 31 Aug 2018, at 4:53, Matus UHLAR - fantomas wrote:
>>Long time ago I learned to configure dynamic IP addresses (dialups) as
>>trusted, but not as internal.
On 31.08.18 09:37, Bill Cole wrote:
>They probably should be neither.
>>In this case, clients are internal, not dialup, but I still think they
>>should not be listed in internal_networks (as I don't trust them not
>>to
>>spoof anything).
>
>If you do not trust them not to spoof anything, they absolutely must
>not be in trusted_networks.
in fact I have to trust them not to spoof at least the from/envelope
addresses. historical reasons, at least until something bad happend.
btw note that ALL_TRUSTED means that message was originated by trusted host,
not relayed by it - any untrusted host will clear this rule.
I have tried to remove them off the trusted_networks.
The only change was that ALL_TRUSTED is gone, and without it in meta,
HDR_ORDER_FTSDMCX* hit.
There are also many rules that search untrusted relays for things like
generic helo and DNS name.
In thic case, setting UP dns could mess things up even more.
As I see it, having those local machines in trusted_networks helps me even
more and it also makes me think if this isn't one of reasons
trusted_networks exist ...
>It seems to me that you have a technical & management arrangement
>unsuited to the SpamAssassin
>trusted_networks/internal_networks/msa_networks logical model.
This is quite possible, but even you have noted that you don't know
everything about parsing Received headers.
> My recommendation would NOT be to modify stock rules that are constructed
>with that logical model as a base assumption, but rather to create your own
>mitigating rules to handle the fact that you seem to want to always accept
>mail from certain internal clients which are nameless, untrustworthy, and
>sources of mail with features that in the world at large mostly correlate
>to spam.
However, I encounter these problems on multiple hosts with the same
conditions, and it's quite possible that different people have similar
issues, so I am searching for solution that helps me (ans poddibly others)
while does not break anything.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Depression is merely anger without enthusiasm.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Bill Cole <sa...@billmail.scconsult.com>.
On 31 Aug 2018, at 4:53, Matus UHLAR - fantomas wrote:
> Note that I list internal clients as trusted, not as internal.
>
> Maybe this is the problem.
Yes, maybe...
> Long time ago I learned to configure dynamic IP addresses (dialups) as
> trusted, but not as internal.
They probably should be neither.
> In this case, clients are internal, not dialup, but I still think they
> should not be listed in internal_networks (as I don't trust them not
> to
> spoof anything).
If you do not trust them not to spoof anything, they absolutely must not
be in trusted_networks.
It seems to me that you have a technical & management arrangement
unsuited to the SpamAssassin
trusted_networks/internal_networks/msa_networks logical model. My
recommendation would NOT be to modify stock rules that are constructed
with that logical model as a base assumption, but rather to create your
own mitigating rules to handle the fact that you seem to want to always
accept mail from certain internal clients which are nameless,
untrustworthy, and sources of mail with features that in the world at
large mostly correlate to spam.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by Matus UHLAR - fantomas <uh...@fantomas.sk>.
>On Thu, 30 Aug 2018, Matus UHLAR - fantomas wrote:
>>That further causes hitting HDR_ORDER_FTSDMCXX_DIRECT and
>>HDR_ORDER_FTSDMCXX_NORDNS in cases where client uses the mail client on
>>local network, without SMTP authentication, and without DNS (which may be
>>quite common in some organizations).
On 30.08.18 16:57, John Hardin wrote:
>Are you experiencing this yourself, so that you can do some testing?
Yes.
>If you do have a repro env, can you check whether that internal
>network is listed as such in the SA config?
>
>Would you be willing to do this and report whether it hits on those
>messages?
>
> header ANY_EXTERNAL_RELAY ALL-EXTERNAL =~ /\S/
> score ANY_EXTERNAL_RELAY 0.001
I have tested: ANY_EXTERNAL_RELAY appears when the client's IP is in
trusted_networks, it does not when it's in internal_networks.
Note that I list internal clients as trusted, not as internal.
Maybe this is the problem.
Long time ago I learned to configure dynamic IP addresses (dialups) as
trusted, but not as internal.
In this case, clients are internal, not dialup, but I still think they
should not be listed in internal_networks (as I don't trust them not to
spoof anything).
>Filtering on "has an external relay" might be preferable to filtering
>on !ALL_TRUSTED since "trust" doesn't say anything about rDNS or it
>being a MUA.
>>note that this problem has been reported on spamassassin-users a month ago:
>>
>>http://spamassassin.1065346.n5.nabble.com/Problem-with-new-rules-td152105.html
>
>I'd say the problems aren't. That's because the ESP was relaying mail
>and not reporting *any* details of the internal handoff, so it looked
>to the recipient like the MSA was a mail client.
>
>rDNS wasn't an issue there.
rDNS is not the issue on my side too (it is an issue though).
HDR_ORDER_FTSDMCXX* is the one I'm trying to solve.
--
Matus UHLAR - fantomas, uhlar@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Atheism is a non-prophet organization.
Re: __HDR_ORDER_FTSDMCXXXX hitting windows live mail (and outlook
express)
Posted by John Hardin <jh...@impsec.org>.
On Thu, 30 Aug 2018, Matus UHLAR - fantomas wrote:
> That further causes hitting HDR_ORDER_FTSDMCXX_DIRECT and
> HDR_ORDER_FTSDMCXX_NORDNS in cases where client uses the mail client on
> local network, without SMTP authentication, and without DNS (which may be
> quite common in some organizations).
Are you experiencing this yourself, so that you can do some testing?
> X-Spam-Status: Yes, score=9.154 required=5.6 tests=[ALL_TRUSTED=-1,
> DOS_OE_TO_MX=3.086, FSL_HELO_NON_FQDN_1=0.001,
> HDR_ORDER_FTSDMCXX_DIRECT=1.999, HDR_ORDER_FTSDMCXX_NORDNS=3.5,
> HTML_MESSAGE=0.001, MIMEOLE_DIRECT_TO_MX=0.293, RDNS_NONE=1.274]
> autolearn=no autolearn_force=no
> X-Mailer: Microsoft Outlook Express 6.00.2900.5931
> X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157
>
> while I agree that fixing RDNS will help, internal networks DNS is not
> always easy, especially when maintained by different people and when
> internal DNS is in LAn, not in DMZ.
Noted.
If you do have a repro env, can you check whether that internal network is
listed as such in the SA config?
Would you be willing to do this and report whether it hits on those
messages?
header ANY_EXTERNAL_RELAY ALL-EXTERNAL =~ /\S/
score ANY_EXTERNAL_RELAY 0.001
Filtering on "has an external relay" might be preferable to filtering on
!ALL_TRUSTED since "trust" doesn't say anything about rDNS or it being a
MUA.
> note that this problem has been reported on spamassassin-users a month ago:
>
> http://spamassassin.1065346.n5.nabble.com/Problem-with-new-rules-td152105.html
I'd say the problems aren't. That's because the ESP was relaying mail and
not reporting *any* details of the internal handoff, so it looked to the
recipient like the MSA was a mail client.
rDNS wasn't an issue there.
--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
Microsoft is not a standards body.
-----------------------------------------------------------------------
518 days since the first commercial re-flight of an orbital booster (SpaceX)